aboutsummaryrefslogtreecommitdiff
path: root/public/index.xml
diff options
context:
space:
mode:
Diffstat (limited to 'public/index.xml')
-rwxr-xr-xpublic/index.xml6059
1 files changed, 6059 insertions, 0 deletions
diff --git a/public/index.xml b/public/index.xml
new file mode 100755
index 0000000..de33fe3
--- /dev/null
+++ b/public/index.xml
@@ -0,0 +1,6059 @@
1<rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
2 <channel>
3 <title>Mitja Felicijan's posts</title>
4 <link>https://mitjafelicijan.com</link>
5 <description>You do not learn by relaxing. You learn by violently assaulting your problem until it surrenders its mysteries to you.</description>
6 <language>en-us</language>
7
8
9
10 <item>
11 <title>Who knows what the world will look like tomorrow</title>
12 <link>https://mitjafelicijan.com/who-knows-what-the-world-will-look-like-tomorrow.html</link>
13 <pubDate>Sat, 08 Jul 2023 18:49:07 &#43;0200</pubDate>
14 <guid>https://mitjafelicijan.com/who-knows-what-the-world-will-look-like-tomorrow.html</guid>
15 <description>This site has gone through a lot of changes over the years.</description>
16 <content:encoded>&lt;p&gt;This site has gone through a lot of changes over the years. From being written
17in Flask and Bottle to moving on to static site generators. I have used and
18tested probably 10s of them my now. From homebrew solutions to the biggest and
19the baddest. From Bash scripts to Node.js disasters. I&#39;ve seen some things, no
20doubt. Not all bad.&lt;/p&gt;
21&lt;p&gt;I&#39;have been closely observing the web and where the trends are going, and I
22don&#39;t like what I see. Instead of internet being this weird place where
23experimentation is happening, it all became stale and formulized. Boring,
24actually. Really boring. And sad. Where is that old, revolutionary FU spirit I
25remember? It&#39;s still there, I know. But it&#39;s being drowned by the voices of
26mediocrity and formulaic boredom.&lt;/p&gt;
27&lt;p&gt;It almost feels like that the internet stopped for 10 years and only now
28something has started happening. With all the insanity around the world. People
29hating people without actual reasons, just because it&#39;s fashionable to hate and
30crowd is saying so. Sad state of affairs.&lt;/p&gt;
31&lt;p&gt;All this is contributing to this overall negativity masked as apathy. Everybody
32walking in lockstep. Instead of being creative are bold, we are just
33re-inventing the world and making the same mistakes. Maybe, just maybe, some
34things are good enough and there is no need to try to be too smart for our own
35good. After N-attempts, maybe something should click inside our heads to maybe
36say: &amp;quot;This thing, opinion, etc. is actually really good, and even after several
37attempts it still holds.&amp;quot;&lt;/p&gt;
38&lt;p&gt;The older I get, the more careful I am of my own thoughts and why I think the
39way I think. More and more, I try to understand people with opposite
40opinions. Far from perfect, but closer to bearable. And then I see people
41hearing or reading a thing on internet and let&#39;s fucking goooooo! Strong
42opinions are a sign of a weak and uneducated mind. I am more and more sure of
43this.&lt;/p&gt;
44&lt;p&gt;It&#39;s gotten to a point where you can with great certainty deduce a person&#39;s
45personality based on one or two opinions. How boring have we become. No wonder
46people can&#39;t talk to each other. These would be very quick conversations anyway.&lt;/p&gt;
47&lt;p&gt;I just got remembered of a song, &amp;quot;Hi Ren&amp;quot;. The ending talks about being stiff
48and not being able to dance. Such an amazing metaphor. And we as people have
49gone so far, we can&#39;t even walk or even crawl normally anymore. We have
50forgotten that the most beautiful things in life have a great deal of
51uncertainty about them. We want instant gratification. Not only that, but we
52want absolute obedience. Complete control over others, because we have zero
53control of ourselves. And all the lies we could tell ourselves will not help us
54in this situation.&lt;/p&gt;
55&lt;p&gt;It is funny how I catch myself from time to time being a complete idiot. It&#39;s
56like having an outer body experience. I can see myself being an idiot, and
57cannot stop myself. It serves as a learning lesson to stop before speaking. To
58think before saying. And to crawl before walking.&lt;/p&gt;
59&lt;p&gt;So there is still time. We can dance once more. All we need to do is stop for a
60second. Me and you. Us two is a start. Let&#39;s not try to change the world, but
61rather nudge ourselves just a tiny bit. And if we only did that. Each of us
62nudged ourselves a small, tiny bit, the world would heal. If we would just put
63down the phones and ignored Internet for a day or two. Put visiting websites
64that feed on us on hold. Listened to just one sentence and try to understand it
65from a person who we completely disagree with. I truly believe that this is
66possible.&lt;/p&gt;
67&lt;p&gt;Life is about suffering and joy. And instead of wishing suffering on others and
68excepting joy for yourselves, we should for a brief moment want suffering for
69ourselves and wish joy on others. Wouldn&#39;t that be an amazing sight to see?&lt;/p&gt;
70&lt;p&gt;I caught myself hating on Rust. And I deeply thought about it afterward. Why did
71I do it? It is obviously not for me. So why the hell was I being so negative
72towards it? I think that I know the answer. I was negative because that is
73easy. Because it&#39;s much easier to hate on things than to say to yourself: &amp;quot;Well,
74you know what? This is not for me. I will focus on creation and not
75destruction. This is who I want to be. This is what fills me with joy and
76purpose.&amp;quot; Where joy is keeping me happy and purpose scares the shit out of me
77and keeps me honest. This is who I want to be. Admit to myself when I am wrong
78and accept the faults that I have without reservation and with courage march on.&lt;/p&gt;
79&lt;p&gt;I just realized that this blog post is a sort of therapy for me. It&#39;s
80cathartic. Going thought the history of this site and remembering all the
81decisions and annoyances that came with it. When I was cursing at the tools. And
82time moved on, and the site is still here. It serves as a reminder that
83perseverance wins at the end. If we just let things go.&lt;/p&gt;
84&lt;p&gt;This came with a decision that simplifying life and removing all the unnecessary
85negativity is key. Rather than worrying about what the internet is saying, what
86the world is trying to take from you, you are the only one who can say no. And
87create instead of destroy.&lt;/p&gt;
88&lt;p&gt;I don&#39;t have an ending for this post, so I will say this. We live in the most
89amazing times in the recorded history, and we should be internally grateful for
90it. Create and study, this should be my mantra. Just create and let the world
91happen. And you feel yourself to be too certain, stop and check how deep in the
92shit you are already. Strong opinions are a sign of a weak and uneducated
93mind. Hate and disdain is for the weak.&lt;/p&gt;
94</content:encoded>
95 </item>
96
97
98
99 <item>
100 <title>Bringing all of my projects together under one umbrella</title>
101 <link>https://mitjafelicijan.com/bringing-all-of-my-projects-together-under-one-umbrella.html</link>
102 <pubDate>Sat, 01 Jul 2023 18:49:07 &#43;0200</pubDate>
103 <guid>https://mitjafelicijan.com/bringing-all-of-my-projects-together-under-one-umbrella.html</guid>
104 <description>What is the issue anyway?</description>
105 <content:encoded>&lt;h2 id=&#34;what-is-the-issue-anyway&#34;&gt;What is the issue anyway?&lt;/h2&gt;
106&lt;p&gt;Over the years, I have accumulated a bunch of virtual servers on my
107&lt;a href=&#34;https://www.digitalocean.com/&#34;&gt;DigitalOcean&lt;/a&gt; account for small experimental
108projects I dabble in. And this has resulted in quite a bill. I mean, I wouldn&#39;t
109care if these projects were actually being used. But there were just being there
110unused and wasting resources. Which makes this an unnecessary burden for me.&lt;/p&gt;
111&lt;p&gt;Most of them are just small HTML pages that have an endpoint or two to read data
112from or to, and for that reason I wrote servers left and right. To be honest,
113all of those things could have been done with &lt;a href=&#34;https://en.wikipedia.org/wiki/Common_Gateway_Interface&#34;&gt;CGI
114scripts&lt;/a&gt; and that would
115have been more than enough.&lt;/p&gt;
116&lt;p&gt;Recently, I decided to stop language hopping and focus on a simpler stack which
117includes C, Go and Lua. And I can accomplish all the things I am interested in.&lt;/p&gt;
118&lt;h2 id=&#34;finding-a-web-server-replacement&#34;&gt;Finding a web server replacement&lt;/h2&gt;
119&lt;p&gt;Usually I had &lt;a href=&#34;https://nginx.org/en/&#34;&gt;Nginx&lt;/a&gt; in front of these small web servers
120and I had to manage SSL certificates and all that jazz. I am bored with these
121things. I don&#39;t want to manage any of this bullshit anymore.&lt;/p&gt;
122&lt;p&gt;So the logical move forward was to find a solid alternative for this. I have
123ended up on &lt;a href=&#34;https://caddyserver.com/&#34;&gt;Caddy server&lt;/a&gt;. I&#39;ve used it in the past
124but kind of forgotten about it. What I really like about it is an ease of use
125and a bunch of out of the box functionalities that come with it.&lt;/p&gt;
126&lt;p&gt;These are the &lt;em&gt;pitch&lt;/em&gt; points from their website:&lt;/p&gt;
127&lt;ul&gt;
128&lt;li&gt;&lt;strong&gt;Secure by Default&lt;/strong&gt;: Caddy is the only web server that uses HTTPS by
129default. A hardened TLS stack with modern protocols preserves privacy and
130exposes MITM attacks.&lt;/li&gt;
131&lt;li&gt;&lt;strong&gt;Config API&lt;/strong&gt;: As its primary mode of configuration, Caddy&#39;s REST API makes
132it easy to automate and integrate with your apps.&lt;/li&gt;
133&lt;li&gt;&lt;strong&gt;No Dependencies&lt;/strong&gt;: Because Caddy is written in Go, its binaries are entirely
134self-contained and run on every platform, including containers without libc.&lt;/li&gt;
135&lt;li&gt;&lt;strong&gt;Modular Stack&lt;/strong&gt;: Take back control over your compute edge. Caddy can be
136extended with everything you need using plugins.&lt;/li&gt;
137&lt;/ul&gt;
138&lt;p&gt;I had just a few requirements:&lt;/p&gt;
139&lt;ul&gt;
140&lt;li&gt;Automatic SSL&lt;/li&gt;
141&lt;li&gt;Static file server&lt;/li&gt;
142&lt;li&gt;Basic authentication&lt;/li&gt;
143&lt;li&gt;CGI script support&lt;/li&gt;
144&lt;/ul&gt;
145&lt;p&gt;And the vanilla version does all of it, but CGI scripts. But that can easily be
146fixed with their modular approach. You can do this on their website and build a
147custom version of the server, or do it with Docker.&lt;/p&gt;
148&lt;p&gt;This is a &lt;code&gt;Dockerfile&lt;/code&gt; I used to build a custom server.&lt;/p&gt;
149&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;FROM&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; caddy:builder AS builder&lt;/span&gt;&lt;span style=&#34;&#34;&gt;
150&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;
151&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;RUN&lt;/span&gt; xcaddy build &lt;span style=&#34;color:#a31515&#34;&gt;\
152&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --with github.com/aksdb/caddy-cgi&lt;span style=&#34;&#34;&gt;
153&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;
154&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;FROM&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; caddy:latest&lt;/span&gt;&lt;span style=&#34;&#34;&gt;
155&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;RUN&lt;/span&gt; apk add --no-cache nano&lt;span style=&#34;&#34;&gt;
156&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;
157&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;COPY&lt;/span&gt; --from=builder /usr/bin/caddy /usr/bin/caddy&lt;span style=&#34;&#34;&gt;
158&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;getting-rid-of-all-the-unnecessary-virtual-machines&#34;&gt;Getting rid of all the unnecessary virtual machines&lt;/h2&gt;
159&lt;p&gt;The next step was to get a handle on the number of virtual servers I have all
160over the place.&lt;/p&gt;
161&lt;p&gt;I decided to move all the projects and services into two main VMs:&lt;/p&gt;
162&lt;ul&gt;
163&lt;li&gt;personal server (still Nginx)
164&lt;ul&gt;
165&lt;li&gt;git server&lt;/li&gt;
166&lt;li&gt;static file server&lt;/li&gt;
167&lt;li&gt;personal blog&lt;/li&gt;
168&lt;/ul&gt;
169&lt;/li&gt;
170&lt;li&gt;projects server (Caddy server)
171&lt;ul&gt;
172&lt;li&gt;personal experiments&lt;/li&gt;
173&lt;li&gt;other projects&lt;/li&gt;
174&lt;/ul&gt;
175&lt;/li&gt;
176&lt;/ul&gt;
177&lt;p&gt;I will focus on projects&#39; server in this post since it&#39;s more interesting.&lt;/p&gt;
178&lt;h2 id=&#34;testing-cgi-scripts&#34;&gt;Testing CGI scripts&lt;/h2&gt;
179&lt;p&gt;The first thing I tested was how CGI scripts work under Caddy. This is
180particularly import to me because almost all of my experiments and mini projects
181need this to work.&lt;/p&gt;
182&lt;p&gt;To configure Caddy server, you must provide the server with a configuration
183file. By default, it&#39;s called &lt;code&gt;Caaddyfile&lt;/code&gt;.&lt;/p&gt;
184&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
185&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;order&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;cgi&lt;/span&gt; before &lt;span style=&#34;color:#a31515&#34;&gt;respond&lt;/span&gt;
186&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
187&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
188&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;font-weight:bold&#34;&gt;examples.mitjafelicijan.com&lt;/span&gt; {
189&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /bash-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/bash-test.sh&lt;/span&gt;
190&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /tcl-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/tcl-test.tcl&lt;/span&gt;
191&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /lua-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/lua-test.lua&lt;/span&gt;
192&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /python-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/python-test.py&lt;/span&gt;
193&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
194&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;root&lt;/span&gt; * &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples&lt;/span&gt;
195&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;file_server&lt;/span&gt;
196&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
197&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;
198&lt;li&gt;The order is very important. Make sure that &lt;code&gt;order cgi before respond&lt;/code&gt; is at
199the top of the configuration file.&lt;/li&gt;
200&lt;li&gt;Also, when you run with Caddy v2, make sure you provide &lt;code&gt;adapter&lt;/code&gt; argument
201like this &lt;code&gt;/usr/bin/caddy run --watch --environ --config /etc/caddy/Caddyfile --adapter caddyfile&lt;/code&gt;. Otherwise, Caddy will try to use a different format for
202config file.&lt;/li&gt;
203&lt;/ul&gt;
204&lt;p&gt;I did a small batch of tests with &lt;a href=&#34;https://www.gnu.org/software/bash/&#34;&gt;Bash&lt;/a&gt;,
205&lt;a href=&#34;https://www.tcl-lang.org/&#34;&gt;Tcl&lt;/a&gt;, &lt;a href=&#34;https://www.lua.org/&#34;&gt;Lua&lt;/a&gt; and
206&lt;a href=&#34;https://www.python.org/&#34;&gt;Python&lt;/a&gt;. Here is a cheat sheet if you need it.&lt;/p&gt;
207&lt;p&gt;Let&#39;s get Bash out of the way first.&lt;/p&gt;
208&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#!/usr/bin/bash
209&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&lt;/span&gt;
210&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;printf &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Content-type: text/plain\n\n&amp;#34;&lt;/span&gt;
211&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
212&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;printf &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hello from Bash\n\n&amp;#34;&lt;/span&gt;
213&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;printf &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PATH_INFO [%s]\n&amp;#34;&lt;/span&gt; $PATH_INFO
214&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;printf &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;QUERY_STRING [%s]\n&amp;#34;&lt;/span&gt; $QUERY_STRING
215&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;printf &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;\n&amp;#34;&lt;/span&gt;
216&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
217&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i in {0..9..1}; &lt;span style=&#34;color:#00f&#34;&gt;do&lt;/span&gt;
218&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; printf &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;gt; %s\n&amp;#34;&lt;/span&gt; $i
219&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;done&lt;/span&gt;
220&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
221&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;exit 0
222&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This one is for Tcl script.&lt;/p&gt;
223&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;#!/usr/bin/tclsh
224&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;
225&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;puts &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Content-type: text/plain\n&amp;#34;&lt;/span&gt;
226&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
227&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;puts &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hello from Tcl\n&amp;#34;&lt;/span&gt;
228&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;puts &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PATH_INFO \[$env(PATH_INFO)\]&amp;#34;&lt;/span&gt;
229&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;puts &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;QUERY_STRING \[$env(QUERY_STRING)\]&amp;#34;&lt;/span&gt;
230&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;puts &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;#34;&lt;/span&gt;
231&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
232&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;{set&lt;/span&gt; i 0&lt;span style=&#34;color:#00f&#34;&gt;}&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;{&lt;/span&gt;$i &amp;lt; 10&lt;span style=&#34;color:#00f&#34;&gt;}&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;{&lt;/span&gt;incr i&lt;span style=&#34;color:#00f&#34;&gt;}&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;{&lt;/span&gt;
233&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; puts &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;gt; $i&amp;#34;&lt;/span&gt;
234&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;}&lt;/span&gt;
235&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And for all you Python enjoyers.&lt;/p&gt;
236&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;#!/usr/bin/python3&lt;/span&gt;
237&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
238&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; os
239&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
240&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Content-type: text/plain&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;)
241&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
242&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hello from Python&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;)
243&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PATH_INFO [&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;]&amp;#34;&lt;/span&gt;.format(os.environ[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;PATH_INFO&amp;#39;&lt;/span&gt;]))
244&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;QUERY_STRING [&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;]&amp;#34;&lt;/span&gt;.format(os.environ[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;QUERY_STRING&amp;#39;&lt;/span&gt;]))
245&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;#34;&lt;/span&gt;)
246&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
247&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; range(10):
248&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;gt; &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;.format(i))
249&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And for the final example, Lua.&lt;/p&gt;
250&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#!/usr/bin/lua&lt;/span&gt;
251&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
252&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Content-type: text/plain&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;)
253&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
254&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hello from Lua&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;)
255&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(string.format(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PATH_INFO [%s]&amp;#34;&lt;/span&gt;, os.getenv(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PATH_INFO&amp;#34;&lt;/span&gt;)))
256&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(string.format(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;QUERY_STRING [%s]&amp;#34;&lt;/span&gt;, os.getenv(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;QUERY_STRING&amp;#34;&lt;/span&gt;)))
257&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print()
258&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
259&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i = 0, 9 &lt;span style=&#34;color:#00f&#34;&gt;do&lt;/span&gt;
260&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(string.format(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;gt; %d&amp;#34;&lt;/span&gt;, i))
261&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
262&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;basic-authentication&#34;&gt;Basic authentication&lt;/h2&gt;
263&lt;p&gt;One thing was also to have an option for some sort of authentication, and
264something like &lt;a href=&#34;https://en.wikipedia.org/wiki/Basic_access_authentication&#34;&gt;Basic access
265authentication&lt;/a&gt; would
266be more than enough.&lt;/p&gt;
267&lt;p&gt;Thankfully, Caddy supports this out of the box already. Below is an updated
268example.&lt;/p&gt;
269&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
270&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;order&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;cgi&lt;/span&gt; before &lt;span style=&#34;color:#a31515&#34;&gt;respond&lt;/span&gt;
271&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
272&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
273&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;font-weight:bold&#34;&gt;examples.mitjafelicijan.com&lt;/span&gt; {
274&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /bash-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/bash-test.sh&lt;/span&gt;
275&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /tcl-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/tcl-test.tcl&lt;/span&gt;
276&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /lua-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/lua-test.lua&lt;/span&gt;
277&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;cgi&lt;/span&gt; /python-test &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples/python-test.py&lt;/span&gt;
278&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
279&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;root&lt;/span&gt; * &lt;span style=&#34;color:#a31515&#34;&gt;/opt/projects/examples&lt;/span&gt;
280&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;file_server&lt;/span&gt;
281&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
282&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;basicauth&lt;/span&gt; * {
283&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;bob&lt;/span&gt; &lt;span style=&#34;&#34;&gt;$&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;2a&lt;/span&gt;&lt;span style=&#34;&#34;&gt;$&lt;/span&gt;14&lt;span style=&#34;&#34;&gt;$&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;/wCgaf9oMnmQa20txB76u.nI1AldGMBT/1J7fXCfgOiRShwz/JOkK&lt;/span&gt;
284&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
285&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
286&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;code&gt;basicauth *&lt;/code&gt; matches everything under this domain/sub-domain and protects it
287with Basic Authentication.&lt;/p&gt;
288&lt;ul&gt;
289&lt;li&gt;&lt;code&gt;bob&lt;/code&gt; is the username&lt;/li&gt;
290&lt;li&gt;&lt;code&gt;hash&lt;/code&gt; is the password&lt;/li&gt;
291&lt;/ul&gt;
292&lt;p&gt;To generate these passwords, execute &lt;code&gt;caddy hash-password&lt;/code&gt; and this will prompt
293you to insert a password twice and spit out a hashed password that you can put
294in your configuration file.&lt;/p&gt;
295&lt;p&gt;Restart the server and you are ready to go.&lt;/p&gt;
296&lt;h2 id=&#34;making-caddy-a-service-with-systemd&#34;&gt;Making Caddy a service with systemd&lt;/h2&gt;
297&lt;p&gt;After the tests were successful, I copied &lt;code&gt;caddy&lt;/code&gt; to &lt;code&gt;/usr/bin/caddy&lt;/code&gt; and copied
298&lt;code&gt;Caddyfile&lt;/code&gt; to &lt;code&gt;/etc/caddy/Caddyfile&lt;/code&gt;.&lt;/p&gt;
299&lt;p&gt;Now off to the systemd. Each systemd service requires you to create a service
300file.&lt;/p&gt;
301&lt;ul&gt;
302&lt;li&gt;I created a &lt;code&gt;/etc/systemd/system/caddy.service&lt;/code&gt; and put the following content
303in the file.&lt;/li&gt;
304&lt;/ul&gt;
305&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;[Unit]&lt;/span&gt;
306&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Description=&lt;span style=&#34;color:#a31515&#34;&gt;Caddy&lt;/span&gt;
307&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Documentation=&lt;span style=&#34;color:#a31515&#34;&gt;https://caddyserver.com/docs/&lt;/span&gt;
308&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;After=&lt;span style=&#34;color:#a31515&#34;&gt;network.target network-online.target&lt;/span&gt;
309&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Requires=&lt;span style=&#34;color:#a31515&#34;&gt;network-online.target&lt;/span&gt;
310&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
311&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;[Service]&lt;/span&gt;
312&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Type=&lt;span style=&#34;color:#a31515&#34;&gt;notify&lt;/span&gt;
313&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;User=&lt;span style=&#34;color:#a31515&#34;&gt;root&lt;/span&gt;
314&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Group=&lt;span style=&#34;color:#a31515&#34;&gt;root&lt;/span&gt;
315&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ExecStart=&lt;span style=&#34;color:#a31515&#34;&gt;/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile --adapter caddyfile&lt;/span&gt;
316&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ExecReload=&lt;span style=&#34;color:#a31515&#34;&gt;/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force --adapter caddyfile&lt;/span&gt;
317&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;TimeoutStopSec=&lt;span style=&#34;color:#a31515&#34;&gt;5s&lt;/span&gt;
318&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;LimitNOFILE=&lt;span style=&#34;color:#a31515&#34;&gt;1048576&lt;/span&gt;
319&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;LimitNPROC=&lt;span style=&#34;color:#a31515&#34;&gt;512&lt;/span&gt;
320&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;PrivateTmp=&lt;span style=&#34;color:#a31515&#34;&gt;true&lt;/span&gt;
321&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ProtectSystem=&lt;span style=&#34;color:#a31515&#34;&gt;full&lt;/span&gt;
322&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;AmbientCapabilities=&lt;span style=&#34;color:#a31515&#34;&gt;CAP_NET_ADMIN CAP_NET_BIND_SERVICE&lt;/span&gt;
323&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
324&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;[Install]&lt;/span&gt;
325&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;WantedBy=&lt;span style=&#34;color:#a31515&#34;&gt;multi-user.target&lt;/span&gt;
326&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;
327&lt;li&gt;You might need to reload systemd with &lt;code&gt;systemctl daemon-reload&lt;/code&gt;.&lt;/li&gt;
328&lt;li&gt;Then I enabled the service with &lt;code&gt;systemctl enable caddy.service&lt;/code&gt;.&lt;/li&gt;
329&lt;li&gt;And then I started the service with &lt;code&gt;systemctl start caddy.service&lt;/code&gt;.&lt;/li&gt;
330&lt;/ul&gt;
331&lt;p&gt;This was about all that I needed to do to get it running. Now I can easily add
332new subdomains and domains to the main configuration file and be done with
333it. No manual Let&#39;s Encrypt shenanigans needed.&lt;/p&gt;
334</content:encoded>
335 </item>
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355 <item>
356 <title>Re-Inventing Task Runner That I Actually Used Daily</title>
357 <link>https://mitjafelicijan.com/re-inventing-task-runner-that-i-actually-used-daily.html</link>
358 <pubDate>Wed, 31 May 2023 12:21:10 &#43;0200</pubDate>
359 <guid>https://mitjafelicijan.com/re-inventing-task-runner-that-i-actually-used-daily.html</guid>
360 <description>Couple of months ago I had this brilliant idea of re-inventing the wheel bymaking an alternative for make.</description>
361 <content:encoded>&lt;p&gt;Couple of months ago I had this brilliant idea of re-inventing the wheel by
362making an alternative for make. And so I went. Boldly into the battle. And to my
363big surprise my attempt resulted in not a completely useless piece of software.&lt;/p&gt;
364&lt;p&gt;My initial requirements were quite simple but soon grow into something more
365ambitious. And looking back I should have stuck to the simple version. My
366laziness was on my side this time though. Because I haven’t implemented some of
367the features I now realise I really didn’t need them and they would bog the
368whole program and make it be something it was never meant to be.&lt;/p&gt;
369&lt;p&gt;My basic requirements were following:&lt;/p&gt;
370&lt;ul&gt;
371&lt;li&gt;Syntax should be a tiny bit inspired by Rake and Rakefiles.&lt;/li&gt;
372&lt;li&gt;Should borrow the overall feel of a unit test experience.&lt;/li&gt;
373&lt;li&gt;Using something like Python would be a bit of an overkill.&lt;/li&gt;
374&lt;li&gt;The program must be statically compiled, so it can run on same architecture
375without libc, musl dependencies or things like that.&lt;/li&gt;
376&lt;li&gt;Install ruby for rake is a bit overkill and can not be done with certain
377really lightweight distributions like Alpine Linux. This tool would be usable
378on such lightweight systems for remote debugging.&lt;/li&gt;
379&lt;li&gt;I want to use it for more than just compiling things. I want to use it as an
380entry-point into a project, and I want this to help me indirectly document the
381project as well.&lt;/li&gt;
382&lt;li&gt;It should be an abstraction over bash shell or the default system shell.
383&lt;ul&gt;
384&lt;li&gt;Each task essentially becomes its own shell instance.&lt;/li&gt;
385&lt;/ul&gt;
386&lt;/li&gt;
387&lt;li&gt;Must work on Linux and macOS systems.&lt;/li&gt;
388&lt;li&gt;By default, running &lt;code&gt;erd&lt;/code&gt; list all the available tasks (when I use make, I
389usually put a disclaimer that you should check Makefile to see all available
390target).&lt;/li&gt;
391&lt;li&gt;Should support passing arguments when you run it from a shell.&lt;/li&gt;
392&lt;li&gt;Normal variable as the same as environmental variables. There is no
393distinction. Every variable is also essentially an environment variable and
394can be used by other programs.&lt;/li&gt;
395&lt;li&gt;State between tasks is not shared, and this makes this “pure” shell instances.&lt;/li&gt;
396&lt;li&gt;Should be single-threaded for the start and later expanded with &lt;code&gt;@spawn&lt;/code&gt;
397command.&lt;/li&gt;
398&lt;li&gt;Variables behave like macros and are preprocessed before evaluation.&lt;/li&gt;
399&lt;li&gt;Should support something like &lt;code&gt;assure&lt;/code&gt; that would check if programs like C
400compiler or Python (whatever the project requires) are installed on a machine.&lt;/li&gt;
401&lt;/ul&gt;
402&lt;p&gt;Quite a reasonable list of requirements. I do this things already in my
403Makefiles or/and Bash scripts. But I would like to avoid repeating myself every
404time I start working on something new.&lt;/p&gt;
405&lt;p&gt;So I started with the following syntax.&lt;/p&gt;
406&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@env on
407&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
408&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# Override the default shell.&lt;/span&gt;
409&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@shell &lt;span style=&#34;color:#a31515&#34;&gt;/bin/&lt;/span&gt;bash
410&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
411&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# Assure that program is installed.&lt;/span&gt;
412&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@assure docker-compose pip python3
413&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
414&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# Load local dotenv files (these are then globally available).&lt;/span&gt;
415&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@dotenv .env
416&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@dotenv .env.sample
417&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@dotenv some_other_file
418&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
419&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# This are local variables but still accessible in tasks.&lt;/span&gt;
420&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@var HI = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;hey&amp;#34;&lt;/span&gt;
421&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@var TOKEN = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sometoken&amp;#34;&lt;/span&gt;
422&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@var EMAIL = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;m@m.com&amp;#34;&lt;/span&gt;
423&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@var PASSWORD = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;pass&amp;#34;&lt;/span&gt;
424&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@var EDITOR = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;vim&amp;#34;&lt;/span&gt;
425&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
426&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@task dev &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Test chars .:&amp;#39;}{]!//&amp;#34;&lt;/span&gt; does
427&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; echo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;...&amp;#34;&lt;/span&gt; $HI
428&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
429&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
430&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@task clean &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Cleans the obj files&amp;#34;&lt;/span&gt; does
431&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; rm .obj
432&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
433&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
434&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@task greet &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Greets the user&amp;#34;&lt;/span&gt; does
435&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; echo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hi user $TOKEN or $WINDOWID $EMAIL&amp;#34;&lt;/span&gt;
436&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
437&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
438&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@task stack &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Starts Docker stack&amp;#34;&lt;/span&gt; does
439&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; docker-compose -f stack.yml up
440&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
441&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
442&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@task todo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Shows all todos in source files and count them&amp;#34;&lt;/span&gt; does
443&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; grep -ir &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;TODO|FIXME&amp;#34;&lt;/span&gt; . | wc -l
444&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
445&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
446&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@task test1 &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;For testing 1&amp;#34;&lt;/span&gt; does
447&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; unknown-command
448&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; echo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;test1&amp;#34;&lt;/span&gt;
449&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ls -lha
450&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
451&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
452&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@task test2 &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;For testing 2&amp;#34;&lt;/span&gt; does
453&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; echo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;test1&amp;#34;&lt;/span&gt;
454&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ls -lha
455&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; docker-compose -f samples/stack.yml up
456&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;end&lt;/span&gt;
457&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;One thing that I really like about Errand. Yes, this is what it is called. And
458it is available at &lt;a href=&#34;https://git.mitjafelicijan.com/errand.git/about/&#34;&gt;https://git.mitjafelicijan.com/errand.git/about/&lt;/a&gt;. Moving
459on. One thing that I really like is that a task is a persistent shell. By that I
460mean, that the whole task, even if it contains multiple command in one shell.
461In make each line in a target is that and you need to combine lines or add &lt;code&gt;\&lt;/code&gt;
462at the end of the line.&lt;/p&gt;
463&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# How you do this things in make.&lt;/span&gt;
464&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;target:
465&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; source .venv/bin/activate &lt;span style=&#34;color:#a31515&#34;&gt;\
466&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; python script.py
467&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This solves this problem. Consider each task and what is being executed in that
468task a shell that will only close when all the tasks are completed.&lt;/p&gt;
469&lt;p&gt;By self-documenting I mean that if you are in a directory with &lt;code&gt;Errandfile&lt;/code&gt; in,
470if you only type &lt;code&gt;erd&lt;/code&gt; and press enter it should by default display all the
471possible targets. In make i was doing this by having a first target be something
472like &lt;code&gt;default&lt;/code&gt; that echos the message “Check Makefile for all available target.”
473Because all of the tasks in Errand require a message I use that to display let’s
474call it table of contents.&lt;/p&gt;
475&lt;p&gt;Because I don’t use any external dependencies this whole thing can be statically
476compiled. So that also checked one of the boxes.&lt;/p&gt;
477&lt;p&gt;It works on Linux and on a Mac so that’s also a bonus. I don’t believe this
478would work on Windows machines because of the way that I use shell instances. By
479you could use something like Windows Subsystem for Linux and run it in
480there. That is a valid option.&lt;/p&gt;
481&lt;p&gt;To finish this essay off, how was it to use it in “real life”. I have to be
482honest. Some of the missing features still bother me. &lt;code&gt;@dotenv&lt;/code&gt; directive is
483still missing and I need to implement this ASAP.&lt;/p&gt;
484&lt;p&gt;Another thing that needs to happen is support for streaming output. Currently
485commands like &lt;code&gt;docker-compose&lt;/code&gt; that runs in foreground mode is not compatible
486with Errand. So commands that stream output are an issue. I need to revisit how
487I initiate shell and how I read stdout and stderr. But that shouldn’t be a
488problem.&lt;/p&gt;
489&lt;p&gt;I have been very satisfied with this thing. I am pleasantly surprised by how
490useful it is. I really wanted to test this in the wild before I commit to it. I
491have more abandoned project than Google and it’s bringing a massive shame to my
492family at this point. So I wanted to be sure that this is even useful. And it
493actually is. Quite surprised at myself.&lt;/p&gt;
494&lt;p&gt;I really need to package this now and write proper docs. And maybe rewrite
495tokeniser. Its atrocious right now. Site to behold! But that is an issue for
496another time.&lt;/p&gt;
497</content:encoded>
498 </item>
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522 <item>
523 <title>I think I was completely wrong about Git workflows</title>
524 <link>https://mitjafelicijan.com/i-was-wrong-about-git-workflows.html</link>
525 <pubDate>Tue, 23 May 2023 12:00:00 &#43;0200</pubDate>
526 <guid>https://mitjafelicijan.com/i-was-wrong-about-git-workflows.html</guid>
527 <description>I have been using some approximation of GitFlow for years now and never reallyquestioned it to be honest.</description>
528 <content:encoded>&lt;p&gt;I have been using some approximation of &lt;a href=&#34;https://jeffkreeftmeijer.com/git-flow/&#34;&gt;Git
529Flow&lt;/a&gt; for years now and never really
530questioned it to be honest. When I create a repo I create develop branch and set
531it as default one and then merge to master from there. Seems reasonable enough.&lt;/p&gt;
532&lt;p&gt;One thing that I have learned is that long living branches are the devil. They
533always end up making a huge mess when they need to be merged eventually into
534master. So by that reason, what is the develop branch if not the longest living
535feature branch. And from my personal experience there was never a situation
536where I wasn’t sweating bullets when I had to merge develop back to master.&lt;/p&gt;
537&lt;p&gt;This realisation started to give me pause. So why the hell am I doing this, and
538is there a better way. Well the solution was always there. And it comes in a
539form of &lt;a href=&#34;https://git-scm.com/book/en/v2/Git-Basics-Tagging&#34;&gt;git tags&lt;/a&gt;.&lt;/p&gt;
540&lt;p&gt;So what are git tags? Git tags are references to specific points in a Git
541repository&#39;s history. They are used to mark important milestones, such as
542releases or significant commits, making it easier to identify and access
543specific versions of a project.&lt;/p&gt;
544&lt;p&gt;Somehow we have all hijacked the meaning of the master branch that it has to be
545the most releasable version of code. And this is also where the confusing about
546versioning the software kicks in. Because master branch implicitly says that we
547are dealing with the rolling release type of a software. And by having a develop
548branch we are hacking around this confusion. With a separation of develop and
549master we lock functionalities into place and forcing a stable vs development
550version of the software.&lt;/p&gt;
551&lt;p&gt;But if that is true and the long living branches are the devil then why have
552develop at all. I think that most of this comes to how continuous integration is
553being done. There usually is no granular access to tags and CD software deploys
554what is present on a specific branch, may that be master for production and
555develop for staging. This is a gross simplification and by having this in place
556we have completely removed tagging as a viable option to create a fix point in
557software cycle that says, this is the production ready code.&lt;/p&gt;
558&lt;p&gt;One cool thing about tags are that you can checkout a specific tag. So they
559behave very similarly as branches in that regard. And you don’t have the
560overhead of having two mainstream branches.&lt;/p&gt;
561&lt;p&gt;So what is the solution? One approach is to use development workflow, where all
562changes are made on the smaller branches and continuously merged into
563master. Where the software is ready to be pushed to production you tag the
564master branch. This approach eliminates the need for long-lived branches and
565simplifies the development process. It also encourages developers to make small,
566incremental changes that can be tested and deployed quickly. However, this
567approach may not be suitable for all projects or teams that heavily rely on
568automated deployment based on branch names only.&lt;/p&gt;
569&lt;p&gt;This also requires that developers always keep production in mind. No more
570living on an island of the develop branch. All your actions and code need to be
571ready to meet production standards on a much smaller timescale.&lt;/p&gt;
572&lt;p&gt;I think that we have complicated the workflow in an honest attempt to make
573things more streamlined but in the process of doing this, we have inadvertently
574made our lives much more complicated.&lt;/p&gt;
575&lt;p&gt;In conclusion, it&#39;s important to re-evaluate our workflows from time to time to
576see if they still make sense and if there are better alternatives available.
577Long-living branches can be problematic, and using tags to mark important
578milestones can simplify the development process.&lt;/p&gt;
579</content:encoded>
580 </item>
581
582
583
584
585
586
587
588 <item>
589 <title>From General Zod to Superman - Crafting Stories in Zed Editor</title>
590 <link>https://mitjafelicijan.com/crafting-stories-in-zed-editor.html</link>
591 <pubDate>Mon, 22 May 2023 12:00:00 &#43;0200</pubDate>
592 <guid>https://mitjafelicijan.com/crafting-stories-in-zed-editor.html</guid>
593 <description>Pretentious title!</description>
594 <content:encoded>&lt;p&gt;Pretentious title! Good start! I have nothing to add to this discussion. I just
595like this editor and wanted to write something here that will remind me to use
596it again in a while when/if it becomes available for Linux.&lt;/p&gt;
597&lt;p&gt;&lt;strong&gt;TLDR:&lt;/strong&gt; I think this code editor is very cool and has a massive potential. I
598hope they don’t mess up with adding a plugin ecosystem to it!&lt;/p&gt;
599&lt;p&gt;Out of morbid curiosity, I started using the &lt;a href=&#34;https://zed.dev/&#34;&gt;Zed editor&lt;/a&gt; on
600my Mac. Zed is a high-performance, multiplayer code editor developed by the
601creators of Atom and Tree-sitter. Written in Rust so it has to be blazingly
602fast! 😊 It&#39;s a joke, calm down.&lt;/p&gt;
603&lt;p&gt;Over the past year, I have switched between &lt;a href=&#34;https://helix-editor.com/&#34;&gt;Helix
604editor&lt;/a&gt; and &lt;a href=&#34;https://code.visualstudio.com/&#34;&gt;VS
605Code&lt;/a&gt;, but for the last couple of months, I have
606been using Helix exclusively.&lt;/p&gt;
607&lt;p&gt;I&#39;ve been genuinely impressed by Zed. When you open a file, it automatically
608detects its type and downloads the corresponding &lt;a href=&#34;https://en.wikipedia.org/wiki/Language_Server_Protocol&#34;&gt;LSP (language
609server)&lt;/a&gt;. The list of
610supported languages is not extensive, but it&#39;s still impressive. It&#39;s a great
611example of how to create a product that stays out of your way.&lt;/p&gt;
612&lt;p&gt;&lt;img src=&#34;/assets/zed/zed-1.png?style=bigimg&#34; alt=&#34;Zed editor&#34; /&gt;&lt;/p&gt;
613&lt;p&gt;For C development it downloaded &lt;a href=&#34;https://clangd.llvm.org/&#34;&gt;clangd&lt;/a&gt; and setting
614up missing dependencies in code was rather easy. For this project I use
615&lt;a href=&#34;https://www.libsdl.org/&#34;&gt;SDL2&lt;/a&gt; for rendering terminal emulator. It’s a hobby
616project, don’t worry about it.&lt;/p&gt;
617&lt;p&gt;If you are going to give this a try and you are using C, I suggest checking two
618files in the root of your project folder. If you don&#39;t have them, create them.&lt;/p&gt;
619&lt;p&gt;&lt;strong&gt;compile_flags.txt&lt;/strong&gt;&lt;/p&gt;
620&lt;pre&gt;&lt;code&gt;-I/opt/homebrew/include
621-I/opt/homebrew/include/SDL2
622&lt;/code&gt;&lt;/pre&gt;
623&lt;p&gt;Easy way of checking what the appropriate includes for a specific library is to
624use &lt;code&gt;pkg-config&lt;/code&gt; and in my case &lt;code&gt;pkg-config SDL2 --cflags-only-I&lt;/code&gt;. But this is
625nothing new to C/C&#43;&#43; devs. Just a noter for people who are using Visual Studio.&lt;/p&gt;
626&lt;p&gt;&lt;strong&gt;.clang-format&lt;/strong&gt;&lt;/p&gt;
627&lt;pre&gt;&lt;code&gt;ColumnLimit: 220
628BasedOnStyle: Mozilla
629&lt;/code&gt;&lt;/pre&gt;
630&lt;p&gt;I prefer Mozilla coding style for C so you can set that up.&lt;/p&gt;
631&lt;p&gt;They really have something special here. Although there is no version available
632for Linux yet, I will stick to Helix. This impressive piece of engineering is,
633above all, an amazing example of craftsmanship.&lt;/p&gt;
634&lt;p&gt;They have a bunch of amazing integrated functionalities like live desktop
635sharing, code sharing in a live coding session. There is a lot of pretentious
636marketing speak there but the product is still amazing!&lt;/p&gt;
637&lt;p&gt;For me the speed and the simplicity of the product was the most impressive
638thing. You get that: it just works feeling. A rare thing in 2023.&lt;/p&gt;
639&lt;p&gt;&lt;img src=&#34;/assets/zed/zed-2.png?style=bigimg&#34; alt=&#34;Zed editor&#34; /&gt;&lt;/p&gt;
640&lt;p&gt;They also managed to add &lt;a href=&#34;https://github.com/features/copilot&#34;&gt;Github Copilot&lt;/a&gt;
641in a non obtrusive way. To me, everything feels very intentional and
642specifically selected. It&#39;s minimal yet maximally effective.&lt;/p&gt;
643&lt;p&gt;&lt;video src=&#34;https://zed.dev/img/post/copilot/copilot-demo.webm&#34; autoplay loop&gt;&lt;/video&gt;&lt;/p&gt;
644&lt;p&gt;It is a perfect balance between VS Code, Jetbrains IDE’s and something like VIM
645or Helix.&lt;/p&gt;
646&lt;p&gt;I just hope they &lt;strong&gt;DON’T&lt;/strong&gt; add plugin support and keep it like it is. They as a
647vendor should add stuff to it with great deliberation and thought. And this way
648the product will stay fast and focused. That’s my two cents.&lt;/p&gt;
649&lt;p&gt;Amazing job!&lt;/p&gt;
650</content:encoded>
651 </item>
652
653
654
655
656
657
658
659 <item>
660 <title>Rekindling my love for programming and enjoying the act of creating</title>
661 <link>https://mitjafelicijan.com/rekindling-my-love-for-programming.html</link>
662 <pubDate>Tue, 16 May 2023 12:00:00 &#43;0200</pubDate>
663 <guid>https://mitjafelicijan.com/rekindling-my-love-for-programming.html</guid>
664 <description>Programming can be a challenging and rewarding experience, but sometimes it&amp;#39;seasy to feel burnt out or disinterested.</description>
665 <content:encoded>&lt;p&gt;Programming can be a challenging and rewarding experience, but sometimes it&#39;s
666easy to feel burnt out or disinterested. I have lost the passion for coding over
667the past couple of months and it looked like I will never enjoy the coding as
668much as I did.&lt;/p&gt;
669&lt;p&gt;I was feeling burnt out with programming. I thought taking a break from it and
670focusing on other activities that I enjoy might be helpful. This way, I could
671come back to programming with a fresh perspective and renewed energy. I also
672thought about learning a new programming language or technology to keep things
673interesting and challenging.&lt;/p&gt;
674&lt;p&gt;However, what I didn&#39;t realize was that learning a new language or technology
675wasn&#39;t going to solve the underlying issue. I needed to take a step back and
676re-evaluate why I had lost my passion for programming in the first place. This
677involved taking a deep look into what I was doing that resulted in this rut.&lt;/p&gt;
678&lt;p&gt;Sometimes, it&#39;s easy to get caught up in the hype of new technologies or
679languages, and we can feel like we&#39;re missing out if we&#39;re not constantly
680learning and experimenting. However, it&#39;s important to remember that the latest
681and greatest isn&#39;t always the best fit for our projects or our
682interests. Instead of constantly chasing the next big thing, it can be helpful
683to focus on what truly interests us and what we&#39;re passionate about. This can
684help us stay motivated and engaged with our work, rather than feeling like we&#39;re
685just going through the motions.&lt;/p&gt;
686&lt;p&gt;I expressed that I had lost my passion for coding over the past couple of
687months, and I realized that the reason behind it was my tendency to spread
688myself too thin and not focus on completing interesting projects. In order to
689regain my passion for coding, I need to focus on projects that truly interest me
690and give me a sense of purpose and motivation.&lt;/p&gt;
691&lt;p&gt;Recently, I have been playing World of Warcraft more frequently and have become
692interested in developing addons for the game.&lt;/p&gt;
693&lt;p&gt;This quickly resulted in me creating three addons that improve the quality of
694life, and I subsequently developed a more useful add-on that encapsulates all
695the others I made.&lt;/p&gt;
696&lt;p&gt;I found it interesting that this action sparked a new interest in me.
697Additionally, I discovered the Lua language, which reminded me that coding
698should be fun rather than just a struggle with a language. It should be pure,
699unadulterated fun.&lt;/p&gt;
700&lt;p&gt;I wasn&#39;t fighting the syntax, nor was I focused on finding the most optimal
701solution. I simply created things without the pressure of making them the best
702they could possibly be.&lt;/p&gt;
703&lt;p&gt;This made me realize that I actually adore simple languages that get out of the
704way and let you express what you want to do. It forced me to rethink a lot about
705what I use and what I actually enjoy.&lt;/p&gt;
706&lt;p&gt;I have decided to stick to the basics. For a scripting language, I will use
707Lua. For networking, I will use Golang. And for any special needs, I will rely
708on C. I do not require Rust, Nim, or Zig. This selection is more than sufficient
709for my needs. I have to stay true to this simplicity. There is something to the
710Occam&#39;s Razor.&lt;/p&gt;
711&lt;p&gt;I&#39;ve been struggling with a lack of creativity lately, but now I&#39;m experiencing
712a real change. I realized I needed to take a step back and stop actively trying
713to address the issue. I needed to stop worrying and overthinking it. I simply
714needed some time. Looking back, I don&#39;t think I&#39;ve taken any significant time
715off in the last 10 years.&lt;/p&gt;
716&lt;p&gt;Suddenly, I find myself with the energy and passion to complete multiple small
717projects. It doesn&#39;t feel like a chore at all. Who knew I needed WoW to
718kickstart everything. Inspiration really does come from the strangest places.&lt;/p&gt;
719</content:encoded>
720 </item>
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748 <item>
749 <title>Trying to build a New kind of terminal emulator for the modern age</title>
750 <link>https://mitjafelicijan.com/trying-to-build-a-new-kind-of-terminal-emulator.html</link>
751 <pubDate>Thu, 26 Jan 2023 12:00:00 &#43;0200</pubDate>
752 <guid>https://mitjafelicijan.com/trying-to-build-a-new-kind-of-terminal-emulator.html</guid>
753 <description>Over the past few weeks, I have been really thinking about terminal emulators,how we interact with computers, the separation of text-based programs and GUIones.</description>
754 <content:encoded>&lt;p&gt;Over the past few weeks, I have been really thinking about terminal emulators,
755how we interact with computers, the separation of text-based programs and GUI
756ones. To be perfectly honest, I got pissed off one evening when I was cleaning
757up files on my computer. Normally, I go into console and do &lt;code&gt;ncdu&lt;/code&gt; and check
758where the junk is. Then I start deleting stuff. Without any discrimination,
759usually. But when it comes to screenshots, I have learned that it&#39;s good to keep
760them somewhere near if I need to refer to something that I was doing. I am an
761avid screenshot taker. So at that point I checked Pictures folder and also did a
762basic search &lt;code&gt;find . -type f -name &amp;quot;*.jpg&amp;quot;&lt;/code&gt; for all the JPEG files in my home
763directory and immediately got pissed off. Why can’t I see thumbnails in my
764terminal? I know why, but why in the year of 2022 this is still a problem. I am
765used to traversing my disk via terminal. I am faster, and I am more comfortable
766this way. But when it comes to visualization, I then need to revert to GUI
767applications and again find the same file to see it. I know that programs like
768&lt;code&gt;feh&lt;/code&gt; and &lt;code&gt;sxiv&lt;/code&gt; are available, but I would just like to see the preview. Like
769&lt;a href=&#34;https://jupyter.org/&#34;&gt;Jupyter notebook&lt;/a&gt; or something similar. Just having it
770inline. Part of a result.&lt;/p&gt;
771&lt;p&gt;It also didn’t help that I was spending some time with the &lt;a href=&#34;https://plan9.io/plan9/&#34;&gt;Plan
7729&lt;/a&gt; Operating system. More specifically
773&lt;a href=&#34;http://9front.org/&#34;&gt;9FRONT&lt;/a&gt;. The way that &lt;a href=&#34;http://acme.cat-v.org/&#34;&gt;ACME editor&lt;/a&gt;
774handles text editing is just wonderful. Different and fresh somehow, even though
775it’s super old.&lt;/p&gt;
776&lt;p&gt;So, I went on a lookout for an interesting way of visualizing results of some
777query. I found these applications to be outstanding examples of how not to be a
778captive of a predetermined way of doing things.&lt;/p&gt;
779&lt;ul&gt;
780&lt;li&gt;&lt;a href=&#34;https://www.wolfram.com/mathematica/&#34;&gt;Wolfram Mathematica&lt;/a&gt;&lt;/li&gt;
781&lt;li&gt;&lt;a href=&#34;https://jupyter.org/&#34;&gt;Jupyter notebooks&lt;/a&gt;&lt;/li&gt;
782&lt;li&gt;&lt;a href=&#34;http://www.9front.org&#34;&gt;Plan 9 / 9FRONT&lt;/a&gt;&lt;/li&gt;
783&lt;li&gt;&lt;a href=&#34;https://templeos.org/&#34;&gt;Temple OS&lt;/a&gt;&lt;/li&gt;
784&lt;li&gt;&lt;a href=&#34;https://www.gnu.org/software/emacs/&#34;&gt;Emacs&lt;/a&gt;&lt;/li&gt;
785&lt;/ul&gt;
786&lt;p&gt;My idea is not as out there as ACME is, but it is a spin on the terminal
787emulators. I like the modes that Vi/Vim provides you with. I like the way the
788Emacs does its own &lt;code&gt;M-x&lt;/code&gt; &lt;code&gt;M-c&lt;/code&gt;. Furthermore, I really like how Mathematica and
789Jupyter present the data in a free flowing form. And I love how Temple OS is
790basically a C interpreter on some level.&lt;/p&gt;
791&lt;blockquote&gt;
792&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; This is part 1 of the journey. Nowhere finished yet. I am just
793tinkering with this at the moment. This whole thing can easily spectacularly
794fail.&lt;/p&gt;
795&lt;/blockquote&gt;
796&lt;p&gt;So I started. I knew that I wanted to have the couple of modes, but I didn’t
797like the repetition of keystrokes, so the only option was to have some sort of
798toggle and indicate to the user that they are in a special mode. Like Vi does
799for Normal and Visual mode.&lt;/p&gt;
800&lt;p&gt;These modes would for the first version be:&lt;/p&gt;
801&lt;ul&gt;
802&lt;li&gt;&lt;em&gt;Preview mode&lt;/em&gt; (toggle with Ctrl &#43; P)
803&lt;ul&gt;
804&lt;li&gt;When this mode would be enabled, the &lt;code&gt;ls&lt;/code&gt; command would try to find images
805from the results and display thumbnails from them in the terminal itself.
806No ASCII art. Proper images. In a grid!&lt;/li&gt;
807&lt;/ul&gt;
808&lt;/li&gt;
809&lt;li&gt;&lt;em&gt;Detach mode&lt;/em&gt; (toggle with Ctrl &#43; D)
810&lt;ul&gt;
811&lt;li&gt;When this mode would be enabled, every command would open a new window
812and execute that command in it. This would be useful for starting &lt;code&gt;htop&lt;/code&gt;
813in a separate window.&lt;/li&gt;
814&lt;/ul&gt;
815&lt;/li&gt;
816&lt;/ul&gt;
817&lt;p&gt;The reason for having these modes togglable is to not ask for previews every
818time. You enable a mode and until you disable it, it behaves that way. Purely
819out of ergonomic reasons.&lt;/p&gt;
820&lt;p&gt;I would like to treat every terminal I open as a session mentally. When I start
821using the terminal, I start digging deeper into the issue I am trying to
822resolve. And while I am doing this, I would like to open detached windows
823etc. A lot of these things can be done easily with something like
824&lt;a href=&#34;https://i3wm.org/&#34;&gt;i3&lt;/a&gt;, but also that pull you out of the context of what you
825were doing. I would like to orchestrate everything from one single point.&lt;/p&gt;
826&lt;p&gt;In planning for this project, I knew that I would need to use a language like C
827and a library such as &lt;a href=&#34;https://www.libsdl.org/&#34;&gt;SDL2&lt;/a&gt; in order to achieve the
828desired results. I had considered other options, but ultimately determined that
829&lt;a href=&#34;https://www.libsdl.org/&#34;&gt;SDL2&lt;/a&gt; was the best fit based on its capabilities and
830reputation in the programming community.&lt;/p&gt;
831&lt;p&gt;At first, I thought the idea of a hardware accelerated terminal was a bit of a
832joke. It seemed like such a niche and unnecessary feature, especially given the
833fact that terminal emulators have been around for decades and have always relied
834on software rendering. But to be fair, &lt;a href=&#34;https://alacritty.org/&#34;&gt;Alacritty&lt;/a&gt; is
835doing the same thing. Well, they are doing a remarkable job at it.&lt;/p&gt;
836&lt;p&gt;So, I embarked on a journey. Everything has to start somewhere. For me, it
837started with creating a window! It has to start somewhere. 🙂&lt;/p&gt;
838&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// Oh, Hi Mark!
839&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// Create the window, obviously.
840&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;SDL_Window *window = SDL_CreateWindow(
841&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; WINDOW_TITLE, SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED,
842&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; WINDOW_WIDTH, WINDOW_HEIGHT,
843&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; SDL_WINDOW_RESIZABLE | SDL_WINDOW_OPENGL | SDL_WINDOW_SHOWN);
844&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I continued like this to get some text displayed on the screen.&lt;/p&gt;
845&lt;p&gt;I noted that
846&lt;a href=&#34;https://wiki.libsdl.org/SDL_ttf/TTF_RenderText_Solid&#34;&gt;&lt;code&gt;TTF_RenderText_Solid&lt;/code&gt;&lt;/a&gt;
847rendered text really poorly. There were no antialiasing at all. In my wisdom, I
848never checked the documentation. Well, that was a fail. To uneducated like me:
849&lt;code&gt;TTF_RenderText_Solid&lt;/code&gt; renders Latin1 text at fast quality to a new 8-bit
850surface. So, that&#39;s why the texts looked like shit. No wonder.&lt;/p&gt;
851&lt;p&gt;Remarks on &lt;code&gt;TTF_RenderText_Solid&lt;/code&gt;: This function will allocate a new 8-bit,
852palettized surface. The surface&#39;s 0 pixel will be the colorkey, giving a
853transparent background. The 1 pixel will be set to the text color.&lt;/p&gt;
854&lt;p&gt;After I replaced it with
855&lt;a href=&#34;https://wiki.libsdl.org/SDL_ttf/TTF_RenderText_LCD&#34;&gt;&lt;code&gt;TTF_RenderText_LCD&lt;/code&gt;&lt;/a&gt; which
856renders Latin1 text at LCD subpixel quality to a new ARGB surface, the text
857started looking good. Really make sure you read the documentation. It’s actually
858good. As a side note, you can find all the documentation regarding &lt;a href=&#34;https://wiki.libsdl.org/&#34;&gt;SDL2 on
859their Wiki&lt;/a&gt;.&lt;/p&gt;
860&lt;p&gt;After that was done, I started working on displaying other things like &lt;code&gt;Preview&lt;/code&gt;
861and &lt;code&gt;Detach&lt;/code&gt; modes. This wasn’t really that hard. In SDL2 you can check all the
862available events with &lt;code&gt;while (SDL_PollEvent(&amp;amp;event) &amp;gt; 0)&lt;/code&gt; and have a bunch of
863switch statements to determine which key is currently being pressed. More about
864keys, &lt;a href=&#34;https://documentation.help/SDL/sdlkey.html&#34;&gt;SDLKey&lt;/a&gt; and mroe about
865pooling the events on
866&lt;a href=&#34;https://documentation.help/SDL/sdlpollevent.html&#34;&gt;SDL_PollEvent&lt;/a&gt;.&lt;/p&gt;
867&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (SDL_PollEvent(&amp;amp;event) &amp;gt; 0)
868&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
869&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;switch&lt;/span&gt; (event.type)
870&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
871&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;case&lt;/span&gt; SDL_QUIT:
872&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; running = false;
873&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;break&lt;/span&gt;;
874&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
875&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;case&lt;/span&gt; SDL_TEXTINPUT:
876&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (!meta_key_pressed)
877&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
878&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; strncat(input_prompt_text, event.text.text, 1);
879&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; update_input_prompt = true;
880&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
881&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;break&lt;/span&gt;;
882&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
883&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
884&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After that was somewhat working correctly, I started creating a struct that
885would hold all the commands and results and I call them Cells. Yes, I stole that
886naming idea from Jupyter.&lt;/p&gt;
887&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;typedef&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;struct&lt;/span&gt;
888&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
889&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; *command;
890&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; *result;
891&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; SDL_Surface *surface;
892&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; SDL_Texture *texture;
893&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; SDL_Rect rect;
894&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;} Cell;
895&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I am at a place now where I am starting to implement scrolling. This will for
896sure be fun to code. Memory management in C is super easy. 😂&lt;/p&gt;
897&lt;p&gt;I have also added a simple &lt;a href=&#34;https://en.wikipedia.org/wiki/INI_file&#34;&gt;INI file like
898configuration&lt;/a&gt; support. It is done in an
899&lt;a href=&#34;https://github.com/nothings/stb/blob/master/docs/stb_howto.txt&#34;&gt;STB style of
900header&lt;/a&gt; and maps
901to specific options supported by the terminal. It is not universal, and the code
902below demonstrates how I will use it in the future.&lt;/p&gt;
903&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#ifndef CONFIG_H
904&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#define CONFIG_H
905&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&lt;/span&gt;
906&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;/*
907&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# This is a comment
908&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;
909&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# This is the first configuration option
910&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;dettach=value11111
911&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;
912&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# This is the second configuration option
913&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;preview=value22222
914&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;
915&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# This is the third configuration option
916&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;debug=value33333
917&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;*/&lt;/span&gt;
918&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
919&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// Define a struct to hold the configuration options
920&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;typedef&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;struct&lt;/span&gt;
921&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
922&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; dettach[256];
923&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; preview[256];
924&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; debug[256];
925&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;} Config;
926&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
927&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// Read the configuration file and return the options as a struct
928&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;extern&lt;/span&gt; Config read_config_file(&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; *filename)
929&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
930&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Create a struct to hold the configuration options
931&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; Config config = {0};
932&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
933&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Open the configuration file
934&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; FILE *file = fopen(filename, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;r&amp;#34;&lt;/span&gt;);
935&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
936&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Read each line from the file
937&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; line[256];
938&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (fgets(line, &lt;span style=&#34;color:#00f&#34;&gt;sizeof&lt;/span&gt;(line), file))
939&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
940&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Check if this line is a comment or empty
941&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (line[0] == &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#&amp;#39;&lt;/span&gt; || line[0] == &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;)
942&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;continue&lt;/span&gt;;
943&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
944&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Parse the line to get the option and value
945&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; option[128], value[128];
946&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (sscanf(line, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;%[^=]=%s&amp;#34;&lt;/span&gt;, option, value) != 2)
947&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;continue&lt;/span&gt;;
948&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
949&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Set the value of the appropriate option in the config struct
950&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (strcmp(option, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;dettach&amp;#34;&lt;/span&gt;) == 0)
951&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
952&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; strncpy(config.option1, value, &lt;span style=&#34;color:#00f&#34;&gt;sizeof&lt;/span&gt;(config.option1));
953&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
954&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (strcmp(option, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;preview&amp;#34;&lt;/span&gt;) == 0)
955&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
956&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; strncpy(config.option2, value, &lt;span style=&#34;color:#00f&#34;&gt;sizeof&lt;/span&gt;(config.option2));
957&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
958&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (strcmp(option, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;debug&amp;#34;&lt;/span&gt;) == 0)
959&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
960&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; strncpy(config.option3, value, &lt;span style=&#34;color:#00f&#34;&gt;sizeof&lt;/span&gt;(config.option3));
961&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
962&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
963&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
964&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Close the configuration file
965&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; fclose(file);
966&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
967&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// Return the configuration options
968&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; config;
969&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
970&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
971&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#endif
972&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This is as far as I managed to get for now. I have a daily job and this
973prohibits me to work on these things full time. But I should probably get back
974and finish this. At least have a simple version working out, so I can start
975testing it on my machines. Fingers crossed. 🕵️‍♂️&lt;/p&gt;
976</content:encoded>
977 </item>
978
979
980
981 <item>
982 <title>Microsoundtrack — That sound that machine makes when struggling</title>
983 <link>https://mitjafelicijan.com/that-sound-that-machine-makes-when-struggling.html</link>
984 <pubDate>Sun, 16 Oct 2022 12:00:00 &#43;0200</pubDate>
985 <guid>https://mitjafelicijan.com/that-sound-that-machine-makes-when-struggling.html</guid>
986 <description>A couple of months ago, I got an idea about micro soundtracks.</description>
987 <content:encoded>&lt;p&gt;A couple of months ago, I got an idea about micro soundtracks. In this concept,
988you are the observer, director, and audience in this tiny movies.&lt;/p&gt;
989&lt;p&gt;What you do is to attempt to imagine what would be happening around you based on
990a title of the song and let the song help you fill the void in your story.&lt;/p&gt;
991&lt;p&gt;I made these songs is Logic Pro X. Every year or so I do this kind of thing and
992make a couple of songs similar to this. But this is the first time I am posting
993about it.&lt;/p&gt;
994&lt;p&gt;You can listen to the whole set on
995&lt;a href=&#34;https://www.youtube.com/watch?v=_5oXBhSmF3c&#34;&gt;Youtube&lt;/a&gt; or scroll down the page
996and there are embedded players for each song.&lt;/p&gt;
997&lt;h2 id=&#34;a-bunch-of-inter-dimensional-people-with-loud-clocks&#34;&gt;A bunch of inter-dimensional people with loud clocks&lt;/h2&gt;
998&lt;p&gt;A group of inter-dimensional people are going up and down the elevator with you
999while having loud clocks around their necks. Each clock ticks on a different
1000frequency. A lot of other sounds are getting drawn into your dimension,
1001resulting in a strange merging of dimensions.&lt;/p&gt;
1002&lt;iframe style=&#34;border: 0; width: 100%; height: 42px;&#34; src=&#34;https://bandcamp.com/EmbeddedPlayer/album=3913808801/size=small/bgcol=ffffff/linkcol=0687f5/track=1349272965/transparent=true/&#34; seamless title=&#34;Bandcamp&#34;&gt;&lt;a href=&#34;https://mitjafelicijan.bandcamp.com/album/that-sound-that-machine-makes-when-struggling&#34;&gt;That sound that machine makes when struggling by Mitja Felicijan&lt;/a&gt;&lt;/iframe&gt;
1003&lt;h2 id=&#34;two-black-holes-conversing-about-the-weather&#34;&gt;Two black holes conversing about the weather&lt;/h2&gt;
1004&lt;p&gt;You are a traveler in a spaceship flying very close to two colliding black holes
1005having a discussion about the weather while tearing each other apart. During all
1006this your ship is getting pulled into the event horizon of both black holes,
1007putting a lot of strain on your spaceship.&lt;/p&gt;
1008&lt;iframe style=&#34;border: 0; width: 100%; height: 42px;&#34; src=&#34;https://bandcamp.com/EmbeddedPlayer/album=3913808801/size=small/bgcol=ffffff/linkcol=0687f5/track=1756714200/transparent=true/&#34; seamless title=&#34;Bandcamp&#34;&gt;&lt;a href=&#34;https://mitjafelicijan.bandcamp.com/album/that-sound-that-machine-makes-when-struggling&#34;&gt;That sound that machine makes when struggling by Mitja Felicijan&lt;/a&gt;&lt;/iframe&gt;
1009&lt;h2 id=&#34;a-planet-where-every-organism-is-a-plant&#34;&gt;A planet where every organism is a plant&lt;/h2&gt;
1010&lt;p&gt;You land on a planet where every living organism is a plant and among those
1011plants some of them are highly intelligent, and you were asked to make first
1012contact with the native species. Your visit takes place in a giant cave where
1013you are meeting these plants, and they are talking to you.&lt;/p&gt;
1014&lt;iframe style=&#34;border: 0; width: 100%; height: 42px;&#34; src=&#34;https://bandcamp.com/EmbeddedPlayer/album=3913808801/size=small/bgcol=ffffff/linkcol=0687f5/track=3710973979/transparent=true/&#34; seamless title=&#34;Bandcamp&#34;&gt;&lt;a href=&#34;https://mitjafelicijan.bandcamp.com/album/that-sound-that-machine-makes-when-struggling&#34;&gt;That sound that machine makes when struggling by Mitja Felicijan&lt;/a&gt;&lt;/iframe&gt;
1015&lt;h2 id=&#34;bio-implants-having-a-fit-and-reprogramming-your-brain&#34;&gt;Bio implants having a fit and reprogramming your brain&lt;/h2&gt;
1016&lt;p&gt;In a distant future where everybody has bio implants, you have just received
1017your first one, which happens to be a brain implant. Something goes wrong, and
1018your implant is starting to misbehave, and you are experiencing brain
1019malfunctions. You are on the streets at night a couple of hours after your
1020procedure. You can feel your sanity breaking down.&lt;/p&gt;
1021&lt;iframe style=&#34;border: 0; width: 100%; height: 42px;&#34; src=&#34;https://bandcamp.com/EmbeddedPlayer/album=3913808801/size=small/bgcol=ffffff/linkcol=0687f5/track=1157430581/transparent=true/&#34; seamless title=&#34;Bandcamp&#34;&gt;&lt;a href=&#34;https://mitjafelicijan.bandcamp.com/album/that-sound-that-machine-makes-when-struggling&#34;&gt;That sound that machine makes when struggling by Mitja Felicijan&lt;/a&gt;&lt;/iframe&gt;
1022&lt;h2 id=&#34;cow-animation&#34;&gt;Cow animation&lt;/h2&gt;
1023&lt;p&gt;I also made this little cow animation. Go into full screen to see the effects in
1024more details.&lt;/p&gt;
1025&lt;p&gt;&lt;video src=&#34;/assets/microsoundtrack/cow.m4v&#34; controls loop&gt;&lt;/video&gt;&lt;/p&gt;
1026</content:encoded>
1027 </item>
1028
1029
1030
1031 <item>
1032 <title>State of Web Technologies and Web development in year 2022</title>
1033 <link>https://mitjafelicijan.com/state-of-web-technologies-and-web-development-in-year-2022.html</link>
1034 <pubDate>Thu, 06 Oct 2022 12:00:00 &#43;0200</pubDate>
1035 <guid>https://mitjafelicijan.com/state-of-web-technologies-and-web-development-in-year-2022.html</guid>
1036 <description>Initial thoughtsThis post is a critique on the current state of web development.</description>
1037 <content:encoded>&lt;h2 id=&#34;initial-thoughts&#34;&gt;Initial thoughts&lt;/h2&gt;
1038&lt;p&gt;&lt;em&gt;This post is a critique on the current state of web development. It is an
1039opinionated post! I will learn more about this in the future, and probably
1040slightly change my mind about some of the things I criticize.&lt;/em&gt;&lt;/p&gt;
1041&lt;p&gt;I have started working on a hobby project about two weeks ago, and I wanted to
1042use that situation as a learning one. Trying new things, new technologies, new
1043tools. I always considered myself to be an adventurous person when it comes to
1044technology. I never shy away from trying new languages, new operating systems
1045etc. Likewise, I find the whole experience satisfying, and it tickles that part
1046of my brain that finds discovery the highest of the mountains to climb.&lt;/p&gt;
1047&lt;p&gt;What I always wanted to make was a coding game, that you would play in a browser
1048(just to eliminate building binaries for each operating system) where you would
1049level up your character and go into these scriptable battles. You know, RPG
1050elements.&lt;/p&gt;
1051&lt;p&gt;So, the natural way to go would be some sort of SPA (single page application)
1052with basic routing and some state management. Nothing crazy.&lt;/p&gt;
1053&lt;blockquote&gt;
1054&lt;p&gt;&lt;strong&gt;Before we move on&lt;/strong&gt;, I have to be transparent. Take my views on this with
1055a grain of salt. I have only scratched the surface with these technologies,
1056and my knowledge is full of gaps. This is my experience using some of these
1057products for the first time or in a limited capacity.&lt;/p&gt;
1058&lt;/blockquote&gt;
1059&lt;p&gt;Having this out of the way, I got myself a fresh pot of coffee and down the
1060rabbit hole I went.&lt;/p&gt;
1061&lt;h2 id=&#34;giving-react-js-a-spin&#34;&gt;Giving React JS a spin&lt;/h2&gt;
1062&lt;p&gt;I first tried &lt;a href=&#34;https://reactjs.org/&#34;&gt;React JS&lt;/a&gt;. I kind of like it. Furthermore,
1063I have worked with libraries like this in the past and also wrote a couple of
1064them (nothing compared to that level), but I had the basic understanding of what
1065was going on. I rolled up a project quickly and had basic things done in a
1066matter of two hours, which was impressive.&lt;/p&gt;
1067&lt;p&gt;I prefer using &lt;a href=&#34;https://tailwindcss.com/&#34;&gt;Tailwind CSS&lt;/a&gt; for my styling
1068pleasures, and integrating that was also a painless experience. It was actually
1069nice to see that some things got better with time. In about 2 minutes I got
1070Tailwind working, and I was able to use classes at my disposal. All that
1071&lt;code&gt;postcss&lt;/code&gt; stuff was taken care of by adding a couple of things in config files
1072(all described really well in their documentation).&lt;/p&gt;
1073&lt;p&gt;It is not that different from Vue which I have had more encounters with in the
1074past People will probably call me a lunatic for saying this. But you know, it is
1075the truth. Same same, but different. I still believe that using libraries like
1076this is beneficial. I am not a JavaScript purist. They all have their quirks,
1077but at the end of the day, I truly believe it’s worth it.&lt;/p&gt;
1078&lt;h2 id=&#34;bundlers-and-transpilers&#34;&gt;Bundlers and Transpilers&lt;/h2&gt;
1079&lt;p&gt;I still reject calling &lt;a href=&#34;https://www.typescriptlang.org/&#34;&gt;Typescript&lt;/a&gt; to
1080&lt;a href=&#34;https://www.javascript.com/&#34;&gt;JavaScript&lt;/a&gt; conversion a &amp;quot;compilation process&amp;quot;. I
1081call them &lt;a href=&#34;https://devopedia.org/transpiler&#34;&gt;transpilers&lt;/a&gt;, and I don’t care! 😈&lt;/p&gt;
1082&lt;p&gt;And if you want to fight this, take a look at this little chart and be mad at
1083it!&lt;/p&gt;
1084&lt;p&gt;&lt;img src=&#34;/assets/state-of-web/compiling-vs-transpiling.png&#34; alt=&#34;Compiling vs Transpiling&#34; /&gt;&lt;/p&gt;
1085&lt;p&gt;The first one that I ever used was &lt;a href=&#34;https://webpack.js.org/&#34;&gt;webpack&lt;/a&gt;, and it
1086was an absolute horrific experience. Saying this, it is an absolutely fantastic
1087tool. I felt more like a config editor than actually a programmer. To be fair,
1088I am a huge fan of &lt;a href=&#34;https://www.gnu.org/software/make/&#34;&gt;make&lt;/a&gt;, and you can do as
1089you wish with this information. I like my build systems simple.&lt;/p&gt;
1090&lt;p&gt;Also, isn’t it interesting that we need something like
1091&lt;a href=&#34;https://babeljs.io/&#34;&gt;Babel&lt;/a&gt; to make JavaScript code work in a browser that has
1092only one client side scripting available, which is by no accident also
1093JavaScript. Why? I know why it’s needed, but seriously, why.&lt;/p&gt;
1094&lt;p&gt;I haven’t used Babel for years now. Or if I did, it was packaged together by
1095some other bundler thingy. Which does not make things better, but at least I
1096didn’t need to worry about it.&lt;/p&gt;
1097&lt;p&gt;I really don’t like complicated build systems. I really don’t like abstracting
1098code and making things appear magical. The older I get, the more I appreciate
1099clear and clean, expressive code. No one-liners, if possible.&lt;/p&gt;
1100&lt;p&gt;But I have to give props to &lt;a href=&#34;https://vitejs.dev/&#34;&gt;Vite&lt;/a&gt;! This was one of the
1101best developer experiences I have ever had. Granted, it still has magical
1102properties. And yes, it still is a bundler and abstracts things to the nth
1103degree. But at least it didn’t force me to configure 700 lines of JSON. And I
1104know that this makes me a hypocrite. You can’t have it all. Nonetheless, my
1105reasoning here is, if using bundlers is inevitable, then at least they should
1106provide an excellent developer experience.&lt;/p&gt;
1107&lt;p&gt;I also noticed that now the catch-all phrase is “blazingly fast” and “lightning
1108fast” and “next generation” and stuff like that. I mean, yeah, tools should get
1109faster with time. But saying that starting a project now takes 2 seconds instead
1110of 20 seconds is something that is a break it or make it kind of a deal is
1111ridiculous. I don’t mind waiting a couple of seconds every couple of days. I
1112also don’t create 700 projects every day, and also who does? This argument has
1113no bite. All I want is a decent reload time (~100ms is more than good enough for
1114me) and that is it.&lt;/p&gt;
1115&lt;p&gt;You don’t need to sell me benefits if I only get them when I start a fresh
1116project, and then try to convince me that this is somehow changing the fate of
1117the universe. First of all, it is not. And second, if this is your only argument
1118for your tool, I would advise you to maybe re-focus your efforts to something
1119else. Vite says that startup times are really fast. And if that would be the
1120only thing differentiating it from other tools, I would ignore it. But it has
1121some really compelling features like &lt;a href=&#34;https://www.geeksforgeeks.org/reactjs-hot-module-replacement/&#34;&gt;Hot Module
1122Replacement&lt;/a&gt; that
1123really works well. It was a joy to use.&lt;/p&gt;
1124&lt;p&gt;So, I will be definitely using Vite in the future.&lt;/p&gt;
1125&lt;h2 id=&#34;jam-stack-mach-stack-no-snack&#34;&gt;Jam Stack, Mach Stack no snack&lt;/h2&gt;
1126&lt;p&gt;Let&#39;s get a couple of the acronyms out of the way, so we all know what we are
1127talking about:&lt;/p&gt;
1128&lt;ul&gt;
1129&lt;li&gt;Jam Stack - JavaScript, API and Markup&lt;/li&gt;
1130&lt;li&gt;Mach Stack - Microservices, API-first, Cloud-Native SaaS, Headless&lt;/li&gt;
1131&lt;/ul&gt;
1132&lt;p&gt;It is so hard to follow all these new trendy things happening around you, that
1133it makes you have a massive &lt;strong&gt;FOMO&lt;/strong&gt; all the time. But on the other hand, you
1134also don’t want to be that old fart that doesn’t move with the times and still
1135writes his trusty jQuery code while listening to Blink 182 All the small things
1136on full blast. It’s a good song, don’t get me wrong, but there are other songs
1137out there.&lt;/p&gt;
1138&lt;p&gt;I have to admit. &lt;a href=&#34;https://vercel.com/&#34;&gt;Vercel&lt;/a&gt; is really cool! Love the
1139simplicity of the service. You could compare it to
1140&lt;a href=&#34;https://www.netlify.com/&#34;&gt;Netlify&lt;/a&gt;. I haven’t tried Netlify extensively, but
1141from a couple of experimental deployments I still prefer Vercel. It is much more
1142streamlined, but maybe this is bias in me. I really like Vercel’s Analytics,
1143which give you a &lt;a href=&#34;https://web.dev/vitals/&#34;&gt;Core Web Vitals report&lt;/a&gt; in their
1144admin console. Kind of cool, I’m not going to lie.&lt;/p&gt;
1145&lt;p&gt;This whole idea about frontend and backend merging into &lt;a href=&#34;https://www.debugbear.com/blog/server-side-rendering&#34;&gt;SSR (server-side
1146rendering)&lt;/a&gt; looks so good
1147on paper. It almost doesn’t come with any major flaws.&lt;/p&gt;
1148&lt;p&gt;But when it comes to the actual implementation, there is much to be desired.
1149I’m going to lump &lt;a href=&#34;https://nextjs.org/&#34;&gt;Next.js&lt;/a&gt; and
1150&lt;a href=&#34;https://nuxtjs.org/&#34;&gt;Nuxt.js&lt;/a&gt; together because they are essentially the same
1151thing, just a different library.&lt;/p&gt;
1152&lt;p&gt;Now comes the reality. Mixing backend and frontend in this manner creates this
1153weird mental model where you kind of rely on magical properties of these
1154libraries. You relinquish control over to them for better developer experience.
1155But is that really true? Initially, I was so stoked about it. However, the more
1156I used them, the more I felt uncomfortable. I felt dirty, actually. Maybe this
1157is because I come from old ways of doing things where you control every step of
1158request, and allowing something to hijack it feels like blasphemy.&lt;/p&gt;
1159&lt;p&gt;More than that, some pretty significant technical issues arose from this. How do
1160you do JWT token authentication? You put it in &lt;code&gt;api&lt;/code&gt; folder and then do some
1161fetching and storing into local state management. But doing this also requires
1162some tinkering with await/async stuff on the React/Vue side of things. And then
1163you need to write middleware for it. And the more I look at it, the more I see
1164that this whole thing was not meant to be used like this, and it all feels and
1165looks like a huge hack.&lt;/p&gt;
1166&lt;p&gt;The issue I have with this is that they over-promise and under-deliver. They
1167want to be an all-in-one replacement for everything, and they don’t deliver on
1168this promise. And how could they?! We have to be fair. It is an impossible task.&lt;/p&gt;
1169&lt;p&gt;They sell you &lt;a href=&#34;https://www.geeksforgeeks.org/overview-of-noops/&#34;&gt;NoOps&lt;/a&gt;, but
1170when you need to accomplish something a little bit more out of the scope of
1171Hello World, you have to make hacky decisions to make it work. And having a
1172deployment strategy that relies on many moving parts is never a good idea.
1173Abstracting too much is usually a sign of bad architecture.&lt;/p&gt;
1174&lt;p&gt;Lately, this has become a huge trend that will for sure bite us in the future.
1175And let’s not get it twisted. By doing this, PaaS providers like
1176&lt;a href=&#34;https://aws.amazon.com/&#34;&gt;AWS&lt;/a&gt;, &lt;a href=&#34;https://cloud.google.com/&#34;&gt;GCS&lt;/a&gt;, etc. obscure
1177their billing, and you end up paying more than you really should. And even if
1178that is not an issue, it comes down to the principle of things. AWS is known for
1179having multiple “currencies“ inside their projects like write operations, read
1180operations, etc. which add up, and it creates this impossible to track billing
1181scheme. It all behaves suspiciously like a pay-to-win game you could find on
1182mobile phones that scams you out of your money.&lt;/p&gt;
1183&lt;p&gt;And as far as I am concerned, the most important thing was me not coding the
1184functionalities for the game I want to make. I was battling libraries and cloud
1185providers. How to deploy, what settings are relevant. Bad documentation or
1186multiple versions of achieving the same thing. You are getting bombarded by all
1187this information, and you don’t really have any control over it.
1188Production-ready code becomes a joke, essentially. Especially if you tend to
1189work on that project for a prolonged period of time.&lt;/p&gt;
1190&lt;p&gt;All of these options end up creating a fatigue. What to choose, what not to
1191choose. Unnecessary worrying about if the stack will still be deemed worthy in
1192six months. There is elegance in simplicity.&lt;/p&gt;
1193&lt;blockquote&gt;
1194&lt;p&gt;JavaScript UI frameworks and libraries work in cycles. Every six months or
1195so, a new one pops up, claiming that it has revolutionized UI development.
1196Thousands of developers adopt it into their new projects, blog posts are
1197written, Stack Overflow questions are asked and answered, and then a newer
1198(and even more revolutionary) framework pops up to usurp the throne.
1199— Ian Allen&lt;/p&gt;
1200&lt;/blockquote&gt;
1201&lt;p&gt;&lt;img src=&#34;/assets/state-of-web/2008-vs-2020.png&#34; alt=&#34;To many options&#34; /&gt;&lt;/p&gt;
1202&lt;p&gt;And this jab at these libraries and cloud providers is not done out of malice.
1203It is a real concern that I have about them. In my life, I have seen
1204technologies come and go, but the basics always stick around. So surrendering
1205all the power you have to a library or a cloud provider is in my opinion a
1206stupid move.&lt;/p&gt;
1207&lt;h2 id=&#34;tailwind-css-still-rocks&#34;&gt;Tailwind CSS still rocks!&lt;/h2&gt;
1208&lt;p&gt;You know, many people say negative things about Tailwind. And after a lot of
1209deliberation, I came to the conclusion that Tailwind is good for two types of
1210developers. Tailwind is good for a complete noob or a senior developer. A
1211complete noob doesn’t really care about inner workings of CSS, and a senior
1212developer also doesn’t care about CSS. Well, at least, not anymore. And
1213developers in between usually have the biggest issues with it. Not always of
1214course, but in a lot of cases.&lt;/p&gt;
1215&lt;p&gt;I like the creature comforts of Tailwind. Being utility first would make me
1216argue that it is actually more similar to &lt;a href=&#34;https://sass-lang.com/&#34;&gt;Sass&lt;/a&gt; or
1217&lt;a href=&#34;https://lesscss.org/&#34;&gt;Less&lt;/a&gt; than something like Bootstrap. Not technically, but
1218ideologically. After I started using it, I never looked back. I use it every
1219time I need to do something web related.&lt;/p&gt;
1220&lt;p&gt;Writing CSS for general things feels like going several steps back. Instead of
1221focusing on what you are actually trying to achieve, you focus on notations like
1222&lt;a href=&#34;https://en.bem.info/methodology/css/&#34;&gt;BEM&lt;/a&gt;, code structuring, optimizing HTML
1223size. Just doing things that make 0.1% difference. You know that saying: Early
1224optimization is the root of all evil. Exactly that.&lt;/p&gt;
1225&lt;p&gt;I am also not saying that Tailwind is the cure for everything. Sometimes custom
1226CSS is necessary. But from what I found out in using it for almost two years in
1227a production environment (on a site getting quite a lot of traffic and
1228constantly being changed), I can say without any reservations that Tailwind
1229saved our asses countless times. We would be rewriting CSS all the time without
1230it. And I don’t really think writing CSS is the best way to spend my time.&lt;/p&gt;
1231&lt;p&gt;I have also noticed that people who criticize Tailwind the most never actually
1232used it in a real project that has a long lifetime with plenty of changes that
1233will happen in the future.&lt;/p&gt;
1234&lt;p&gt;But you know, whatever floats your boat!&lt;/p&gt;
1235&lt;h2 id=&#34;code-maintainability&#34;&gt;Code maintainability&lt;/h2&gt;
1236&lt;p&gt;Somehow, people also stopped talking about maintenance. If you constantly try to
1237catch the latest and greatest train, you are by that logic always trying new
1238things. Which is a good thing if you want to learn about technologies and try
1239them. But for the production environment, you have to have a stable stack that
1240doesn’t change every 6 months.&lt;/p&gt;
1241&lt;p&gt;You can lock dependencies for sure. Nevertheless, the hype train moves along
1242anyway. And the mindset this breeds goes against locking the code. This
1243bleeding-edge rolling release cycle is not helping. That is why enterprise
1244solutions usually look down on these popular stacks and only do bare minimum to
1245appear hip and cool.&lt;/p&gt;
1246&lt;p&gt;With that said, I still think that progress is good, but should be taken with a
1247grain of salt. If your project is something that should be built once and then
1248rarely updated, going with the latest stack is a possible way to go. But, if you
1249are working on a project that lasts for years, you should probably approach it
1250with some level of caution. Web development is often times too volatile.&lt;/p&gt;
1251&lt;h2 id=&#34;web-development-has-a-marketing-issue&#34;&gt;Web development has a marketing issue&lt;/h2&gt;
1252&lt;p&gt;I noticed that almost every project now has this marketing spin put on it.
1253Everything is blazingly fast now. I get it, they are competing for your
1254attention, but what happened to just being truthful and not inflating reality.&lt;/p&gt;
1255&lt;p&gt;And in order to appeal to mass market, they leave things out of their marketing
1256materials. These open-source projects are now behaving more and more like
1257companies do. Which is a scary thought on its self.&lt;/p&gt;
1258&lt;p&gt;And we are also seeing a rise in a concept of building a company in the open,
1259which is a good thing, don&#39;t get me wrong. But when it is using open-source to
1260lure people and then lock them in their ecosystem, there is where I have issues
1261with it.&lt;/p&gt;
1262&lt;p&gt;This might be because I have been using GNU/Linux for 20 years now and have been
1263so beholden for my success to open-source that I see issues when open-source is
1264being used to trick people into a false sense of security that these projects
1265are built in the spirit of open-source. Because there is a difference. They are
1266NOT! They have a really specific goal in mind. And the open-source is being used
1267as a delivery system. Which is in my opinion disgusting!&lt;/p&gt;
1268&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;
1269&lt;p&gt;I will end my post with this. Web development is running now in circles. People
1270are discovering &lt;a href=&#34;https://www.tutorialspoint.com/remote-procedure-call-rpc&#34;&gt;RPC&lt;/a&gt;
1271now and this is the now the next big thing. &lt;a href=&#34;https://graphql.org/&#34;&gt;GraphQL&lt;/a&gt; is
1272so passé. And I am so tired of it all. Of blazingly fast libraries, of all these
1273new technologies that are actually just a remake of old ones. Of just the
1274general spirit of the web. I will just use what I already know. Which worked 10
1275years ago and will work 10 years after this. I will adopt a couple of little
1276tools like Vite. But I will not waste my time on this anymore.&lt;/p&gt;
1277&lt;p&gt;It was a good exercise to get in touch with what’s new now. Nothing really
1278changed that much. FOMO is now cured! Now I have to get my ass back to actually
1279code and make the project that I wanted to make in the first place.&lt;/p&gt;
1280</content:encoded>
1281 </item>
1282
1283
1284
1285
1286
1287 <item>
1288 <title>Aerial photography of algae spotted on river Sava</title>
1289 <link>https://mitjafelicijan.com/aerial-photography-of-algae-spotted-on-river-sava.html</link>
1290 <pubDate>Sat, 13 Aug 2022 12:00:00 &#43;0200</pubDate>
1291 <guid>https://mitjafelicijan.com/aerial-photography-of-algae-spotted-on-river-sava.html</guid>
1292 <description>This is a bit of a different post than I usually write, but quite interestingone to me.</description>
1293 <content:encoded>&lt;p&gt;This is a bit of a different post than I usually write, but quite interesting
1294one to me. River Sava has plenty of hydropower plants located down the stream.
1295This makes regulating the strength of a current easier than normally. Because of
1296lower stream strength and high temperatures, algae has formed on the river.
1297This is the first time I&#39;ve seen something like this in my whole life.&lt;/p&gt;
1298&lt;p&gt;Below are some photographs taken from a DJI drone capturing the event.&lt;/p&gt;
1299&lt;p&gt;&lt;img src=&#34;/assets/algae-sava/dji-algae-0.jpg&#34; alt=&#34;Algae on Sava&#34; /&gt;&lt;/p&gt;
1300&lt;p&gt;&lt;img src=&#34;/assets/algae-sava/dji-algae-1.jpg&#34; alt=&#34;Algae on Sava&#34; /&gt;&lt;/p&gt;
1301&lt;p&gt;&lt;img src=&#34;/assets/algae-sava/dji-algae-2.jpg&#34; alt=&#34;Algae on Sava&#34; /&gt;&lt;/p&gt;
1302&lt;p&gt;&lt;img src=&#34;/assets/algae-sava/dji-algae-3.jpg&#34; alt=&#34;Algae on Sava&#34; /&gt;&lt;/p&gt;
1303&lt;p&gt;&lt;img src=&#34;/assets/algae-sava/dji-algae-4.jpg&#34; alt=&#34;Algae on Sava&#34; /&gt;&lt;/p&gt;
1304&lt;p&gt;&lt;img src=&#34;/assets/algae-sava/dji-algae-5.jpg&#34; alt=&#34;Algae on Sava&#34; /&gt;&lt;/p&gt;
1305&lt;p&gt;I will try to get more photos of this in the future days and if something
1306intriguing shows up will post it again on the blog.&lt;/p&gt;
1307</content:encoded>
1308 </item>
1309
1310
1311
1312 <item>
1313 <title>What would DNA sound if synthesized to an audio file</title>
1314 <link>https://mitjafelicijan.com/what-would-dna-sound-if-synthesized.html</link>
1315 <pubDate>Tue, 05 Jul 2022 12:00:00 &#43;0200</pubDate>
1316 <guid>https://mitjafelicijan.com/what-would-dna-sound-if-synthesized.html</guid>
1317 <description>IntroductionLately, I have been thinking a lot about the nature of life, what are thefoundation blocks of life and things like that.</description>
1318 <content:encoded>&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
1319&lt;p&gt;Lately, I have been thinking a lot about the nature of life, what are the
1320foundation blocks of life and things like that. It&#39;s remarkable how complex and
1321on the other hand simple the creation is when you look at it. The miracle of
1322life keeps us grounded when our imagination goes wild. If the DNA are the blocks
1323of life, you could consider them to be an API nature provided us to better
1324understand all of this chaos masquerading as order.&lt;/p&gt;
1325&lt;p&gt;I have been reading a lot about superintelligence and our somehow misguided path
1326to create general artificial intelligence. What would the building blocks or our
1327creation look like? Is the compression really the ultimate storage of
1328information? Will our creation also ponder this questions when creating new
1329worlds for themselves, or will we just disappear into the vastness of
1330possibilities? It is a little offensive that we are playing God whilst being
1331completely ignorant of our own reality. Who knows! Like many other
1332breakthroughs, this one will also come at a cost not known to us when it finally
1333happens.&lt;/p&gt;
1334&lt;p&gt;To keep things a bit lighter, I decided to convert some popular DNA sequences
1335into an audio files for us to listen to. I am not the first one, nor I will be
1336the last one to do this. But it is an interesting exercise in better
1337understanding the relationship between art and science. Maybe listening to DNA
1338instead of parsing it will find a way into better understanding, or at least
1339enjoying the creation and cryptic nature of life.&lt;/p&gt;
1340&lt;h2 id=&#34;dna-encoding-and-primer-example&#34;&gt;DNA encoding and primer example&lt;/h2&gt;
1341&lt;p&gt;I have been exploring DNA in the past in my post from about 3 years ago in
1342&lt;a href=&#34;/encoding-binary-data-into-dna-sequence.html&#34;&gt;Encoding binary data into DNA
1343sequence&lt;/a&gt; where I have been
1344converting all sorts of data into DNA sequences.&lt;/p&gt;
1345&lt;p&gt;This will be a similar exercise but instead of converting to DNA, I will be
1346generating tones from Nucleotides.&lt;/p&gt;
1347&lt;table&gt;
1348&lt;thead&gt;
1349&lt;tr&gt;
1350&lt;th&gt;Nucleotides&lt;/th&gt;
1351&lt;th&gt;Note&lt;/th&gt;
1352&lt;th&gt;Frequency&lt;/th&gt;
1353&lt;/tr&gt;
1354&lt;/thead&gt;
1355&lt;tbody&gt;
1356&lt;tr&gt;
1357&lt;td&gt;&lt;strong&gt;A&lt;/strong&gt; (Adenine)&lt;/td&gt;
1358&lt;td&gt;A&lt;/td&gt;
1359&lt;td&gt;440 Hz&lt;/td&gt;
1360&lt;/tr&gt;
1361&lt;tr&gt;
1362&lt;td&gt;&lt;strong&gt;C&lt;/strong&gt; (Cytosine)&lt;/td&gt;
1363&lt;td&gt;C&lt;/td&gt;
1364&lt;td&gt;783.99 Hz&lt;/td&gt;
1365&lt;/tr&gt;
1366&lt;tr&gt;
1367&lt;td&gt;&lt;strong&gt;G&lt;/strong&gt; (Guanine)&lt;/td&gt;
1368&lt;td&gt;G&lt;/td&gt;
1369&lt;td&gt;523.25 Hz&lt;/td&gt;
1370&lt;/tr&gt;
1371&lt;tr&gt;
1372&lt;td&gt;&lt;strong&gt;T&lt;/strong&gt; (Thymine)&lt;/td&gt;
1373&lt;td&gt;D&lt;/td&gt;
1374&lt;td&gt;587.33 Hz&lt;/td&gt;
1375&lt;/tr&gt;
1376&lt;/tbody&gt;
1377&lt;/table&gt;
1378&lt;p&gt;Since we do not have T in equal-tempered scale, I choose D to represent T note.&lt;/p&gt;
1379&lt;p&gt;You can check &lt;a href=&#34;https://pages.mtu.edu/~suits/notefreqs.html&#34;&gt;Frequencies for equal-tempered scale, A4 = 440
1380Hz&lt;/a&gt;. For this tuning, we also
1381choose &lt;code&gt;Speed of Sound = 345 m/s = 1130 ft/s = 770 miles/hr&lt;/code&gt;.&lt;/p&gt;
1382&lt;p&gt;Now that we have this out of the way, we can also brush up on the DNA sequencing
1383a bit. This is a famous quote I also used for the encoding tests, and it goes
1384like this.&lt;/p&gt;
1385&lt;blockquote&gt;
1386&lt;p&gt;How wonderful that we have met with a paradox. Now we have some hope of
1387making progress.
1388― Niels Bohr&lt;/p&gt;
1389&lt;/blockquote&gt;
1390&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;gt;SEQ1
1391&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;GACAGCTTGTGTACAAGTGTGCTTGCTCGCGAGCGGGTACGCGCGTGGGCTAACAAGTGA
1392&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;GCCAGCAGGTGAACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGCTGGCGGGTGA
1393&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ACAAGTGTGCCGGTGAGCCAACAAGCAGACAAGTAAGCAGGTACGCAGGCGAGCTTGTCA
1394&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ACTCACAAGATCGCTTGTGTACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGTAT
1395&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;GCTTGCTGGCGGACAAGCCAGCTTGTAAGCGGACAAGCTTGCGCACAAGCTGGCAGGCCT
1396&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;GCCGGCTCGCGTACAAATTCACAAGTAAGTACGCTTGCGTGTACGCGGGTATGTATACTC
1397&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;AACCTCACCAAACGGGACAAGATCGCCGGCGGGCTAGTATACAAGAACGCTTGCCAGTAC
1398&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;AACC
1399&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This is what we gonna work with to get things rolling forward, when creating
1400parser and waveform generator.&lt;/p&gt;
1401&lt;h2 id=&#34;parsing-dna-data&#34;&gt;Parsing DNA data&lt;/h2&gt;
1402&lt;p&gt;This step is rather simple one. All we need to do is parse input DNA sequence in
1403&lt;a href=&#34;https://en.wikipedia.org/wiki/FASTA_format&#34;&gt;FASTA format&lt;/a&gt; well known in
1404&lt;a href=&#34;https://en.wikipedia.org/wiki/Bioinformatics&#34;&gt;Bioinformatics&lt;/a&gt; to extract single
1405Nucleotides that will be converted into separate tones based on equal-tempered
1406scale explained above.&lt;/p&gt;
1407&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;nucleotide_tone_map = {
1408&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;A&amp;#39;&lt;/span&gt;: 440,
1409&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;C&amp;#39;&lt;/span&gt;: 523.25,
1410&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;G&amp;#39;&lt;/span&gt;: 783.99,
1411&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;T&amp;#39;&lt;/span&gt;: 587.33, &lt;span style=&#34;color:#008000&#34;&gt;# converted to D&lt;/span&gt;
1412&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
1413&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1414&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; split(word):
1415&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; [char &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; char &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; word]
1416&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1417&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; generate_from_dna_sequence(sequence):
1418&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; nucleotide &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; split(sequence):
1419&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(nucleotide, nucleotide_tone_map[nucleotide])
1420&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;generating-sine-wave&#34;&gt;Generating sine wave&lt;/h2&gt;
1421&lt;p&gt;Because we are essentially creating a long stream of notes we will be appending
1422sine notes to a global array we will later use for creating a WAV file out of
1423it.&lt;/p&gt;
1424&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; math
1425&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1426&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; append_sinewave(freq=440.0, duration_milliseconds=500, volume=1.0):
1427&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;global&lt;/span&gt; audio
1428&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1429&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; num_samples = duration_milliseconds * (sample_rate / 1000.0)
1430&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1431&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; x &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; range(int(num_samples)):
1432&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; audio.append(volume * math.sin(2 * math.pi * freq * (x / sample_rate)))
1433&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1434&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt;
1435&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The sine wave generated here is the standard beep. If you want something more
1436aggressive, you could try a square or saw tooth waveform.&lt;/p&gt;
1437&lt;h2 id=&#34;generating-a-wav-file-from-accumulated-sine-waves&#34;&gt;Generating a WAV file from accumulated sine waves&lt;/h2&gt;
1438&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; wave
1439&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; struct
1440&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1441&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; save_wav(file_name):
1442&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; wav_file = wave.open(file_name, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;w&amp;#39;&lt;/span&gt;)
1443&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; nchannels = 1
1444&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; sampwidth = 2
1445&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1446&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; nframes = len(audio)
1447&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; comptype = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;NONE&amp;#39;&lt;/span&gt;
1448&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; compname = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;not compressed&amp;#39;&lt;/span&gt;
1449&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; wav_file.setparams((nchannels, sampwidth, sample_rate, nframes, comptype, compname))
1450&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1451&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; sample &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; audio:
1452&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; wav_file.writeframes(struct.pack(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;h&amp;#39;&lt;/span&gt;, int(sample * 32767.0)))
1453&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1454&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; wav_file.close()
1455&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;44100 is the industry standard sample rate - CD quality. If you need to save on
1456file size, you can adjust it downwards. The standard for low quality is, 8000 or
14578kHz.&lt;/p&gt;
1458&lt;p&gt;WAV files here are using short, 16 bit, signed integers for the sample size.
1459So, we multiply the floating-point data we have by 32767, the maximum value for
1460a short integer.&lt;/p&gt;
1461&lt;blockquote&gt;
1462&lt;p&gt;It is theoretically possible to use the floating point -1.0 to 1.0 data
1463directly in a WAV file, but not obvious how to do that using the wave module
1464in Python.&lt;/p&gt;
1465&lt;/blockquote&gt;
1466&lt;h2 id=&#34;generating-spectograms&#34;&gt;Generating Spectograms&lt;/h2&gt;
1467&lt;p&gt;I have tried two methods of doing this and both were just fine. I however opted
1468out to use the &lt;a href=&#34;https://linux.die.net/man/1/sox&#34;&gt;SoX - Sound eXchange, the Swiss Army knife of audio
1469manipulation&lt;/a&gt; one because it didn&#39;t require
1470anything else.&lt;/p&gt;
1471&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sox output.wav -n spectrogram -o spectrogram.png
1472&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;An example spectrogram of Ludwig van Beethoven Symphony No. 6 First movement.&lt;/p&gt;
1473&lt;audio controls&gt;
1474 &lt;source src=&#34;/assets/dna-synthesized/symphony-no6-1st-movement.mp3&#34; type=&#34;audio/mpeg&#34;&gt;
1475&lt;/audio&gt;
1476&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/symphony-no6-1st-movement.png&#34; alt=&#34;Ludwig van Beethoven Symphony No. 6 First movement&#34; /&gt;&lt;/p&gt;
1477&lt;p&gt;The other option could also be in combination with
1478&lt;a href=&#34;http://www.gnuplot.info/&#34;&gt;gnuplot&lt;/a&gt;. This would require an intermediary step,
1479however.&lt;/p&gt;
1480&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sox output.wav audio.dat
1481&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;tail -n&#43;3 audio.dat &amp;gt; audio_only.dat
1482&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;gnuplot audio.gpi
1483&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And input file &lt;code&gt;audio.gpi&lt;/code&gt; that would be passed to gnuplot looks something like
1484this.&lt;/p&gt;
1485&lt;pre&gt;&lt;code&gt;# set output format and size
1486set term png size 1000,280
1487
1488# set output file
1489set output &amp;quot;audio.png&amp;quot;
1490
1491# set y range
1492set yr [-1:1]
1493
1494# we want just the data
1495unset key
1496unset tics
1497unset border
1498set lmargin 0
1499set rmargin 0
1500set tmargin 0
1501set bmargin 0
1502
1503# draw rectangle to change background color
1504set obj 1 rectangle behind from screen 0,0 to screen 1,1
1505set obj 1 fillstyle solid 1.0 fillcolor rgbcolor &amp;quot;#ffffff&amp;quot;
1506
1507# draw data with foreground color
1508plot &amp;quot;audio_only.dat&amp;quot; with lines lt rgb &#39;red&#39;
1509&lt;/code&gt;&lt;/pre&gt;
1510&lt;h2 id=&#34;pre-generated-sequences&#34;&gt;Pre-generated sequences&lt;/h2&gt;
1511&lt;p&gt;What I did was take interesting parts from an animal&#39;s genome and feed it to a
1512tone generator script. This then generated a WAV file and I converted those to
1513MP3, so they can be played in a browser. The last step was creating a
1514spectrogram based on a WAV file.&lt;/p&gt;
1515&lt;h3 id=&#34;niels-bohr-quote&#34;&gt;Niels Bohr quote&lt;/h3&gt;
1516&lt;audio controls&gt;
1517 &lt;source src=&#34;/assets/dna-synthesized/quote/out.mp3&#34; type=&#34;audio/mpeg&#34;&gt;
1518&lt;/audio&gt;
1519&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/quote/spectogram.png&#34; alt=&#34;Spectogram&#34; /&gt;&lt;/p&gt;
1520&lt;h3 id=&#34;mouse&#34;&gt;Mouse&lt;/h3&gt;
1521&lt;p&gt;This is part of a mouse genome &lt;code&gt;Mus_musculus.GRCm39.dna.nonchromosomal&lt;/code&gt;. You
1522can get &lt;a href=&#34;http://ftp.ensembl.org/pub/release-106/fasta/mus_musculus/dna/&#34;&gt;genom data
1523here&lt;/a&gt;.&lt;/p&gt;
1524&lt;audio controls&gt;
1525 &lt;source src=&#34;/assets/dna-synthesized/mouse/out.mp3&#34; type=&#34;audio/mpeg&#34;&gt;
1526&lt;/audio&gt;
1527&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/mouse/spectogram.png&#34; alt=&#34;Spectogram&#34; /&gt;&lt;/p&gt;
1528&lt;h3 id=&#34;bison&#34;&gt;Bison&lt;/h3&gt;
1529&lt;p&gt;This is part of a bison genome &lt;code&gt;Bison_bison_bison.Bison_UMD1.0.cdna&lt;/code&gt;. You can
1530get &lt;a href=&#34;http://ftp.ensembl.org/pub/release-106/fasta/bison_bison_bison/cdna/&#34;&gt;genom data
1531here&lt;/a&gt;.&lt;/p&gt;
1532&lt;audio controls&gt;
1533 &lt;source src=&#34;/assets/dna-synthesized/bison/out.mp3&#34; type=&#34;audio/mpeg&#34;&gt;
1534&lt;/audio&gt;
1535&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/bison/spectogram.png&#34; alt=&#34;Spectogram&#34; /&gt;&lt;/p&gt;
1536&lt;h3 id=&#34;taurus&#34;&gt;Taurus&lt;/h3&gt;
1537&lt;p&gt;This is part of a taurus genome &lt;code&gt;Bos_taurus.ARS-UCD1.2.cdna&lt;/code&gt;. You can get
1538&lt;a href=&#34;http://ftp.ensembl.org/pub/release-106/fasta/bos_taurus/cdna/&#34;&gt;genom data
1539here&lt;/a&gt;.&lt;/p&gt;
1540&lt;audio controls&gt;
1541 &lt;source src=&#34;/assets/dna-synthesized/taurus/out.mp3&#34; type=&#34;audio/mpeg&#34;&gt;
1542&lt;/audio&gt;
1543&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/taurus/spectogram.png&#34; alt=&#34;Spectogram&#34; /&gt;&lt;/p&gt;
1544&lt;h2 id=&#34;making-a-drummer-out-of-a-dna-sequence&#34;&gt;Making a drummer out of a DNA sequence&lt;/h2&gt;
1545&lt;p&gt;To make things even more interesting, I decided to send this data via MIDI to my
1546&lt;a href=&#34;https://www.elektron.se/en/model-samples&#34;&gt;Elektron Model:Samples&lt;/a&gt;. This is a
1547really cool piece of equipment that supports MIDI in via USB and 3.5 mm audio
1548jack.&lt;/p&gt;
1549&lt;p&gt;Elektron is connected to my MacBook via USB cable and audio out is patched to a
1550Sony Bluetooth speaker I have that supports 3.5 mm audio in. Elektron doesn&#39;t
1551have internal speakers.&lt;/p&gt;
1552&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/elektron/IMG_0619.jpg&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
1553&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/elektron/IMG_0620.jpg&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
1554&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/elektron/IMG_0622.jpg&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
1555&lt;p&gt;For communicating with Elektron, I choose &lt;code&gt;pygame&lt;/code&gt; Python module that has MIDI
1556built in. With this, it was rather simple to send notes to the device. All I did
1557was map MIDI notes to the actual Nucleotides.&lt;/p&gt;
1558&lt;p&gt;Before all of this I also checked Audio MIDI Setup app under MacOS and checked
1559MIDI Studio by pressing ⌘-2.&lt;/p&gt;
1560&lt;p&gt;&lt;img src=&#34;/assets/dna-synthesized/elektron/midi-studio.jpg&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
1561&lt;p&gt;The whole script that parses and send notes to the Elektron looks like this.&lt;/p&gt;
1562&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; pygame.midi
1563&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; time
1564&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1565&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;pygame.midi.init()
1566&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1567&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(pygame.midi.get_default_output_id())
1568&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(pygame.midi.get_device_info(0))
1569&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1570&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;player = pygame.midi.Output(1)
1571&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;player.set_instrument(2)
1572&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1573&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; send_note(note, velocity):
1574&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;global&lt;/span&gt; player
1575&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; player.note_on(note, velocity)
1576&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; time.sleep(0.3)
1577&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; player.note_off(note, velocity)
1578&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1579&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1580&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;nucleotide_midi_map = {
1581&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;A&amp;#39;&lt;/span&gt;: 60,
1582&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;C&amp;#39;&lt;/span&gt;: 90,
1583&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;G&amp;#39;&lt;/span&gt;: 160,
1584&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;T&amp;#39;&lt;/span&gt;: 180, &lt;span style=&#34;color:#008000&#34;&gt;# is D&lt;/span&gt;
1585&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
1586&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1587&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;with&lt;/span&gt; open(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;quote.fa&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#00f&#34;&gt;as&lt;/span&gt; f:
1588&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; sequence = f.read().replace(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&amp;#39;&lt;/span&gt;)
1589&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1590&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; nucleotide &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; [char &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; char &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; sequence]:
1591&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Playing nucleotide &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; with MIDI note &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;.format(
1592&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; nucleotide, nucleotide_midi_map[nucleotide]))
1593&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; send_note(nucleotide_midi_map[nucleotide], 127)
1594&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1595&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;del&lt;/span&gt; player
1596&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;pygame.midi.quit()
1597&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;video src=&#34;/assets/dna-synthesized/elektron/elektron.mp4&#34; controls&gt;&lt;/video&gt;&lt;/p&gt;
1598&lt;p&gt;All of this could be made much more interesting if I choose different
1599instruments for different Nucleotides, or doing more funky stuff with Elektron.
1600But for now, this should be enough. It is just a proof of concept. Something to
1601play around with.&lt;/p&gt;
1602&lt;h2 id=&#34;going-even-further&#34;&gt;Going even further&lt;/h2&gt;
1603&lt;p&gt;As you probably notice, the end results are quite similar to each other. This is
1604to be expected because we are operating only with 4 notes essentially. What
1605could make this more interesting is using something like
1606&lt;a href=&#34;https://supercollider.github.io/&#34;&gt;Supercollider&lt;/a&gt; to create more interesting
1607sounds. By transposing notes or using effects based on repeated data in a
1608sequence. Possibilities are endless.&lt;/p&gt;
1609&lt;p&gt;It is really astonishing what can be achieved with a little bit of code and an
1610idea. I could see this becoming an interesting background soundscape instrument
1611if done properly. It could replace random note generator with something more
1612intriguing, biological, natural.&lt;/p&gt;
1613&lt;p&gt;I actually find the results fascinating. I took some time and listened to this
1614music of nature. Even though it&#39;s quite the same, it&#39;s also quite different.
1615The subtle differences on repeat kind of creates music on its own. Makes you
1616wonder. It kind of puts Occam’s Razor in its place. Nature for sure loves to
1617make things as energy efficient as possible.&lt;/p&gt;
1618</content:encoded>
1619 </item>
1620
1621
1622
1623 <item>
1624 <title>Trying out Helix code editor as my main editor</title>
1625 <link>https://mitjafelicijan.com/tying-out-helix-code-editor.html</link>
1626 <pubDate>Thu, 30 Jun 2022 12:00:00 &#43;0200</pubDate>
1627 <guid>https://mitjafelicijan.com/tying-out-helix-code-editor.html</guid>
1628 <description>I have been searching for a lightweight code editor for quite some time.</description>
1629 <content:encoded>&lt;p&gt;I have been searching for a lightweight code editor for quite some time. One of
1630the main reasons was that I wanted something that doesn&#39;t burn through CPU and
1631RAM usage is not through the roof. I have been mostly using Visual Studio Code.
1632It&#39;s been an outstanding editor. I have no quarrel with it at all. It&#39;s just
1633time to spice life up with something new.&lt;/p&gt;
1634&lt;p&gt;I have been on this search for a couple of years. I have tried Vim, Neovim,
1635Emacs, Doom Emacs, Micro and couple more. Among most of them, I liked Micro and
1636Doom Emacs the most. Micro editor was a little too basic for me. And Doom Emacs
1637was a bit too hardcore. This does not reflect on any of the editors. It&#39;s just
1638my personal preference.&lt;/p&gt;
1639&lt;blockquote&gt;
1640&lt;p&gt;I tried Helix Editor about a year ago. But I didn&#39;t pay attention to it.
1641Tried it and saw it&#39;s similar to Vi and just said no. I was premature to
1642dismiss it.&lt;/p&gt;
1643&lt;/blockquote&gt;
1644&lt;p&gt;One of the things I actually miss is line wrapping for certain files. When
1645writing Markdown, line wrapping would be very helpful. Editing such a document
1646is frustrating to say the least. Some of the Markdown to HTML converters don&#39;t
1647take kindly of new lines between sentences. Not paragraphs, sentences. And I use
1648Markdown to write this blog you are reading.&lt;/p&gt;
1649&lt;p&gt;But other than this, I have been extremely satisfied by it. It&#39;s been a pleasant
1650surprise. There have been zero issues with the editor.&lt;/p&gt;
1651&lt;p&gt;One thing to do before you are able to use autocompletion and make use Language
1652Server support is to install the language server with NPM.&lt;/p&gt;
1653&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;npm install -g typescript typescript-language-server
1654&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I am still getting used to the keyboard shortcuts and getting better. What Helix
1655does really well is packing in sane defaults and even though because currently
1656there is no plugin support I haven&#39;t found any need for them. It has all that
1657you would need. It goes to extreme measures to show a user what is going on with
1658popups that show you what the keyboard shortcuts are.&lt;/p&gt;
1659&lt;p&gt;And it comes us packed with many
1660&lt;a href=&#34;https://github.com/helix-editor/helix/wiki/Themes&#34;&gt;really good themes&lt;/a&gt;.&lt;/p&gt;
1661&lt;p&gt;&lt;img src=&#34;/assets/helix-editor/editor.png&#34; alt=&#34;Editor&#34; /&gt;&lt;/p&gt;
1662&lt;p&gt;It&#39;s still young but has this mature feeling to it. It has sane defaults and
1663mimics Vim (works a bit differently, but the overall idea is similar).&lt;/p&gt;
1664</content:encoded>
1665 </item>
1666
1667
1668
1669 <item>
1670 <title>Wireless Application Protocol and the mobile web before the web</title>
1671 <link>https://mitjafelicijan.com/wap-mobile-web-before-the-web.html</link>
1672 <pubDate>Thu, 30 Dec 2021 12:00:00 &#43;0200</pubDate>
1673 <guid>https://mitjafelicijan.com/wap-mobile-web-before-the-web.html</guid>
1674 <description>A little stroll down the history laneAbout two weeks ago, I watched this outstanding documentary on YouTubeSpringboard: the secret history of the first realsmartphone about the history ofsmartphones and phones in general.</description>
1675 <content:encoded>&lt;h2 id=&#34;a-little-stroll-down-the-history-lane&#34;&gt;A little stroll down the history lane&lt;/h2&gt;
1676&lt;p&gt;About two weeks ago, I watched this outstanding documentary on YouTube
1677&lt;a href=&#34;https://www.youtube.com/watch?v=b9_Vh9h3Ohw&#34;&gt;Springboard: the secret history of the first real
1678smartphone&lt;/a&gt; about the history of
1679smartphones and phones in general. It brought back so many memories. I never had
1680an actual smartphone before the Android. The closest to smartphone was &lt;a href=&#34;https://www.gsmarena.com/sony_ericsson_p1-1982.php&#34;&gt;Sony
1681Ericsson P1&lt;/a&gt;. A fantastic
1682phone and I broke it in Prague after a party and that was one of those rare
1683occasions where I was actually mad at myself. But nevertheless, after that
1684phone, the next one was an Android one.&lt;/p&gt;
1685&lt;p&gt;Before that, I only owned normal phones from Nokia and Siemens etc. Nothing
1686special, actually. These are the phones we are talking about. Before 2007.
1687Apple and Android phones didn&#39;t exist yet.&lt;/p&gt;
1688&lt;p&gt;These phones were rocking:&lt;/p&gt;
1689&lt;ul&gt;
1690&lt;li&gt;No selfie cameras.&lt;/li&gt;
1691&lt;li&gt;~2 inch displays.&lt;/li&gt;
1692&lt;li&gt;~120 MHz beast CPU&#39;s.&lt;/li&gt;
1693&lt;li&gt;144p main cameras.&lt;/li&gt;
1694&lt;li&gt;But they had a headphone jack.&lt;/li&gt;
1695&lt;/ul&gt;
1696&lt;p&gt;Let&#39;s take a look at these beauties.&lt;/p&gt;
1697&lt;p&gt;&lt;img src=&#34;/assets/wap/phones.gif&#34; alt=&#34;Old phones&#34; /&gt;&lt;/p&gt;
1698&lt;h2 id=&#34;wap---wireless-application-protocol&#34;&gt;WAP - Wireless Application Protocol&lt;/h2&gt;
1699&lt;p&gt;Not that one! We are talking about Wireless Application Protocol and not Cardi
1700B&#39;s song 😃&lt;/p&gt;
1701&lt;p&gt;WAP stands for Wireless Application Protocol. It is a protocol designed for
1702micro-browsers, and it enables the access of internet in the mobile devices. It
1703uses the mark-up language WML (Wireless Markup Language and not HTML), WML is
1704defined as XML 1.0 application. Furthermore, it enables creating web
1705applications for mobile devices. In 1998, WAP Forum was founded by Ericson,
1706Motorola, Nokia and Unwired Planet whose aim was to standardize the various
1707wireless technologies via protocols.
1708&lt;a href=&#34;https://www.geeksforgeeks.org/wireless-application-protocol/&#34;&gt;(source)&lt;/a&gt;&lt;/p&gt;
1709&lt;p&gt;WAP protocol was resulted by the joint efforts of the various members of WAP
1710Forum. In 2002, WAP forum was merged with various other forums of the industry,
1711resulting in the formation of Open Mobile Alliance (OMA).
1712&lt;a href=&#34;https://www.geeksforgeeks.org/wireless-application-protocol/&#34;&gt;(source)&lt;/a&gt;&lt;/p&gt;
1713&lt;p&gt;These were some wild times. Devices had tiny screens and data transmission rates
1714were abominable. But they were capable of rendering WML (Wireless Markup
1715Language). This was very similar to HTML, actually. It is a markup language,
1716after all.&lt;/p&gt;
1717&lt;p&gt;These pages could be served by &lt;a href=&#34;https://apache.org/&#34;&gt;Apache&lt;/a&gt; and could be
1718generated by CGI scripts on the backend. The only difference was the limited
1719markup language.&lt;/p&gt;
1720&lt;h2 id=&#34;wml---wireless-markup-language&#34;&gt;WML - Wireless Markup Language&lt;/h2&gt;
1721&lt;p&gt;Just like web browsers use HTML for content structure, older mobile device
1722browsers use WML - if you need to support really old mobile phones using WML
1723browsers, you will need to know about it. WML is XML-based (an XML vocabulary
1724just like XHTML and MathML, but not HTML) and does not use the same metaphor as
1725HTML. HTML is a single document with some metadata packed away in the head, and
1726a body encapsulating the visible page. With WML, the metaphor does not envisage
1727a page, but rather a deck of cards. A WML file might have several pages or cards
1728contained within it.
1729&lt;a href=&#34;https://www.w3.org/wiki/Introduction_to_mobile_web&#34;&gt;(source)&lt;/a&gt;&lt;/p&gt;
1730&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;?xml version=&amp;#34;1.0&amp;#34;?&amp;gt;&lt;/span&gt;
1731&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;!DOCTYPE wml PUBLIC &amp;#34;-//WAPFORUM//DTD WML 1.1//EN&amp;#34; &amp;#34;http://www.wapforum.org/DTD/wml_1.1.xml&amp;#34;&amp;gt;&lt;/span&gt;
1732&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;wml&amp;gt;
1733&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;card id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;home&amp;#34;&lt;/span&gt; title=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Example Homepage&amp;#34;&lt;/span&gt;&amp;gt;
1734&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;Welcome to the Example homepage&amp;lt;/p&amp;gt;
1735&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/card&amp;gt;
1736&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;/wml&amp;gt;
1737&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;There is an amazing tutorial on &lt;a href=&#34;https://www.tutorialspoint.com/wml/index.htm&#34;&gt;Tutorialpoint about
1738WML&lt;/a&gt;.&lt;/p&gt;
1739&lt;h2 id=&#34;converting-digg-to-wml&#34;&gt;Converting Digg to WML&lt;/h2&gt;
1740&lt;p&gt;This task is completely useless and not really feasible nowadays, but I had to
1741give it a try for old-time sake. Since the data is already there in a form of
1742RSS feed, I could take this feed and parse it and create a WML version of the
1743homepage.&lt;/p&gt;
1744&lt;p&gt;We will need:&lt;/p&gt;
1745&lt;ul&gt;
1746&lt;li&gt;Python3 &#43; Pip&lt;/li&gt;
1747&lt;li&gt;ImageMagick&lt;/li&gt;
1748&lt;li&gt;feedparser and mako templating&lt;/li&gt;
1749&lt;/ul&gt;
1750&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# for fedora 35&lt;/span&gt;
1751&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo dnf install ImageMagick python3-pip
1752&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1753&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# tempalting engine for python&lt;/span&gt;
1754&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;pip install mako --user
1755&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1756&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# for parsing rss feeds&lt;/span&gt;
1757&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;pip install feedparser --user
1758&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Project folder structure should look like the following.&lt;/p&gt;
1759&lt;pre&gt;&lt;code&gt;12:43:53 m@khan wap → tree -L 1
1760.
1761├── generate.py
1762└── template.wml
1763
1764&lt;/code&gt;&lt;/pre&gt;
1765&lt;p&gt;After that, I created a small template for the homepage.&lt;/p&gt;
1766&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;?xml version=&amp;#34;1.0&amp;#34;?&amp;gt;&lt;/span&gt;
1767&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;!DOCTYPE wml PUBLIC &amp;#34;-//WAPFORUM//DTD WML 1.2//EN&amp;#34; &amp;#34;http://www.wapforum.org/DTD/wml_1.2.xml&amp;#34;&amp;gt;&lt;/span&gt;
1768&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1769&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;wml&amp;gt;
1770&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1771&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;card title=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Digg - What the Internet is talking about right now&amp;#34;&lt;/span&gt;&amp;gt;
1772&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1773&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; % for item in entries:
1774&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;&amp;lt;img src=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/images/${item.id}.jpg&amp;#34;&lt;/span&gt; width=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;175&amp;#34;&lt;/span&gt; height=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;95&amp;#34;&lt;/span&gt; alt=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;${item.title}&amp;#34;&lt;/span&gt; /&amp;gt;&amp;lt;/p&amp;gt;
1775&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;&amp;lt;small&amp;gt;${item.kicker}&amp;lt;/small&amp;gt;&amp;lt;/p&amp;gt;
1776&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;&amp;lt;big&amp;gt;&amp;lt;b&amp;gt;${item.title}&amp;lt;/b&amp;gt;&amp;lt;/big&amp;gt;&amp;lt;/p&amp;gt;
1777&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;${item.description}&amp;lt;/p&amp;gt;
1778&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; % endfor
1779&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1780&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/card&amp;gt;
1781&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1782&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;/wml&amp;gt;
1783&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And the parser that parses RSS feed looks like this.&lt;/p&gt;
1784&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; os
1785&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; feedparser
1786&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; mako.template &lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; Template
1787&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1788&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;os.system(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;mkdir -p www/images&amp;#39;&lt;/span&gt;)
1789&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1790&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;template = Template(filename=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;template.wml&amp;#39;&lt;/span&gt;)
1791&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1792&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;feed = feedparser.parse(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;https://digg.com/rss/top.xml&amp;#39;&lt;/span&gt;)
1793&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1794&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;entries = feed.entries[:15]
1795&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1796&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; entry &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; entries:
1797&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Processing image with id &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&lt;/span&gt;.format(entry.id))
1798&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; os.system(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;wget -q -O www/images/&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;.jpg &amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;#39;&lt;/span&gt;.format(entry.id, entry.links[1].href))
1799&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; os.system(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;convert www/images/&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;.jpg -type Grayscale -resize 175x -depth 3 -quality 30 www/images/&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;.jpg&amp;#39;&lt;/span&gt;.format(entry.id, entry.id))
1800&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1801&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;html = template.render(entries = entries)
1802&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1803&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;with&lt;/span&gt; open(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;www/index.wml&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;w&#43;&amp;#39;&lt;/span&gt;) &lt;span style=&#34;color:#00f&#34;&gt;as&lt;/span&gt; fp:
1804&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fp.write(html)
1805&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This script will create a folder &lt;code&gt;www&lt;/code&gt; and in the folder &lt;code&gt;www/images&lt;/code&gt; for
1806storing resized images.&lt;/p&gt;
1807&lt;blockquote&gt;
1808&lt;p&gt;Be sure you don&#39;t use SSL and use just normal HTTP for serving the content.
1809These old phones will have problems with TLS 1.3 etc.&lt;/p&gt;
1810&lt;/blockquote&gt;
1811&lt;p&gt;If you look at the python file, I convert all the images into tiny B&amp;amp;W images.
1812They should be WBMP (Wireless BitMaP) but I choose JPEGs for this, and it seems
1813to work properly.&lt;/p&gt;
1814&lt;p&gt;Because I currently don&#39;t have a phone old enough to test it on, I used an
1815emulator. And it was really hard to find one. I found &lt;a href=&#34;http://wap-proof.sharewarejunction.com/&#34;&gt;WAP
1816Proof&lt;/a&gt; on shareware junction, and it
1817did the job well enough. I will try to find and actual device to test it on.&lt;/p&gt;
1818&lt;p&gt;&lt;video src=&#34;/assets/wap/emulator.mp4&#34; controls&gt;&lt;/video&gt;&lt;/p&gt;
1819&lt;p&gt;If you are using Nginx to serve the contents, add a directive to the hosts file
1820that will automatically server &lt;code&gt;index.wml&lt;/code&gt; file.&lt;/p&gt;
1821&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;server&lt;/span&gt; {
1822&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;index&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;index.wml&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;index.html&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;index.htm&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;index.nginx-debian.html&lt;/span&gt;;
1823&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
1824&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;
1825&lt;p&gt;Well, this was pointless, but very fun! I hope you enjoyed it as much as I did.
1826I will try to find an old phone to test it on. If you have any questions, feel
1827free to ask in the comments.&lt;/p&gt;
1828</content:encoded>
1829 </item>
1830
1831
1832
1833 <item>
1834 <title>Running Golang application as PID 1 with Linux kernel</title>
1835 <link>https://mitjafelicijan.com/running-golang-application-as-pid1.html</link>
1836 <pubDate>Sat, 25 Dec 2021 12:00:00 &#43;0200</pubDate>
1837 <guid>https://mitjafelicijan.com/running-golang-application-as-pid1.html</guid>
1838 <description>Unikernels, kernels, and alikeI have been reading a lot aboutunikernernels lately and found themvery intriguing.</description>
1839 <content:encoded>&lt;h2 id=&#34;unikernels-kernels-and-alike&#34;&gt;Unikernels, kernels, and alike&lt;/h2&gt;
1840&lt;p&gt;I have been reading a lot about
1841&lt;a href=&#34;https://en.wikipedia.org/wiki/Unikernel&#34;&gt;unikernernels&lt;/a&gt; lately and found them
1842very intriguing. When you push away all the marketing speak and look at the
1843idea, it makes a lot of sense.&lt;/p&gt;
1844&lt;blockquote&gt;
1845&lt;p&gt;A unikernel is a specialized, single address space machine image constructed
1846by using library operating systems. (&lt;a href=&#34;https://en.wikipedia.org/wiki/Unikernel&#34;&gt;Wikipedia&lt;/a&gt;)&lt;/p&gt;
1847&lt;/blockquote&gt;
1848&lt;p&gt;I really like the explanation from the article
1849&lt;a href=&#34;https://queue.acm.org/detail.cfm?id=2566628&#34;&gt;Unikernels: Rise of the Virtual Library Operating System&lt;/a&gt;.
1850Really worth a read.&lt;/p&gt;
1851&lt;p&gt;If we compare a normal operating system to a unikernel side by side, they would
1852look something like this.&lt;/p&gt;
1853&lt;p&gt;&lt;img src=&#34;/assets/pid1/unikernels.png&#34; alt=&#34;Virtual machines vs Containers vs Unikernels&#34; /&gt;&lt;/p&gt;
1854&lt;p&gt;From this image, we can see how the complexity significantly decreases with
1855the use of Unikernels. This comes with a price, of course. Unikernels are hard
1856to get running and require a lot of work since you don&#39;t have an actual proper
1857kernel running in the background providing network access and drivers etc.&lt;/p&gt;
1858&lt;p&gt;So as a half step to make the stack simpler, I started looking into using
1859Linux kernel as a base and going from there. I came across this
1860&lt;a href=&#34;https://www.youtube.com/watch?v=Sk9TatW9ino&#34;&gt;Youtube video talking about Building the Simplest Possible Linux System&lt;/a&gt;
1861by &lt;a href=&#34;https://landley.net&#34;&gt;Rob Landley&lt;/a&gt; and apart from statically compiling the
1862application to be run as PID1 there was really no other obstacles.&lt;/p&gt;
1863&lt;h2 id=&#34;what-is-pid-1&#34;&gt;What is PID 1?&lt;/h2&gt;
1864&lt;p&gt;PID 1 is the first process that Linux kernel starts after the boot process.
1865It also has a couple of unique properties that are unique to it.&lt;/p&gt;
1866&lt;ul&gt;
1867&lt;li&gt;When the process with PID 1 dies for any reason, all other processes are
1868killed with KILL signal.&lt;/li&gt;
1869&lt;li&gt;When any process having children dies for any reason, its children are
1870re-parented to process with PID 1.&lt;/li&gt;
1871&lt;li&gt;Many signals which have default action of Term do not have one for PID 1.&lt;/li&gt;
1872&lt;li&gt;When the process with PID 1 dies for any reason, kernel panics, which
1873result in system crash.&lt;/li&gt;
1874&lt;/ul&gt;
1875&lt;p&gt;PID 1 is considered as an Init application which takes care of running other
1876and handling services like:&lt;/p&gt;
1877&lt;ul&gt;
1878&lt;li&gt;sshd,&lt;/li&gt;
1879&lt;li&gt;nginx,&lt;/li&gt;
1880&lt;li&gt;pulseaudio,&lt;/li&gt;
1881&lt;li&gt;etc.&lt;/li&gt;
1882&lt;/ul&gt;
1883&lt;p&gt;If you are on a Linux machine, you can check what your process is with PID 1
1884by running the following.&lt;/p&gt;
1885&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cat /proc/1/status
1886&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Name: systemd
1887&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Umask: 0000
1888&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;State: S (sleeping)
1889&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Tgid: 1
1890&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Ngid: 0
1891&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Pid: 1
1892&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;PPid: 0
1893&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;...
1894&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;As we can see on my machine the process with id of 1 is &lt;a href=&#34;https://systemd.io/&#34;&gt;systemd&lt;/a&gt;
1895which is a software suite that provides an array of system components for Linux
1896operating systems. If you look closely you can also see that the &lt;code&gt;PPid&lt;/code&gt;
1897(process id of the parent process) is &lt;code&gt;0&lt;/code&gt; which additionally confirms that
1898this process doesn&#39;t have a parent.&lt;/p&gt;
1899&lt;h2 id=&#34;so-why-even-run-application-as-pid-1-instead-of-just-using-a-container&#34;&gt;So why even run application as PID 1 instead of just using a container?&lt;/h2&gt;
1900&lt;p&gt;Containers are wonderful, but they come with a lot of baggage. And because they
1901are in their nature layered, the images require quite a lot of space and also a
1902lot of additional software to handle them. They are not as lightweight as they
1903seem, and many popular images require 500 MB plus disk space.&lt;/p&gt;
1904&lt;p&gt;The idea of running this as PID 1 would result in a significantly smaller footprint,
1905as we will see later in the post.&lt;/p&gt;
1906&lt;blockquote&gt;
1907&lt;p&gt;You could run a simple init system inside Docker container described more
1908in this article &lt;a href=&#34;https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/&#34;&gt;Docker and the PID 1 zombie reaping problem&lt;/a&gt;.&lt;/p&gt;
1909&lt;/blockquote&gt;
1910&lt;h2 id=&#34;the-master-plan&#34;&gt;The master plan&lt;/h2&gt;
1911&lt;ol&gt;
1912&lt;li&gt;Compile Linux kernel with the default definitions.&lt;/li&gt;
1913&lt;li&gt;Prepare a Hello World application in Golang that is statically compiled.&lt;/li&gt;
1914&lt;li&gt;Run it with &lt;a href=&#34;https://www.qemu.org/&#34;&gt;QEMU&lt;/a&gt; and providing Golang application
1915as init application / PID 1.&lt;/li&gt;
1916&lt;/ol&gt;
1917&lt;p&gt;For the sake of simplicity we will not be cross-compiling any of it and just
1918use the 64bit version.&lt;/p&gt;
1919&lt;h2 id=&#34;compiling-linux-kernel&#34;&gt;Compiling Linux kernel&lt;/h2&gt;
1920&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.7.tar.xz
1921&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ tar xf linux-5.15.7.tar.xz
1922&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1923&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cd linux-5.15.7
1924&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1925&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ make clean
1926&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1927&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# read more about this https://stackoverflow.com/a/41886394&lt;/span&gt;
1928&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ make defconfig
1929&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1930&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ time make -j &lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt;nproc&lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt;
1931&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1932&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cd ..
1933&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;At this point we have kernel image that is located in &lt;code&gt;arch/x86_64/boot/bzImage&lt;/code&gt;.
1934We will use this in QEMU later.&lt;/p&gt;
1935&lt;p&gt;To make our lives a bit easier lets move the kernel image to another place.
1936Lets create a folder &lt;code&gt;bin/&lt;/code&gt; in the root of our project with &lt;code&gt;mkdir -p bin&lt;/code&gt;.&lt;/p&gt;
1937&lt;p&gt;At this point we can copy &lt;code&gt;bzImage&lt;/code&gt; to &lt;code&gt;bin/&lt;/code&gt; folder with
1938&lt;code&gt;cp linux-5.15.7/arch/x86_64/boot/bzImage bin/bzImage&lt;/code&gt;.&lt;/p&gt;
1939&lt;p&gt;The folder structure of this experiment should look like this.&lt;/p&gt;
1940&lt;pre&gt;&lt;code&gt;pid1/
1941 bin/
1942 bzImage
1943 linux-5.15.7/
1944 linux-5.15.7.tar.xz
1945&lt;/code&gt;&lt;/pre&gt;
1946&lt;h2 id=&#34;preparing-pid-1-application-in-golang&#34;&gt;Preparing PID 1 application in Golang&lt;/h2&gt;
1947&lt;p&gt;This step is relatively easy. The only thing we must have in mind that we will
1948need to compile the binary as a static one.&lt;/p&gt;
1949&lt;p&gt;Let&#39;s create &lt;code&gt;init.go&lt;/code&gt; file in the root of the project.&lt;/p&gt;
1950&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;package&lt;/span&gt; main
1951&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1952&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; (
1953&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;fmt&amp;#34;&lt;/span&gt;
1954&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;time&amp;#34;&lt;/span&gt;
1955&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;)
1956&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1957&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;func&lt;/span&gt; main() {
1958&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; {
1959&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fmt.Println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hello from Golang&amp;#34;&lt;/span&gt;)
1960&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; time.Sleep(1 * time.Second)
1961&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
1962&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
1963&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;If you notice, we have a forever loop in the main, with a simple sleep of 1
1964second to not overwhelm the CPU. This is because PID 1 should never complete
1965and/or exit. That would result in a kernel panic. Which is BAD!&lt;/p&gt;
1966&lt;p&gt;There are two ways of compiling Golang application. Statically and dynamically.&lt;/p&gt;
1967&lt;p&gt;To statically compile the binary, use the following command.&lt;/p&gt;
1968&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ go build -ldflags=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;-extldflags=-static&amp;#34;&lt;/span&gt; init.go
1969&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;We can also check if the binary is statically compiled with:&lt;/p&gt;
1970&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ file init
1971&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;init: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=Ypu8Zw_4NBxm1Yxg2OYO/H5x721rQ9uTPiDVh-VqP/vZN7kXfGG1zhX_qdHMgH/9vBfmK81tFrygfOXDEOo, not stripped
1972&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
1973&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ ldd init
1974&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;not a dynamic executable
1975&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;At this point, we need to create &lt;a href=&#34;https://www.linuxfromscratch.org/blfs/view/svn/postlfs/initramfs.html&#34;&gt;initramfs&lt;/a&gt;
1976(abbreviated from &amp;quot;initial RAM file system&amp;quot;, is the successor of initrd. It
1977is a cpio archive of the initial file system that gets loaded into memory
1978during the Linux startup process).&lt;/p&gt;
1979&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ echo init | cpio -o --format=newc &amp;gt; initramfs
1980&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ mv initramfs bin/initramfs
1981&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The projects at this stage should look like this.&lt;/p&gt;
1982&lt;pre&gt;&lt;code&gt;pid1/
1983 bin/
1984 bzImage
1985 initramfs
1986 linux-5.15.7/
1987 linux-5.15.7.tar.xz
1988 init.go
1989&lt;/code&gt;&lt;/pre&gt;
1990&lt;h2 id=&#34;running-all-of-it-with-qemu&#34;&gt;Running all of it with QEMU&lt;/h2&gt;
1991&lt;p&gt;&lt;a href=&#34;https://www.qemu.org/&#34;&gt;QEMU&lt;/a&gt; is a free and open-source hypervisor. It emulates
1992the machine&#39;s processor through dynamic binary translation and provides a set
1993of different hardware and device models for the machine, enabling it to run a
1994variety of guest operating systems.&lt;/p&gt;
1995&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ qemu-system-x86_64 -serial stdio -kernel bin/bzImage -initrd bin/initramfs -append &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;console=ttyS0&amp;#34;&lt;/span&gt; -m 128
1996&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ qemu-system-x86_64 -serial stdio -kernel bin/bzImage -initrd bin/initramfs -append &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;console=ttyS0&amp;#34;&lt;/span&gt; -m 128
1997&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] Linux version 5.15.7 (m@khan) (gcc (GCC) 11.2.1 20211203 (Red Hat 11.2.1-7), GNU ld version 2.37-10.fc35) &lt;span style=&#34;color:#008000&#34;&gt;#7 SMP Mon Dec 13 10:23:25 CET 2021&lt;/span&gt;
1998&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] Command line: console=ttyS0
1999&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] x86/fpu: x87 FPU will use FXSAVE
2000&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] signal: max sigframe size: 1440
2001&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] BIOS-provided physical RAM map:
2002&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
2003&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
2004&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
2005&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007fdffff] usable
2006&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] BIOS-e820: [mem 0x0000000007fe0000-0x0000000007ffffff] reserved
2007&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
2008&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] NX (Execute Disable) protection: active
2009&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] SMBIOS 2.8 present.
2010&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] DMI: QEMU Standard PC (i440FX &#43; PIIX, 1996), BIOS 1.14.0-6.fc35 04/01/2014
2011&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 0.000000] tsc: Fast TSC calibration failed
2012&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;...
2013&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.016106] ALSA device list:
2014&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.016329] No soundcards found.
2015&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.053176] Freeing unused kernel image (initmem) memory: 1368K
2016&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.056095] Write protecting the kernel read-only data: 20480k
2017&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.058248] Freeing unused kernel image (text/rodata gap) memory: 2032K
2018&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.058811] Freeing unused kernel image (rodata/data gap) memory: 500K
2019&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.059164] Run /init as init process
2020&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Hello from Golang
2021&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.386879] tsc: Refined TSC clocksource calibration: 3192.032 MHz
2022&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.387114] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2e02e31fa14, max_idle_ns: 440795264947 ns
2023&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.387380] clocksource: Switched to clocksource tsc
2024&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[ 2.587895] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
2025&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Hello from Golang
2026&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Hello from Golang
2027&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Hello from Golang
2028&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The whole &lt;a href=&#34;/assets/pid1/qemu.log&#34;&gt;log file here&lt;/a&gt;.&lt;/p&gt;
2029&lt;h2 id=&#34;size-comparison&#34;&gt;Size comparison&lt;/h2&gt;
2030&lt;p&gt;The cool thing about this approach is that the Linux kernel and the application
2031together only take around 12 MB, which is impressive as hell. And we need to
2032also know that the size of bzImage (Linux kernel) could be greatly decreased
2033by going into &lt;code&gt;make menuconfig&lt;/code&gt; and removing a ton of features from the kernel,
2034making the size even smaller. I managed to get kernel size down to 2 MB and
2035still working properly.&lt;/p&gt;
2036&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;total 12M
2037&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-rw-r--r--. 1 m m 9.3M Dec 13 10:24 bzImage
2038&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;-rw-r--r--. 1 m m 1.9M Dec 27 01:19 initramfs
2039&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;creating-iso-image-and-running-it-with-gnome-boxes&#34;&gt;Creating ISO image and running it with Gnome Boxes&lt;/h2&gt;
2040&lt;p&gt;First we need to create proper folder structure with &lt;code&gt;mkdir -p iso/boot/grub&lt;/code&gt;.&lt;/p&gt;
2041&lt;p&gt;Then we need to download the &lt;a href=&#34;https://github.com/littleosbook/littleosbook/raw/master/files/stage2_eltorito&#34;&gt;grub binary&lt;/a&gt;.
2042You can read more about this program on &lt;a href=&#34;https://github.com/littleosbook/littleosbook&#34;&gt;https://github.com/littleosbook/littleosbook&lt;/a&gt;.&lt;/p&gt;
2043&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ wget -O iso/boot/grub/stage2_eltorito https://github.com/littleosbook/littleosbook/raw/master/files/stage2_eltorito
2044&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ tree iso/boot/
2045&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;iso/boot/
2046&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;├── bzImage
2047&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;├── grub
2048&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;│   ├── menu.lst
2049&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;│   └── stage2_eltorito
2050&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;└── initramfs
2051&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Let&#39;s copy files into proper folders.&lt;/p&gt;
2052&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cp stage2_eltorito iso/boot/grub/
2053&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cp bin/bzImage iso/boot/
2054&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cp bin/initramfs iso/boot/
2055&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Lets create a GRUB config file at &lt;code&gt;nano iso/boot/grub/menu.lst&lt;/code&gt; with contents.&lt;/p&gt;
2056&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;default=&lt;span style=&#34;color:#a31515&#34;&gt;0&lt;/span&gt;
2057&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;timeout=&lt;span style=&#34;color:#a31515&#34;&gt;5&lt;/span&gt;
2058&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2059&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;title GoAsPID1
2060&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;kernel /boot/bzImage
2061&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;initrd /boot/initramfs
2062&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Let&#39;s create iso file by using genisoimage:&lt;/p&gt;
2063&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;genisoimage -R &lt;span style=&#34;color:#a31515&#34;&gt;\
2064&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -b boot/grub/stage2_eltorito &lt;span style=&#34;color:#a31515&#34;&gt;\
2065&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -no-emul-boot &lt;span style=&#34;color:#a31515&#34;&gt;\
2066&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -boot-load-size 4 &lt;span style=&#34;color:#a31515&#34;&gt;\
2067&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -A os &lt;span style=&#34;color:#a31515&#34;&gt;\
2068&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -input-charset utf8 &lt;span style=&#34;color:#a31515&#34;&gt;\
2069&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -quiet &lt;span style=&#34;color:#a31515&#34;&gt;\
2070&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -boot-info-table &lt;span style=&#34;color:#a31515&#34;&gt;\
2071&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -o GoAsPID1.iso &lt;span style=&#34;color:#a31515&#34;&gt;\
2072&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; iso
2073&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This will produce &lt;code&gt;GoAsPID1.iso&lt;/code&gt; which you can use with &lt;a href=&#34;https://www.virtualbox.org/&#34;&gt;Virtualbox&lt;/a&gt;
2074or &lt;a href=&#34;https://apps.gnome.org/app/org.gnome.Boxes/&#34;&gt;Gnome Boxes&lt;/a&gt;.&lt;/p&gt;
2075&lt;p&gt;&lt;video src=&#34;/assets/pid1/boxes.mp4&#34; controls&gt;&lt;/video&gt;&lt;/p&gt;
2076&lt;h2 id=&#34;is-running-applications-as-pid-1-even-worth-it&#34;&gt;Is running applications as PID 1 even worth it?&lt;/h2&gt;
2077&lt;p&gt;Well, the answer to this is not as simple as one would think. Sometimes it is
2078and sometimes it&#39;s not. For embedded systems and very specialized applications
2079it is worth for sure. But in normal uses, I don&#39;t think so. It was an interesting
2080exercise in compiling kernels and looking at the guts of the Linux kernel,
2081but sticking to containers for most of the things is a better option in my
2082opinion.&lt;/p&gt;
2083&lt;p&gt;An interesting experiment would be creating an image that supports networking
2084and could be deployed to AWS as an EC2 instance and observing how it fares.
2085But in that case, we would need to write some sort of supervisor that would
2086run on a separate EC2 that would check if other EC2 instances are running
2087properly. Remember that if your application fails, kernel panics and the
2088whole machine is inoperable in this case.&lt;/p&gt;
2089</content:encoded>
2090 </item>
2091
2092
2093
2094 <item>
2095 <title>Debian based riced up distribution for Developers and DevOps folks</title>
2096 <link>https://mitjafelicijan.com/debian-based-riced-up-distribution-for-developers-and-devops-folks.html</link>
2097 <pubDate>Fri, 03 Dec 2021 12:00:00 &#43;0200</pubDate>
2098 <guid>https://mitjafelicijan.com/debian-based-riced-up-distribution-for-developers-and-devops-folks.html</guid>
2099 <description>IntroductionI have been using Ubuntu for quite a longtime now.</description>
2100 <content:encoded>&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
2101&lt;p&gt;I have been using &lt;a href=&#34;https://ubuntu.com/&#34;&gt;Ubuntu&lt;/a&gt; for quite a longtime now. I have
2102used &lt;a href=&#34;https://www.debian.org/&#34;&gt;Debian&lt;/a&gt; in the past and
2103&lt;a href=&#34;https://manjaro.org/&#34;&gt;Manjaro&lt;/a&gt;. Also had &lt;a href=&#34;https://archlinux.org/&#34;&gt;Arch&lt;/a&gt; for
2104some time and even ran &lt;a href=&#34;https://www.gentoo.org/&#34;&gt;Gentoo&lt;/a&gt; way back.&lt;/p&gt;
2105&lt;p&gt;What I learned from all this is that I prefer running a bit older versions and
2106having them be stable than run bleeding edge rolling release. For that reason, I
2107stuck with Ubuntu for a couple of years now. I am also at a point in my life
2108where I just don&#39;t care what is cool or hip anymore. I just want a stable system
2109that doesn&#39;t get in my way.&lt;/p&gt;
2110&lt;p&gt;During all this, I noticed that these distributions were getting very bloated
2111and a lot of software got included that I usually uninstall on fresh
2112installation. Maybe this is my OCD speaking, but why do I have to give fresh
2113installation min 1 GB of ram out of the box just to have a blank screen in front
2114of me? I get it, there are many things included in the distro to make my life
2115easier. I understand. But at this point I have a feeling that modern Linux
2116distributions are becoming similar to &lt;a href=&#34;https://devhumor.com/content/uploads/images/August2017/node-modules.jpg&#34;&gt;Node.js project with
2117node_modules&lt;/a&gt;.
2118Just a crazy number of packages serving very little or no purpose, just
2119supporting other software.&lt;/p&gt;
2120&lt;p&gt;I felt I needed a fresh start. To start over with something minimal and clean.
2121Something that would put a little more joy into using a computer again.&lt;/p&gt;
2122&lt;p&gt;For the first version, I wanted to target the following machines I have at home
2123that I want this thing to work on.&lt;/p&gt;
2124&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# My main stationary work machine&lt;/span&gt;
2125&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Resolution: 3840x1080 (Super Ultrawide Monitor 32:9)
2126&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;CPU: Intel i7-8700 (12) @ 4.600GHz
2127&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;GPU: AMD ATI Radeon RX 470/480/570/570X/580/580X/590
2128&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Memory: 32020MiB
2129&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# Thinkpad x220 for testing things and goofing around&lt;/span&gt;
2130&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Resolution: 1366x768
2131&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;CPU: Intel i5-2520M (4) @ 3.200GHz
2132&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;GPU: Intel 2nd Generation Core Processor Family
2133&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Memory: 15891MiB
2134&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;how-should-i-approach-this&#34;&gt;How should I approach this?&lt;/h2&gt;
2135&lt;p&gt;I knew I wanted to use &lt;a href=&#34;https://www.debian.org/CD/netinst/&#34;&gt;minimal Debian netinst
2136&lt;/a&gt; for the base to give myself a head
2137start. No reason to go through changing the installer and also testing all that
2138behemoth of a thing. So, some sort of ricing was the only logical option to get
2139this thing of the grounds somewhat quickly.&lt;/p&gt;
2140&lt;blockquote&gt;
2141&lt;p&gt;&lt;strong&gt;What is ricing anyway?&lt;/strong&gt;
2142The term “RICE” stands for Race Inspired Cosmetic Enhancement. A group of
2143people (could be one, idk) decided to see if they could tweak their own
2144distros like they/others did their cars. This gave rise to a community of
2145Linux/Unix enthusiasts trying to make their distros look cooler and better
2146than others... For more information, read this article
2147&lt;a href=&#34;https://pesos.github.io/2020/07/14/what-is-ricing.html&#34;&gt;What in the world is ricing!?&lt;/a&gt;.&lt;/p&gt;
2148&lt;/blockquote&gt;
2149&lt;p&gt;I didn&#39;t want this to just be a set of config files for theming purpose. I
2150wanted this to include a set of pre-installed tools and services that are being
2151used all the time by a modern developer. Theming is just a tiny part of it.
2152Fonts being applied across the distro and things like that.&lt;/p&gt;
2153&lt;p&gt;First, I choose terminal installer and left it to load additional components.
2154Avoid using graphical installer in this case.&lt;/p&gt;
2155&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/install-00.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2156&lt;p&gt;After that I selected hostname and created a normal user and set password for
2157that user and root user and choose guided mode for disk partitioning.&lt;/p&gt;
2158&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/install-01.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2159&lt;p&gt;I left it run to install all the things required for the base system and opted
2160out of scanning additional media for use by the package manager. Those will be
2161downloaded from the internet during installation.&lt;/p&gt;
2162&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/install-02.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2163&lt;p&gt;I opted out of the popularity contest, and &lt;strong&gt;now comes the important part&lt;/strong&gt;.
2164Uncheck all the boxes in Software selection and only leave &#39;standard system
2165utilities&#39;. I also left an SSH server, so I was able to log in to the machine
2166from my main PC.&lt;/p&gt;
2167&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/install-03.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2168&lt;p&gt;At this point, I installed GRUB bootloader on the disk where I installed the
2169system.&lt;/p&gt;
2170&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/install-04.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2171&lt;p&gt;That concluded the installation of base Debian and after restarting the computer
2172I was prompted with the login screen.&lt;/p&gt;
2173&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/install-05.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2174&lt;p&gt;Now that I had the base installation, it was time to choose what software do I
2175want to include in this so-called distribution. I wanted out of the box
2176developer experience, so I had plenty to choose.&lt;/p&gt;
2177&lt;p&gt;Let&#39;s not waste time and go through the list.&lt;/p&gt;
2178&lt;h2 id=&#34;desktop-environments&#34;&gt;Desktop environments&lt;/h2&gt;
2179&lt;p&gt;I have been using &lt;a href=&#34;https://www.gnome.org/&#34;&gt;Gnome&lt;/a&gt; for my whole Linux life. From
2180version 2 forward. It&#39;s been quite a ride. I hated version 3 when it came out
2181and replaced version 2. But I got used to it. And now with version 40&#43; they also
2182made couple of changes which I found both frustrating and presently surprised.&lt;/p&gt;
2183&lt;p&gt;The amount of vertical space you loose because of the beefy title bars on
2184windows is ridiculous. And then in case of
2185&lt;a href=&#34;https://gnunn1.github.io/tilix-web/&#34;&gt;Tilix&lt;/a&gt; you also have tabs, and you are
2186100px deep. Vertical space is one of the most important things for a
2187developer. The more real estate you have, the more code you can have in a
2188viewport.&lt;/p&gt;
2189&lt;p&gt;But on the other hand, I still love how Gnome feels and looks. I gotta give them
2190that. They really are trying to make Gnome feel unified and modern.&lt;/p&gt;
2191&lt;p&gt;Regardless of all the nice things Gnome has, I was looking at the tiling window
2192managers for some time, but never had the nerve to actually go with it. But now
2193was the ideal time to give it a go. No guts, no glory kind of a thing.&lt;/p&gt;
2194&lt;p&gt;One of the requirements for me was easy custom layouts because I use a really
2195strange monitor with aspect ratio of 32:9. So relying on included layouts most
2196of them have is a non-starter.&lt;/p&gt;
2197&lt;p&gt;What I was doing in Gnome was having windows in a layout like the diagram
2198below. This is my common practice. And if you look at it you can clearly see I
2199was replicating tiling window manager setup in Gnome.&lt;/p&gt;
2200&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/layout.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2201&lt;p&gt;That made me look into a bunch of tiling window managers and then tested them
2202out. Candidates I was looking at were:&lt;/p&gt;
2203&lt;ul&gt;
2204&lt;li&gt;&lt;a href=&#34;https://i3wm.org/&#34;&gt;i3&lt;/a&gt;&lt;/li&gt;
2205&lt;li&gt;&lt;a href=&#34;https://github.com/baskerville/bspwm&#34;&gt;bspwm&lt;/a&gt;&lt;/li&gt;
2206&lt;li&gt;&lt;a href=&#34;https://awesomewm.org/index.html&#34;&gt;awesome&lt;/a&gt;&lt;/li&gt;
2207&lt;li&gt;&lt;a href=&#34;https://xmonad.org/&#34;&gt;XMonad&lt;/a&gt;&lt;/li&gt;
2208&lt;li&gt;&lt;a href=&#34;https://swaywm.org/&#34;&gt;sway&lt;/a&gt;&lt;/li&gt;
2209&lt;li&gt;&lt;a href=&#34;http://www.qtile.org/&#34;&gt;Qtile&lt;/a&gt;&lt;/li&gt;
2210&lt;li&gt;&lt;a href=&#34;https://dwm.suckless.org/&#34;&gt;dwm&lt;/a&gt;&lt;/li&gt;
2211&lt;/ul&gt;
2212&lt;p&gt;You can also check article &lt;a href=&#34;https://www.tecmint.com/best-tiling-window-managers-for-linux/&#34;&gt;13 Best Tiling Window Managers for
2213Linux&lt;/a&gt; I was
2214referencing while testing them out.&lt;/p&gt;
2215&lt;p&gt;While all of them provided what I needed, I liked i3 the most. What particular
2216caught my eye was the ease to use and tree based layouts which allows flexible
2217layouts. I know others can be set up also to have custom layouts other than&lt;br /&gt;
2218spiral, dwindle etc. I think i3 is a good entry-level window manager for
2219somebody like me.&lt;/p&gt;
2220&lt;h2 id=&#34;batteries-included&#34;&gt;Batteries included&lt;/h2&gt;
2221&lt;p&gt;The source for the whole thing is located on Github
2222&lt;a href=&#34;https://github.com/mitjafelicijan/dfd-rice&#34;&gt;https://github.com/mitjafelicijan/dfd-rice&lt;/a&gt;.&lt;/p&gt;
2223&lt;p&gt;Currenly included:&lt;/p&gt;
2224&lt;ul&gt;
2225&lt;li&gt;&lt;code&gt;non-free&lt;/code&gt; (enables non-free packages in apt)&lt;/li&gt;
2226&lt;li&gt;&lt;code&gt;sudo&lt;/code&gt; (adds sudo and adds user to sudo group)&lt;/li&gt;
2227&lt;li&gt;&lt;code&gt;essentials&lt;/code&gt; (gcc, htop, zip, curl, etc...)&lt;/li&gt;
2228&lt;li&gt;&lt;code&gt;wifi&lt;/code&gt; (network manager nmtui)&lt;/li&gt;
2229&lt;li&gt;&lt;code&gt;desktop&lt;/code&gt; (i3, dmenu, fonts, configurations)&lt;/li&gt;
2230&lt;li&gt;&lt;code&gt;pulseaudio&lt;/code&gt; (pulseaudio with pavucontrol)&lt;/li&gt;
2231&lt;li&gt;&lt;code&gt;code-editors&lt;/code&gt; (vim, micro, vscode)&lt;/li&gt;
2232&lt;li&gt;&lt;code&gt;ohmybash&lt;/code&gt; (make bash pretty)&lt;/li&gt;
2233&lt;li&gt;&lt;code&gt;file-managers&lt;/code&gt; (mc)&lt;/li&gt;
2234&lt;li&gt;&lt;code&gt;git-ui&lt;/code&gt; (terminal git gui)&lt;/li&gt;
2235&lt;li&gt;&lt;code&gt;meld&lt;/code&gt; (diff tool)&lt;/li&gt;
2236&lt;li&gt;&lt;code&gt;profiling&lt;/code&gt; (kcachegrind, valgrind, strace, ltrace)&lt;/li&gt;
2237&lt;li&gt;&lt;code&gt;browsers&lt;/code&gt; (brave, firefox, chromium)&lt;/li&gt;
2238&lt;li&gt;programming languages:
2239&lt;ul&gt;
2240&lt;li&gt;&lt;code&gt;python&lt;/code&gt;&lt;/li&gt;
2241&lt;li&gt;&lt;code&gt;golang&lt;/code&gt;&lt;/li&gt;
2242&lt;li&gt;&lt;code&gt;nodejs&lt;/code&gt;&lt;/li&gt;
2243&lt;li&gt;&lt;code&gt;rust&lt;/code&gt;&lt;/li&gt;
2244&lt;li&gt;&lt;code&gt;nim&lt;/code&gt;&lt;/li&gt;
2245&lt;li&gt;&lt;code&gt;php&lt;/code&gt;&lt;/li&gt;
2246&lt;li&gt;&lt;code&gt;ruby&lt;/code&gt;&lt;/li&gt;
2247&lt;/ul&gt;
2248&lt;/li&gt;
2249&lt;li&gt;&lt;code&gt;docker&lt;/code&gt; (with docker-compose)&lt;/li&gt;
2250&lt;li&gt;&lt;code&gt;ansible&lt;/code&gt;&lt;/li&gt;
2251&lt;/ul&gt;
2252&lt;p&gt;Install script also allows you to install only specific packages (example for:
2253essentials ohmybash docker rust).&lt;/p&gt;
2254&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;su - root &lt;span style=&#34;color:#a31515&#34;&gt;\
2255&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; bash -c &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;$(&lt;/span&gt;wget -q https://raw.github.com/mitjafelicijan/dfd-rice/master/tools/install.sh -O -&lt;span style=&#34;color:#00f&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt; -- &lt;span style=&#34;color:#a31515&#34;&gt;\
2256&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; essentials ohmybash docker rust
2257&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Currently, most of these recipes use what Debian and this is totally fine with
2258me since I never use bleeding edge features of a package. But if something major
2259would come to light, I will replace it with a possible compilation script or
2260something similar.&lt;/p&gt;
2261&lt;p&gt;This is some of the output from the installation script.&lt;/p&gt;
2262&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/script.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2263&lt;p&gt;Let&#39;s take a look at some examples in the installation script.&lt;/p&gt;
2264&lt;h3 id=&#34;docker-recipe&#34;&gt;Docker recipe&lt;/h3&gt;
2265&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# docker&lt;/span&gt;
2266&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print_header &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Installing Docker&amp;#34;&lt;/span&gt;
2267&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --yes --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
2268&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;echo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;deb [arch=&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;$(&lt;/span&gt;dpkg --print-architecture&lt;span style=&#34;color:#00f&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian &lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;$(&lt;/span&gt;lsb_release -cs&lt;span style=&#34;color:#00f&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; stable&amp;#34;&lt;/span&gt; | tee /etc/apt/sources.list.d/docker.list &amp;gt; /dev/null
2269&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;apt update
2270&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;apt -y install docker-ce docker-ce-cli containerd.io docker-compose
2271&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2272&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;systemctl start docker
2273&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;systemctl enable docker
2274&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;systemctl status docker --no-pager
2275&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2276&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;/sbin/usermod -aG docker $USERNAME
2277&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;making-bash-pretty&#34;&gt;Making bash pretty&lt;/h3&gt;
2278&lt;p&gt;I really like &lt;a href=&#34;https://ohmyz.sh/&#34;&gt;Oh My Zsh&lt;/a&gt;, but I don&#39;t like zsh shell. When
2279I used it, I constantly needed to be aware of it and running bash scripts was a
2280pain. So, I was really delighted when I found out that a version for bash
2281existed called &lt;a href=&#34;https://ohmybash.nntoan.com/&#34;&gt;Oh My Bash&lt;/a&gt;. Let&#39;s take a look at
2282the recipe for installing it.&lt;/p&gt;
2283&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# ohmybash&lt;/span&gt;
2284&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print_header &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Enabling OhMyBash&amp;#34;&lt;/span&gt;
2285&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo -u $USERNAME sh -c &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;$(&lt;/span&gt;curl -fsSL https://raw.github.com/ohmybash/oh-my-bash/master/tools/install.sh&lt;span style=&#34;color:#00f&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt; &amp;amp;
2286&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;T1=&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;!&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;
2287&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;wait &lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;T1&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;
2288&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Because OhMyBash does &lt;code&gt;exec bash&lt;/code&gt; at the end, this traps our script inside
2289another shell and our script cannot continue. For that reason, I executed this
2290in background. But that presents a new problem. Because this is executed in
2291background, we lose track of progress naturally. And that strange trick with
2292&lt;code&gt;T1=${!}&lt;/code&gt; and &lt;code&gt;wait ${T1}&lt;/code&gt; waits for the background process to finish before
2293continuing to another task in bash script.&lt;/p&gt;
2294&lt;p&gt;Check &lt;a href=&#34;https://www.cloudsavvyit.com/12277/how-to-use-multi-threaded-processing-in-bash-scripts/&#34;&gt;Multi-Threaded Processing in Bash Scripts&lt;/a&gt;
2295for more details.&lt;/p&gt;
2296&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;
2297&lt;p&gt;Take a look at
2298&lt;a href=&#34;https://github.com/mitjafelicijan/dfd-rice/blob/develop/tools/install.sh&#34;&gt;https://github.com/mitjafelicijan/dfd-rice/blob/develop/tools/install.sh&lt;/a&gt; script
2299to get familiar with it. This is just a first iteration and I will continue to
2300update it because I need this in my life.&lt;/p&gt;
2301&lt;p&gt;The current version boots in 4s to the login prompt, and after you log in, the
2302desktop environment loads in 2s. So, its fast, very fast. And on clean boot, I
2303measured ~230 MB of RAM usage.&lt;/p&gt;
2304&lt;p&gt;And this is how it looks with two terminals side by side. I really like the
2305simplicity and clean interface. I will polish the colors and stuff like that,
2306but I really do like the results.&lt;/p&gt;
2307&lt;p&gt;&lt;img src=&#34;/assets/dfd-rice/desktop.png&#34; alt=&#34;&#34; /&gt;&lt;/p&gt;
2308</content:encoded>
2309 </item>
2310
2311
2312
2313 <item>
2314 <title>List of essential Linux commands for server management</title>
2315 <link>https://mitjafelicijan.com/linux-cheatsheet.html</link>
2316 <pubDate>Sun, 01 Aug 2021 12:00:00 &#43;0200</pubDate>
2317 <guid>https://mitjafelicijan.com/linux-cheatsheet.html</guid>
2318 <description>Generate SSH keyssh-keygen -t ed25519 -C &amp;#34;your_email@example.</description>
2319 <content:encoded>&lt;p&gt;&lt;strong&gt;Generate SSH key&lt;/strong&gt;&lt;/p&gt;
2320&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh-keygen -t ed25519 -C &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;your_email@example.com&amp;#34;&lt;/span&gt;
2321&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2322&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# when no support for Ed25519 present&lt;/span&gt;
2323&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh-keygen -t rsa -b 4096 -C &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;your_email@example.com&amp;#34;&lt;/span&gt;
2324&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Note: By default SSH keys get stored to &lt;code&gt;/home/&amp;lt;username&amp;gt;/.ssh/&lt;/code&gt; folder.&lt;/p&gt;
2325&lt;p&gt;&lt;strong&gt;Login to host via SSH&lt;/strong&gt;&lt;/p&gt;
2326&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# connect to host as your local username&lt;/span&gt;
2327&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh host
2328&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2329&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# connect to host as user&lt;/span&gt;
2330&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh &amp;lt;user&amp;gt;@&amp;lt;host&amp;gt;
2331&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2332&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# connect to host using port&lt;/span&gt;
2333&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh -p &amp;lt;port&amp;gt; &amp;lt;user&amp;gt;@&amp;lt;host&amp;gt;
2334&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Execute command on a server through SSH&lt;/strong&gt;&lt;/p&gt;
2335&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# execute one command&lt;/span&gt;
2336&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh root@100.100.100.100 &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;ls /root&amp;#34;&lt;/span&gt;
2337&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2338&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# execute many commands&lt;/span&gt;
2339&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh root@100.100.100.100 &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;cd /root;touch file.txt&amp;#34;&lt;/span&gt;
2340&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays currently logged in users in the system&lt;/strong&gt;&lt;/p&gt;
2341&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;w
2342&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays Linux system information&lt;/strong&gt;&lt;/p&gt;
2343&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;uname
2344&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays kernel release information&lt;/strong&gt;&lt;/p&gt;
2345&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;uname -r
2346&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Shows the system hostname&lt;/strong&gt;&lt;/p&gt;
2347&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;hostname
2348&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Shows system reboot history&lt;/strong&gt;&lt;/p&gt;
2349&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;last reboot
2350&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays information about the user&lt;/strong&gt;&lt;/p&gt;
2351&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo apt install finger
2352&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;finger &amp;lt;username&amp;gt;
2353&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays IP addresses and all the network interfaces&lt;/strong&gt;&lt;/p&gt;
2354&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ip addr show
2355&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Downloads a file from an online source&lt;/strong&gt;&lt;/p&gt;
2356&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;wget https://example.com/example.tgz
2357&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Note: If URL contains ?, &amp;amp; enclose the URL in double quotes.&lt;/p&gt;
2358&lt;p&gt;&lt;strong&gt;Compress a file with gzip&lt;/strong&gt;&lt;/p&gt;
2359&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# will not keep the original file&lt;/span&gt;
2360&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;gzip file.txt
2361&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2362&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# will keep the original file&lt;/span&gt;
2363&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;gzip --keep file.txt
2364&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Interactive disk usage analyzer&lt;/strong&gt;&lt;/p&gt;
2365&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo apt install ncdu
2366&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2367&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ncdu
2368&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ncdu &amp;lt;path/to/directory&amp;gt;
2369&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Install Node.js using the Node Version Manager&lt;/strong&gt;&lt;/p&gt;
2370&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
2371&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;source ~/.bashrc
2372&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2373&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;nvm install v13
2374&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Too long; didn&#39;t read&lt;/strong&gt;&lt;/p&gt;
2375&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;npm install -g tldr
2376&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2377&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;tldr tar
2378&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Combine all Nginx access logs to one big log file&lt;/strong&gt;&lt;/p&gt;
2379&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;zcat -f /var/log/nginx/access.log* &amp;gt; /var/log/nginx/access-all.log
2380&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Set up Redis server&lt;/strong&gt;&lt;/p&gt;
2381&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo apt install redis-server redis-tools
2382&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2383&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# check if server is running&lt;/span&gt;
2384&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo service redis status
2385&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2386&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# set and get a key value&lt;/span&gt;
2387&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;redis-cli set mykey myvalue
2388&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;redis-cli get mykey
2389&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2390&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# interactive shell&lt;/span&gt;
2391&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;redis-cli
2392&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Generate statistics of your webserver&lt;/strong&gt;&lt;/p&gt;
2393&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo apt install goaccess
2394&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2395&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# check if installed&lt;/span&gt;
2396&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;goaccess -v
2397&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2398&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# combine logs&lt;/span&gt;
2399&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;zcat -f /var/log/nginx/access.log* &amp;gt; /var/log/nginx/access-all.log
2400&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2401&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# export to single html&lt;/span&gt;
2402&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;goaccess &lt;span style=&#34;color:#a31515&#34;&gt;\
2403&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --log-file=/var/log/nginx/access-all.log &lt;span style=&#34;color:#a31515&#34;&gt;\
2404&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --log-format=COMBINED &lt;span style=&#34;color:#a31515&#34;&gt;\
2405&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --exclude-ip=0.0.0.0 &lt;span style=&#34;color:#a31515&#34;&gt;\
2406&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --ignore-crawlers &lt;span style=&#34;color:#a31515&#34;&gt;\
2407&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --real-os &lt;span style=&#34;color:#a31515&#34;&gt;\
2408&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --output=/var/www/html/stats.html
2409&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2410&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# cleanup afterwards&lt;/span&gt;
2411&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;rm /var/log/nginx/access-all.log
2412&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Search for a given pattern in files&lt;/strong&gt;&lt;/p&gt;
2413&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;grep -r ‘pattern’ files
2414&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Find proccess ID for a specific program&lt;/strong&gt;&lt;/p&gt;
2415&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;pgrep nginx
2416&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Print name of current/working directory&lt;/strong&gt;&lt;/p&gt;
2417&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;pwd
2418&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Creates a blank new file&lt;/strong&gt;&lt;/p&gt;
2419&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;touch newfile.txt
2420&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays first lines in a file&lt;/strong&gt;&lt;/p&gt;
2421&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -n &amp;lt;x&amp;gt; presents the number of lines (10 by default)&lt;/span&gt;
2422&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;head -n 20 somefile.txt
2423&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays last lines in a file&lt;/strong&gt;&lt;/p&gt;
2424&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -n &amp;lt;x&amp;gt; presents the number of lines (10 by default)&lt;/span&gt;
2425&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;tail -n 20 somefile.txt
2426&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2427&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -f follows the changes in file (doesn&amp;#39;t closes)&lt;/span&gt;
2428&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;tail -f somefile.txt
2429&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Count lines in a file&lt;/strong&gt;&lt;/p&gt;
2430&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;wc -l somefile.txt
2431&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Find all instances of the file&lt;/strong&gt;&lt;/p&gt;
2432&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo apt install mlocate
2433&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2434&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;locate somefile.txt
2435&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Find file names that begin with ‘index’ in /home folder&lt;/strong&gt;&lt;/p&gt;
2436&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;find /home/ -name &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;index&amp;#34;&lt;/span&gt;
2437&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Find files larger than 100MB in the home folder&lt;/strong&gt;&lt;/p&gt;
2438&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;find /home -size &#43;100M
2439&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays block devices related information&lt;/strong&gt;&lt;/p&gt;
2440&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;lsblk
2441&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays free space on mounted systems&lt;/strong&gt;&lt;/p&gt;
2442&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;df -h
2443&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays free and used memory in the system&lt;/strong&gt;&lt;/p&gt;
2444&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;free -h
2445&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Displays all active listening ports&lt;/strong&gt;&lt;/p&gt;
2446&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo apt install net-tools
2447&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2448&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;netstat -pnltu
2449&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Kill a process violently&lt;/strong&gt;&lt;/p&gt;
2450&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;kill -9 &amp;lt;pid&amp;gt;
2451&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;List files opened by user&lt;/strong&gt;&lt;/p&gt;
2452&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;lsof -u &amp;lt;user&amp;gt;
2453&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Execute &amp;quot;df -h&amp;quot;, showing periodic updates&lt;/strong&gt;&lt;/p&gt;
2454&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -n 1 means every second&lt;/span&gt;
2455&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;watch -n 1 df -h
2456&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;</content:encoded>
2457 </item>
2458
2459
2460
2461 <item>
2462 <title>My journey from being an internet über consumer to being a full hominum again</title>
2463 <link>https://mitjafelicijan.com/from-internet-consumer-to-full-hominum-again.html</link>
2464 <pubDate>Fri, 30 Jul 2021 12:00:00 &#43;0200</pubDate>
2465 <guid>https://mitjafelicijan.com/from-internet-consumer-to-full-hominum-again.html</guid>
2466 <description>It&amp;#39;s been almost a year since I started purging all my online accounts andgoing down this rabbit hole of being almost independent of the current internetmachine.</description>
2467 <content:encoded>&lt;p&gt;It&#39;s been almost a year since I started purging all my online accounts and
2468going down this rabbit hole of being almost independent of the current internet
2469machine. Even though I initially thought that I will have problems adapting,
2470I was pleasantly surprised that the transition went so smoothly. Even better,
2471it brought many benefits to my life. Such as increased focus, less stress
2472about trivial things, etc.&lt;/p&gt;
2473&lt;p&gt;It all started with me doing small changes like unsubscribing from emails that I
2474have either subscribed to by accepting terms and conditions. Or even some more
2475malicious emails that I was getting because I was on a shared mailing list. And
2476the later ones I hate the most of all. How the hell do they keep sharing my
2477email and sending me unsolicited emails and get away with it? I have a suspicion
2478that these marketing people share an Excel file between them and keep
2479resubscribing me when they import lists into Mailchimp or similar software.&lt;/p&gt;
2480&lt;p&gt;It&#39;s fascinating to see how much crap you get subscribed to when you are not
2481paying attention. It got so bad that my primary Gmail address is a full of junk
2482and need constant monitoring and cleaning up. And because I want to have Inbox
2483Zero, this presents an additional problem for me.&lt;/p&gt;
2484&lt;p&gt;The stress that email presented for me didn&#39;t occur to me for a long time. I was
2485noticing that I was unable to go through one single hour without hysterically
2486refreshing email. And if somebody wrote me something, I needed to see it right
2487then, even though I didn&#39;t immediately reply to it. I can only describe this
2488with FOMO (fear of missing out). I have no other explanation than that. It was
2489crippling, and I was constantly context switching, which I will address further
2490down this post in more details.&lt;/p&gt;
2491&lt;p&gt;This was one of the reasons why I spawned up my personal email server, and I am
2492using it now as my primary and person email. I still have Gmail as my “junk”
2493email that I use for throw away stuff. I log in to Gmail once a week and check
2494if there are any important emails that I got, but apart from that, it&#39;s sitting
2495dormant and collecting dust.&lt;/p&gt;
2496&lt;p&gt;The more I was watching the world loose it&#39;s self with allowing anti freedom
2497things to happen to it, the more I started to realize that something has to
2498change. I don&#39;t have the power to change the world. And I also don&#39;t have a
2499grandiose opinion of myself to even think to try it. But what I can do is to not
2500subscribe to this consumer way of thinking. I will not be complicit in this. My
2501moral and ethical stances won&#39;t allow it. So, this brings us to the second part
2502of my journey.&lt;/p&gt;
2503&lt;p&gt;I was using all these 3rd party services because I was either lazy or OK with
2504the drawbacks of them. I watched these services and companies became more and
2505privacy policies and everybody is OK with accepting them, and they pray on that
2506more evil. It is evil if you sell your user&#39;s data in this manner. Nobody reads
2507flaw in human nature. I really hate the hypocrisy they manage to muster. These
2508companies prey on our laziness, and we are at fault here. Nobody else. And I
2509truly understand the reasons why we rather accept and move on, and not object
2510and have our lives a little more difficult. They have perfected this through
2511years of small changes that make us a little more dependent on them. You could
2512not convince a person to give away all his rights and data in one day. This was
2513gradual and slow. And it caught us all in surprise. When I really stopped and
2514thought about it, I felt repulsed. By really stopping and thinking about it, I
2515really mean stopping and thinking about it. Thoroughly and in depth.&lt;/p&gt;
2516&lt;p&gt;Each step I took depleted my character a bit more. Like I was trading myself bit
2517by bit without understanding what it all meant. What it meant to be a full
2518person, not divided by all this bought attention they want from me. They don&#39;t
2519just get your data, but they also take your attention away from you. They
2520scatter your and go with the divide and conquer tactic from there. And a person
2521divided is a person not fully there. Not at the moment. Not alive fully.&lt;/p&gt;
2522&lt;p&gt;I was unable to form long thoughts. Well, I thought I was. But now that I see
2523what being a full person is again, I can see that I was not at my 100% back
2524then.&lt;/p&gt;
2525&lt;p&gt;A revolt was inevitable. There was no other way of continuing my story without
2526it. Without taking back my attention, my thoughts, my time, and my privacy,
2527regardless of how too late it maybe is.&lt;/p&gt;
2528&lt;p&gt;This has nothing to do with conspiracy theories. Even less with changing the
2529world. All I wanted was to get my life back in order and not waste the energy
2530that could be spent in other, better places.&lt;/p&gt;
2531&lt;p&gt;I started reading more. I can focus now fully on things I work on. Furthermore,
2532I have the mental acuity that I never had before. My mind feels sharp. I don&#39;t
2533get angry so much. I can cherish the finer things in life now without the need
2534to interpret them intellectually. Not only that, but I have a feeling of
2535belonging again. Sense of purpose has returned with a vengeance. And I can now
2536help people without depleting myself.&lt;/p&gt;
2537&lt;p&gt;The last step so far was to finish closing all the remaining online accounts
2538that I still had. And when I was thinking what value they bring me, I wasn&#39;t
2539surprised that the answer was none. I wasn&#39;t logging in them and using them. I
2540stopped being afraid of FOMO. If somebody wants to get in contact me, they will
2541find a way. I am one search away.&lt;/p&gt;
2542&lt;p&gt;We are not beholden to anybody. Our lives are our own. So dare yourself to
2543delete Facebook, LinkedIn. To unsubscribe. Dare yourself to take your time and
2544attention back. Use that time and energy to go for a walk without thinking about
2545work. Read a book instead of reading comment on social media that you will
2546forget in an hour. Enrich your life instead of wasting it. It only requires a
2547small step. And you will feel the benefits immediately. Lose the weight of the
2548world that is crushing you without your consent.&lt;/p&gt;
2549</content:encoded>
2550 </item>
2551
2552
2553
2554 <item>
2555 <title>Simple world clock with eInk display and Raspberry Pi Zero</title>
2556 <link>https://mitjafelicijan.com/simple-world-clock-with-eiink-display-and-raspberry-pi-zero.html</link>
2557 <pubDate>Sat, 26 Jun 2021 12:00:00 &#43;0200</pubDate>
2558 <guid>https://mitjafelicijan.com/simple-world-clock-with-eiink-display-and-raspberry-pi-zero.html</guid>
2559 <description>Our team is spread across the world, from the USA all the way to Australia, sohaving some sort of world clock makes sense.</description>
2560 <content:encoded>&lt;p&gt;Our team is spread across the world, from the USA all the way to Australia, so
2561having some sort of world clock makes sense.&lt;/p&gt;
2562&lt;p&gt;Currently, I am using an extension for Gnome called &lt;a href=&#34;https://extensions.gnome.org/extension/2657/timezones-extension/&#34;&gt;Timezone
2563extension&lt;/a&gt;,
2564and it serves the purpose quite well.&lt;/p&gt;
2565&lt;p&gt;But I also have a bunch of electronics that I bought through the time, and I am
2566not using any of them, and it&#39;s time to stop hording this stuff and use it in a
2567project.&lt;/p&gt;
2568&lt;p&gt;A while ago I bought a small eInk display &lt;a href=&#34;https://shop.pimoroni.com/products/inky-phat?variant=12549254217811&#34;&gt;Inky
2569pHAT&lt;/a&gt; and I
2570have a bunch of &lt;a href=&#34;https://www.raspberrypi.org/products/raspberry-pi-zero/&#34;&gt;Raspberry Pi&#39;s
2571Zero&lt;/a&gt; lying around that
2572I really need to use.&lt;/p&gt;
2573&lt;p&gt;&lt;img src=&#34;/assets/world-clock/hardware.jpg&#34; alt=&#34;Inky pHAT, Raspberry Pi Zero&#34; /&gt;&lt;/p&gt;
2574&lt;p&gt;Since the Inky &lt;a href=&#34;https://shop.pimoroni.com/products/inky-phat?variant=12549254217811&#34;&gt;Inky
2575pHAT&lt;/a&gt; is
2576essentially a HAT, it can easily be added on top of the &lt;a href=&#34;https://www.raspberrypi.org/products/raspberry-pi-zero/&#34;&gt;Raspberry Pi
2577Zero&lt;/a&gt;.&lt;/p&gt;
2578&lt;p&gt;First, I installed the necessary software on Raspberry Pi with &lt;code&gt;pip3 install inky&lt;/code&gt;.&lt;/p&gt;
2579&lt;p&gt;And then I created a file &lt;code&gt;clock.py&lt;/code&gt; in home directory &lt;code&gt;/home/pi&lt;/code&gt;.&lt;/p&gt;
2580&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;#!/usr/bin/env python&lt;/span&gt;
2581&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -*- coding: utf-8 -*-&lt;/span&gt;
2582&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2583&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; sys
2584&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; os
2585&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; inky.auto &lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; auto
2586&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; PIL &lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; Image, ImageFont, ImageDraw
2587&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; font_fredoka_one &lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; FredokaOne
2588&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2589&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;clocks = [
2590&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;America/New_York&amp;#39;&lt;/span&gt;,
2591&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Europe/Ljubljana&amp;#39;&lt;/span&gt;,
2592&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Australia/Brisbane&amp;#39;&lt;/span&gt;,
2593&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]
2594&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2595&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;board = auto()
2596&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;board.set_border(board.WHITE)
2597&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;board.rotation = 90
2598&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2599&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;img = Image.new(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;P&amp;#39;&lt;/span&gt;, (board.WIDTH, board.HEIGHT))
2600&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;draw = ImageDraw.Draw(img)
2601&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2602&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;big_font = ImageFont.truetype(FredokaOne, 18)
2603&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;small_font = ImageFont.truetype(FredokaOne, 13)
2604&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2605&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;x = board.WIDTH / 3
2606&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;y = board.HEIGHT / 3
2607&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2608&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;idx = 1
2609&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; clock &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; clocks:
2610&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ctime = os.popen(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;TZ=&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34; date &#43;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%a&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;,%H:%M&amp;#34;&amp;#39;&lt;/span&gt;.format(clock))
2611&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ctime = ctime.read().strip().split(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;,&amp;#39;&lt;/span&gt;)
2612&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; city = clock.split(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;/&amp;#39;&lt;/span&gt;)[1].replace(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;_&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39; &amp;#39;&lt;/span&gt;)
2613&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2614&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; draw.text((15, (idx*y)-y&#43;10), city, fill=board.BLACK, font=small_font)
2615&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; draw.text((110, (idx*y)-y&#43;7), str(ctime[0]), fill=board.BLACK, font=big_font)
2616&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; draw.text((155, (idx*y)-y&#43;7), str(ctime[1]), fill=board.BLACK, font=big_font)
2617&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2618&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; idx &#43;= 1
2619&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2620&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;board.set_image(img)
2621&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;board.show()
2622&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And because eInk displays are rather slow to refresh and the clock requires
2623refreshing only once a minute, this can be done through cronjob.&lt;/p&gt;
2624&lt;p&gt;Before we add this job to cron we need to make &lt;code&gt;clock.py&lt;/code&gt; executable with &lt;code&gt;chmod &#43;x clock.py&lt;/code&gt;.&lt;/p&gt;
2625&lt;p&gt;Then we add a cronjob with &lt;code&gt;crontab -e&lt;/code&gt;.&lt;/p&gt;
2626&lt;pre&gt;&lt;code&gt;* * * * * /home/pi/clock.py
2627&lt;/code&gt;&lt;/pre&gt;
2628&lt;p&gt;So, we end up with a result like this.&lt;/p&gt;
2629&lt;p&gt;&lt;img src=&#34;/assets/world-clock/world-clock.jpg&#34; alt=&#34;World Clock&#34; /&gt;&lt;/p&gt;
2630&lt;p&gt;And for the enclosure that can be 3D printed, but I haven&#39;t yet something like
2631this can be used.&lt;/p&gt;
2632&lt;iframe id=&#34;vs_iframe&#34; src=&#34;https://www.viewstl.com/?embedded&amp;url=https%3A%2F%2Fmitjafelicijan.com%2Fassets%2Fworld-clock%2Fenclosure.stl&amp;color=gray&amp;bgcolor=white&amp;edges=no&amp;orientation=front&amp;noborder=no&#34; style=&#34;border:0;margin:0;width:100%;height:400px;&#34;&gt;&lt;/iframe&gt;
2633&lt;p&gt;You can download my &lt;a href=&#34;/assets/world-clock/enclosure.stl&#34;&gt;STL file for the enclosure
2634here&lt;/a&gt;, but make sure that dimensions make
2635sense and also opening for USB port should be added or just use a drill and some
2636hot glue to make it stick in the enclosure.&lt;/p&gt;
2637</content:encoded>
2638 </item>
2639
2640
2641
2642 <item>
2643 <title>Using GoAccess with Nginx to replace Google Analytics</title>
2644 <link>https://mitjafelicijan.com/using-goaccess-with-nginx-to-replace-google-analytics.html</link>
2645 <pubDate>Mon, 25 Jan 2021 12:00:00 &#43;0200</pubDate>
2646 <guid>https://mitjafelicijan.com/using-goaccess-with-nginx-to-replace-google-analytics.html</guid>
2647 <description>IntroductionI know!</description>
2648 <content:encoded>&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
2649&lt;p&gt;I know! You cannot simply replace Google Analytics with parsing access logs and
2650displaying a couple of charts. But to be honest, I actually never used Google
2651Analytics to the fullest extent and was usually interested in seeing page hits
2652and which pages were visited most often.&lt;/p&gt;
2653&lt;p&gt;I recently moved my blog from Firebase to a VPS and also decided to remove
2654Google Analytics tracking code from the site since its quite malicious and
2655tracks users across other pages also and is creating a profile of a user, and
2656I&#39;ve had it. But I also need some insight of what is happening on a server and
2657which content is being read the most etc.&lt;/p&gt;
2658&lt;p&gt;I have looked at many existing solutions like:&lt;/p&gt;
2659&lt;ul&gt;
2660&lt;li&gt;&lt;a href=&#34;https://umami.is/&#34;&gt;Umami&lt;/a&gt;&lt;/li&gt;
2661&lt;li&gt;&lt;a href=&#34;https://github.com/sheshbabu/freshlytics&#34;&gt;Freshlytics&lt;/a&gt;&lt;/li&gt;
2662&lt;li&gt;&lt;a href=&#34;https://matomo.org/&#34;&gt;Matomo&lt;/a&gt;&lt;/li&gt;
2663&lt;/ul&gt;
2664&lt;p&gt;But the more I looked at them the more I noticed that I am replacing one evil
2665with another one. Don&#39;t get me wrong. Some of these solutions are absolutely
2666fantastic but would require installation of databases and something like PHP or
2667Node. And I was not ready to put those things on my fresh server. Also having
2668Docker installed is out of the question.&lt;/p&gt;
2669&lt;h2 id=&#34;opting-for-log-parsing&#34;&gt;Opting for log parsing&lt;/h2&gt;
2670&lt;p&gt;So, I defaulted to parsing already existing logs and generating HTML reports
2671from this data.&lt;/p&gt;
2672&lt;p&gt;I found this amazing software &lt;a href=&#34;https://goaccess.io/&#34;&gt;GoAccess&lt;/a&gt; which provides
2673all the functionalities I need, and it&#39;s a single binary. Written in Go.&lt;/p&gt;
2674&lt;p&gt;GoAccess can be used in two different modes.&lt;/p&gt;
2675&lt;p&gt;&lt;img src=&#34;/assets/goaccess/goaccess-dash-term.png&#34; alt=&#34;GoAccess Terminal&#34; /&gt;&lt;/p&gt;
2676&lt;center&gt;&lt;i&gt;Running in a terminal&lt;/i&gt;&lt;/center&gt;
2677&lt;p&gt;&lt;img src=&#34;/assets/goaccess/goaccess-dash-html.png&#34; alt=&#34;GoAccess HTML&#34; /&gt;&lt;/p&gt;
2678&lt;center&gt;&lt;i&gt;Running in a browser&lt;/i&gt;&lt;/center&gt;
2679&lt;p&gt;I, however, need this to run in a browser. So, the second option is the way to
2680go. The Idea is to periodically run cronjob and export this report into a folder
2681that gets then server by Nginx behind a Basic authentication.&lt;/p&gt;
2682&lt;h2 id=&#34;getting-nginx-ready&#34;&gt;Getting Nginx ready&lt;/h2&gt;
2683&lt;p&gt;I choose Ubuntu on &lt;a href=&#34;https://www.digitalocean.com/&#34;&gt;DigitalOcean&lt;/a&gt;. First I
2684installed &lt;a href=&#34;https://nginx.org/en/&#34;&gt;Nginx&lt;/a&gt;, and
2685&lt;a href=&#34;https://letsencrypt.org/getting-started/&#34;&gt;Letsencrypt&lt;/a&gt; certbot and all the
2686necessary dependencies.&lt;/p&gt;
2687&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# log in as root user&lt;/span&gt;
2688&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo su -
2689&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2690&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# first let&amp;#39;s update the system&lt;/span&gt;
2691&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;apt update &amp;amp;&amp;amp; apt upgrade -y
2692&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2693&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# let&amp;#39;s install&lt;/span&gt;
2694&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;apt install nginx certbot python3-certbot-nginx apache2-utils
2695&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After all this is installed we can create a new configuration for a statistics.
2696Stats will be available at &lt;code&gt;stats.domain.com&lt;/code&gt;.&lt;/p&gt;
2697&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# creates directory where html will be hosted&lt;/span&gt;
2698&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;mkdir -p /var/www/html/stats.domain.com
2699&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2700&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cp /etc/nginx/sites-available/default /etc/nginx/sites-available/stats.domain.com
2701&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;nano /etc/nginx/sites-available/stats.domain.com
2702&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;server&lt;/span&gt; {
2703&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;root&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;/var/www/html/stats.domain.com&lt;/span&gt;;
2704&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;server_name&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;stats.domain.com&lt;/span&gt;;
2705&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2706&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;index&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;index.html&lt;/span&gt;;
2707&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;location&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;/&lt;/span&gt; {
2708&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;try_files&lt;/span&gt; $uri $uri/ =404;
2709&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
2710&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
2711&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now we check if the configuration is ok. We can do this with &lt;code&gt;nginx -t&lt;/code&gt;. If all
2712is ok, we can restart Nginx with &lt;code&gt;service nginx restart&lt;/code&gt;.&lt;/p&gt;
2713&lt;p&gt;After all that you should add A record for this domain that points to IP of a
2714droplet.&lt;/p&gt;
2715&lt;p&gt;Before enabling SSL you should test if DNS records have propagated with &lt;code&gt;curl stats.domain.com&lt;/code&gt;.&lt;/p&gt;
2716&lt;p&gt;Now, it&#39;s time to provision TLS certificate. To achieve this, you execute
2717command &lt;code&gt;certbot --nginx&lt;/code&gt;. Follow the wizard and when you are asked about
2718redirection always choose 2 (always redirect to HTTPS).&lt;/p&gt;
2719&lt;p&gt;When this is done you can visit &lt;a href=&#34;https://stats.domain.com&#34;&gt;https://stats.domain.com&lt;/a&gt; and you should get 404
2720not found error which is correct.&lt;/p&gt;
2721&lt;h2 id=&#34;getting-goaccess-ready&#34;&gt;Getting GoAccess ready&lt;/h2&gt;
2722&lt;p&gt;If you are using Debian like system GoAccess should be available in repository.
2723Otherwise refer to the official website.&lt;/p&gt;
2724&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;apt install goaccess
2725&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;To enable Geo location we also need one additiona thing.&lt;/p&gt;
2726&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cd /var/www/html/stats.stats.com
2727&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;wget https://github.com/P3TERX/GeoLite.mmdb/raw/download/GeoLite2-City.mmdb
2728&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now we create a shell script that will be executed every 10 minutes.&lt;/p&gt;
2729&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;nano /var/www/html/stats.domain.com/generate-stats.sh
2730&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Contents of this file should look like this.&lt;/p&gt;
2731&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#!/bin/sh
2732&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&lt;/span&gt;
2733&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;zcat -f /var/log/nginx/access.log* &amp;gt; /var/log/nginx/access-all.log
2734&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2735&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;goaccess &lt;span style=&#34;color:#a31515&#34;&gt;\
2736&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --log-file=/var/log/nginx/access-all.log &lt;span style=&#34;color:#a31515&#34;&gt;\
2737&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --log-format=COMBINED &lt;span style=&#34;color:#a31515&#34;&gt;\
2738&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --exclude-ip=0.0.0.0 &lt;span style=&#34;color:#a31515&#34;&gt;\
2739&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --geoip-database=/var/www/html/stats.domain.com/GeoLite2-City.mmdb &lt;span style=&#34;color:#a31515&#34;&gt;\
2740&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --ignore-crawlers &lt;span style=&#34;color:#a31515&#34;&gt;\
2741&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --real-os &lt;span style=&#34;color:#a31515&#34;&gt;\
2742&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; --output=/var/www/html/stats.domain.com/index.html
2743&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2744&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;rm /var/log/nginx/access-all.log
2745&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Because after a while nginx creates multiple files with access logs we use
2746&lt;a href=&#34;https://linux.die.net/man/1/zcat&#34;&gt;&lt;code&gt;zcat&lt;/code&gt;&lt;/a&gt; to extract Gziped contents and create
2747a file that has all the access logs. After this file is used we delete it.&lt;/p&gt;
2748&lt;p&gt;If you want to exclude your home IP&#39;s result look at the &lt;code&gt;--exclude-ip&lt;/code&gt; option
2749in script and instead of &lt;code&gt;0.0.0.0&lt;/code&gt; add your own home IP address. You can find
2750your home IP by executing &lt;code&gt;curl ifconfig.me&lt;/code&gt; from your local machine and NOT
2751from the droplet.&lt;/p&gt;
2752&lt;p&gt;Test the script by executing &lt;code&gt;sh /var/www/html/stats.domain.com/generate-stats.sh&lt;/code&gt; and then checking
2753&lt;code&gt;https://stats.domain.com&lt;/code&gt;. If you can see stats instead of 404 than you are
2754set.&lt;/p&gt;
2755&lt;p&gt;It&#39;s time to add this script to cron with &lt;code&gt;cron -e&lt;/code&gt;.&lt;/p&gt;
2756&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;*/10 * * * * sh /&lt;span style=&#34;color:#00f&#34;&gt;var&lt;/span&gt;/www/html/stats.domain.com/generate-stats.sh
2757&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;securing-with-basic-authentication&#34;&gt;Securing with Basic authentication&lt;/h2&gt;
2758&lt;p&gt;You probably don&#39;t want stats to be publicly available, so we should create a
2759user and a password for Basic authentication.&lt;/p&gt;
2760&lt;p&gt;First we create a password for a user &lt;code&gt;stats&lt;/code&gt; with &lt;code&gt;htpasswd -c /etc/nginx/.htpasswd stats&lt;/code&gt;.&lt;/p&gt;
2761&lt;p&gt;Now we update config file with &lt;code&gt;nano /etc/nginx/sites-available/stats.domain.com&lt;/code&gt;. You probably noticed that the
2762file looks a bit different from before. This is because &lt;code&gt;certbot&lt;/code&gt; added
2763additional rules for SSL.&lt;/p&gt;
2764&lt;p&gt;Your location portion the config file should now look like. You should add
2765&lt;code&gt;auth_basic&lt;/code&gt; and &lt;code&gt;auth_basic_user_file&lt;/code&gt; lines to the file.&lt;/p&gt;
2766&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;location&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;/&lt;/span&gt; {
2767&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;try_files&lt;/span&gt; $uri $uri/ =404;
2768&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;auth_basic&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Private&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;Property&amp;#34;&lt;/span&gt;;
2769&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;auth_basic_user_file&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;/etc/nginx/.htpasswd&lt;/span&gt;;
2770&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
2771&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Test if config is still ok with &lt;code&gt;nginx -t&lt;/code&gt; and if it is you can restart Nginx
2772with &lt;code&gt;service nginx restart&lt;/code&gt;.&lt;/p&gt;
2773&lt;p&gt;If you now visit &lt;code&gt;https://stats.domain.com&lt;/code&gt; you should be prompted for username
2774and password. If not, try reopening your browser.&lt;/p&gt;
2775&lt;p&gt;That is all. You now have analytics for your server that gets refreshed every 10
2776minutes.&lt;/p&gt;
2777</content:encoded>
2778 </item>
2779
2780
2781
2782 <item>
2783 <title>Replacing Dropbox in favor of DigitalOcean spaces</title>
2784 <link>https://mitjafelicijan.com/replacing-dropbox-in-favor-of-digitalocean-spaces.html</link>
2785 <pubDate>Sun, 24 Jan 2021 12:00:00 &#43;0200</pubDate>
2786 <guid>https://mitjafelicijan.com/replacing-dropbox-in-favor-of-digitalocean-spaces.html</guid>
2787 <description>A few months ago I experimented with DigitalOcean spaces as my backup solutionthat could replace Dropboxeventually.</description>
2788 <content:encoded>&lt;p&gt;A few months ago I experimented with DigitalOcean spaces as my backup solution
2789that could &lt;a href=&#34;/digitalocean-spaces-to-sync-between-computers.html&#34;&gt;replace Dropbox
2790eventually&lt;/a&gt;. That solution
2791worked quite nicely, and I was amazed how smashing together a couple of existing
2792solutions would work this fine.&lt;/p&gt;
2793&lt;p&gt;I have been running that solution in the background for a couple of months now
2794and kind of forgot about it. But recent developments around deplatforming and
2795having us people hostages of technology and big companies speed up my goals to
2796become less dependent on
2797&lt;a href=&#34;https://edition.cnn.com/2020/12/17/tech/google-antitrust-lawsuit/index.html&#34;&gt;Google&lt;/a&gt;,
2798&lt;a href=&#34;https://www.pcworld.com/article/2048680/dropbox-takes-a-peek-at-files.html&#34;&gt;Dropbox&lt;/a&gt;
2799etc and take back some control.&lt;/p&gt;
2800&lt;p&gt;I am not a conspiracy theory nut, but to be honest, what these companies are
2801doing lately is out of control. It is a matter of principle at this point. I
2802have almost completely degoogled my life all the way from ditching Gmail,
2803YouTube and most of the services surrounding Google. And I must tell you, I feel
2804so good. I haven&#39;t felt this way for a long time.&lt;/p&gt;
2805&lt;p&gt;&lt;strong&gt;Anyways. Let&#39;s get to the meat of things.&lt;/strong&gt;&lt;/p&gt;
2806&lt;p&gt;Before you continue you should read my post about &lt;a href=&#34;/digitalocean-spaces-to-sync-between-computers.html&#34;&gt;syncing to
2807Dropbox&lt;/a&gt;.&lt;/p&gt;
2808&lt;blockquote&gt;
2809&lt;p&gt;Also to note, I am using Linux on my machine with Gnome desktop environment.
2810This should work on MacOS too. To use this on Windows I suggest using
2811&lt;a href=&#34;https://docs.microsoft.com/en-us/windows/wsl/install-win10&#34;&gt;Subsystem for Linux&lt;/a&gt;
2812or &lt;a href=&#34;https://www.cygwin.com/&#34;&gt;Cygwin&lt;/a&gt;.&lt;/p&gt;
2813&lt;/blockquote&gt;
2814&lt;h2 id=&#34;folder-structure&#34;&gt;Folder structure&lt;/h2&gt;
2815&lt;p&gt;I liked structure from Dropbox. One folder where everything is located and
2816synced. So, that&#39;s why adopted this also for my sync setup.&lt;/p&gt;
2817&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;&#34;&gt;~&lt;/span&gt;/Vault
2818&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;&#34;&gt;↳&lt;/span&gt; backup
2819&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;&#34;&gt;↳&lt;/span&gt; bin
2820&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;&#34;&gt;↳&lt;/span&gt; documents
2821&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;&#34;&gt;↳&lt;/span&gt; projects
2822&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;All of my code is located in &lt;code&gt;~/Vault/projects&lt;/code&gt; folder. And most of the projects
2823are Git repositories. I do not use this sync method for backup per see but in
2824case I reinstall my machine I can easily recreate all the important folder
2825structure with one quick command. No external drives needed that can fail etc.&lt;/p&gt;
2826&lt;h2 id=&#34;sync-script&#34;&gt;Sync script&lt;/h2&gt;
2827&lt;p&gt;My sync script is located in &lt;code&gt;~/Vault/bin/vault-backup.sh&lt;/code&gt;&lt;/p&gt;
2828&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#!/bin/bash
2829&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&lt;/span&gt;
2830&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# dconf load /com/gexperts/Tilix/ &amp;lt; tilix.dconf&lt;/span&gt;
2831&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# 0 2 * * * sh ~/Vault/bin/vault-backup.sh&lt;/span&gt;
2832&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2833&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cd ~/Vault/backup/dotfiles
2834&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2835&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;MACHINE=&lt;span style=&#34;color:#00f&#34;&gt;$(&lt;/span&gt;whoami&lt;span style=&#34;color:#00f&#34;&gt;)&lt;/span&gt;@&lt;span style=&#34;color:#00f&#34;&gt;$(&lt;/span&gt;hostname&lt;span style=&#34;color:#00f&#34;&gt;)&lt;/span&gt;
2836&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;mkdir -p $MACHINE
2837&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cd $MACHINE
2838&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2839&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cp ~/.config/VSCodium/User/settings.json settings.json
2840&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cp ~/.s3cfg s3cfg
2841&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cp ~/.bash_extended bash_extended
2842&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cp ~/.ssh ssh -rf
2843&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2844&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;codium --list-extensions &amp;gt; vscode-extension.txt
2845&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dconf dump /com/gexperts/Tilix/ &amp;gt; tilix.dconf
2846&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2847&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cd ~/Vault
2848&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;s3cmd sync --delete-removed --exclude &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;node_modules/*&amp;#39;&lt;/span&gt; --exclude &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;.git/*&amp;#39;&lt;/span&gt; --exclude &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;.venv/*&amp;#39;&lt;/span&gt; ./ s3://bucket-name/backup/
2849&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2850&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;echo &lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt;date &#43;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;%D %T&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt; &amp;gt;&amp;gt; ~/.vault.log
2851&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
2852&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;notify-send &lt;span style=&#34;color:#a31515&#34;&gt;\
2853&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -u normal &lt;span style=&#34;color:#a31515&#34;&gt;\
2854&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; -i /usr/share/icons/Adwaita/96x96/status/security-medium-symbolic.symbolic.png &lt;span style=&#34;color:#a31515&#34;&gt;\
2855&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Vault sync succeded at `date &#43;&amp;#34;&lt;/span&gt;%D %T&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;`&amp;#34;&lt;/span&gt;
2856&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This script also backups some of the dotfiles I use and sends notification to
2857Gnome notification center. It is a straightforward solution. Nothing special
2858going on.&lt;/p&gt;
2859&lt;blockquote&gt;
2860&lt;p&gt;One obvious benefit of this is that I can omit syncing Node&#39;s &lt;code&gt;node_modules&lt;/code&gt;
2861or Python&#39;s &lt;code&gt;.venv&lt;/code&gt; and &lt;code&gt;.git&lt;/code&gt; folders.&lt;/p&gt;
2862&lt;/blockquote&gt;
2863&lt;p&gt;You can use this script in a combination with &lt;a href=&#34;https://en.wikipedia.org/wiki/Cron&#34;&gt;Cron&lt;/a&gt;.&lt;/p&gt;
2864&lt;pre&gt;&lt;code&gt;0 2 * * * sh ~/Vault/bin/vault-backup.sh
2865&lt;/code&gt;&lt;/pre&gt;
2866&lt;p&gt;When you start syncing your local stuff with a remote server you can review your
2867items on DigitalOcean.&lt;/p&gt;
2868&lt;p&gt;&lt;img src=&#34;/assets/dropbox-sync/dropbox-spaces.png&#34; alt=&#34;Dropbox Spaces&#34; /&gt;&lt;/p&gt;
2869&lt;p&gt;I have been using this script now for quite some time, and it&#39;s working
2870flawlessly. I also uninstalled Dropbox and stopped using it completely.&lt;/p&gt;
2871&lt;p&gt;All I need to do is write a Bash script that does the reverse and downloads from
2872remote server to local folder. This could be another post.&lt;/p&gt;
2873</content:encoded>
2874 </item>
2875
2876
2877
2878 <item>
2879 <title>Using Digitalocean Spaces to sync between computers</title>
2880 <link>https://mitjafelicijan.com/digitalocean-spaces-to-sync-between-computers.html</link>
2881 <pubDate>Wed, 09 Sep 2020 12:00:00 &#43;0200</pubDate>
2882 <guid>https://mitjafelicijan.com/digitalocean-spaces-to-sync-between-computers.html</guid>
2883 <description>I&amp;#39;ve been using Dropbox for probably 10&#43; yearsnow and I-ve became so used to it that it runs in the background that I don&amp;#39;teven imagine a world without it.</description>
2884 <content:encoded>&lt;p&gt;I&#39;ve been using &lt;a href=&#34;https://www.dropbox.com/&#34;&gt;Dropbox&lt;/a&gt; for probably &lt;strong&gt;10&#43; years&lt;/strong&gt;
2885now and I-ve became so used to it that it runs in the background that I don&#39;t
2886even imagine a world without it. But it&#39;s not without problems.&lt;/p&gt;
2887&lt;p&gt;At first I had problems with &lt;code&gt;.venv&lt;/code&gt; environments for Python and the only
2888solution for excluding synchronization for this folder was to manually exclude a
2889specific folder which is not really scalable. FYI, my whole project folder is
2890synced on &lt;a href=&#34;https://www.dropbox.com/&#34;&gt;Dropbox&lt;/a&gt;. This of course introduced a lot
2891of syncing of files and folders that are not needed or even break things on
2892other machines. In the case of &lt;strong&gt;Python&lt;/strong&gt;, I couldn&#39;t use that on my second
2893machine. I needed to delete &lt;code&gt;.venv&lt;/code&gt; folder and pip it again which synced files
2894again to the main machine. This was very frustrating. &lt;strong&gt;Nodejs&lt;/strong&gt; handles this
2895much nicer and I can just run the scripts without deleting &lt;code&gt;node_modules&lt;/code&gt; again
2896and reinstalling. However, &lt;code&gt;node_modules&lt;/code&gt; is a beast of its own. It creates so
2897many files that OS has a problem counting them when you check the folder
2898contents for size.&lt;/p&gt;
2899&lt;p&gt;I wanted something similar to Dropbox. I could without the instant syncing but
2900it would need to be fast and had the option for me to exclude folders like
2901&lt;code&gt;node_modules, .venv, .git&lt;/code&gt; and folders like that.&lt;/p&gt;
2902&lt;p&gt;I went on a hunt for an alternative to &lt;a href=&#34;https://www.dropbox.com/&#34;&gt;Dropbox&lt;/a&gt;
2903and found:&lt;/p&gt;
2904&lt;ul&gt;
2905&lt;li&gt;&lt;a href=&#34;https://tresorit.com/&#34;&gt;Tresorit&lt;/a&gt;&lt;/li&gt;
2906&lt;li&gt;&lt;a href=&#34;https://sync.com&#34;&gt;Sync.com&lt;/a&gt;&lt;/li&gt;
2907&lt;li&gt;&lt;a href=&#34;https://www.box.com/&#34;&gt;Box&lt;/a&gt;&lt;/li&gt;
2908&lt;/ul&gt;
2909&lt;p&gt;You know, the usual list of suspects. I didn&#39;t include &lt;a href=&#34;https://drive.google.com&#34;&gt;Google
2910drive&lt;/a&gt; or &lt;a href=&#34;https://onedrive.live.com/&#34;&gt;One drive&lt;/a&gt;
2911since they are even more draconian than Dropbox.&lt;/p&gt;
2912&lt;blockquote&gt;
2913&lt;p&gt;All this does not stem from me being paranoid but recently these companies
2914have became more and more aggressive and they keep violating our privacy when
2915they share our data with 3rd party services. It is getting out of control.&lt;/p&gt;
2916&lt;/blockquote&gt;
2917&lt;p&gt;So, my main problem was still there. No way of excluding a specific folder from
2918syncing. And before we go into &amp;quot;&lt;em&gt;But you have git, isn&#39;t that enough?&lt;/em&gt;&amp;quot;, I must
2919say, that many of the files (PDFs, spreadsheets, etc) I have in a &lt;code&gt;git&lt;/code&gt; repo
2920don&#39;t get pushed upstream to Git and I still want to have them synced across my
2921computers.&lt;/p&gt;
2922&lt;p&gt;I initially wanted to use &lt;a href=&#34;https://linux.die.net/man/1/rsync&#34;&gt;rsync&lt;/a&gt; but I would
2923need to then have a remote VPS or transfer between my computers directly. I
2924wanted a solution where all my files could be accessible to me without my
2925machine.&lt;/p&gt;
2926&lt;blockquote&gt;
2927&lt;p&gt;&lt;strong&gt;WARNING: This solution will cost you money!&lt;/strong&gt; DigitalOcean Spaces are $5 per
2928month and there are some bandwidth limitations and if you go beyond that you get
2929billed additionally.&lt;/p&gt;
2930&lt;/blockquote&gt;
2931&lt;p&gt;Then I remembered that I could use something like
2932&lt;a href=&#34;https://en.wikipedia.org/wiki/Amazon_S3&#34;&gt;S3&lt;/a&gt; since it has versioning and is
2933fully managed. I didn&#39;t want to go down the AWS rabbit hole with this so I
2934choose &lt;a href=&#34;https://www.digitalocean.com/products/spaces/&#34;&gt;DigitalOcean Spaces&lt;/a&gt;.&lt;/p&gt;
2935&lt;p&gt;Then I needed a command-line tool to sync between source and target. I found
2936this nice tool &lt;a href=&#34;https://s3tools.org/s3cmd&#34;&gt;s3cmd&lt;/a&gt; and it is in the Ubuntu
2937repositories.&lt;/p&gt;
2938&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo apt install s3cmd
2939&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After installation will I create a new Space bucket on DigitalOcean. Remember
2940the zone you will choose because you will need it when you will configure
2941&lt;code&gt;s3cmd&lt;/code&gt;.&lt;/p&gt;
2942&lt;p&gt;Then I visited &lt;a href=&#34;https://cloud.digitalocean.com/account/api/tokens&#34;&gt;Digitalocean Applications &amp;amp;
2943API&lt;/a&gt; and generated &lt;strong&gt;Spaces
2944access keys&lt;/strong&gt;. Save both key and secret somewhere safe because when you will
2945leave the page secret will not be available anymore to you and you will need to
2946re-generate it.&lt;/p&gt;
2947&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# enter your key and secret and correct endpoint&lt;/span&gt;
2948&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# my endpoint is ams3.digitaloceanspaces.com because&lt;/span&gt;
2949&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# I created my bucket in Amsterdam regiin&lt;/span&gt;
2950&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;s3cmd --configure
2951&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After that I played around with options for &lt;code&gt;s3cmd&lt;/code&gt; and got to the following
2952command.&lt;/p&gt;
2953&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# I executed this command from my projects folder&lt;/span&gt;
2954&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cd projects
2955&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;s3cmd sync --delete-removed --exclude &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;node_modules/*&amp;#39;&lt;/span&gt; --exclude &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;.git/*&amp;#39;&lt;/span&gt; --exclude &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;.venv/*&amp;#39;&lt;/span&gt; ./ s3://my-bucket-name/projects/
2956&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;When syncing int he other direction you will need to change the order of the
2957&lt;code&gt;SOURCE&lt;/code&gt; and &lt;code&gt;TARGET&lt;/code&gt; to &lt;code&gt;s3://my-bucket-name/projects/&lt;/code&gt; and &lt;code&gt;./&lt;/code&gt;.&lt;/p&gt;
2958&lt;blockquote&gt;
2959&lt;p&gt;Be sure that all the paths have trailing slash so that sync knows that this
2960are directories.&lt;/p&gt;
2961&lt;/blockquote&gt;
2962&lt;p&gt;I am planning to implement some sort of a &lt;code&gt;.ignore&lt;/code&gt; file that will enable me to
2963have a project-specific exclude options.&lt;/p&gt;
2964&lt;p&gt;I am currently running this every hour as a cronjob which is perfectly fine for
2965now when I am testing how this whole thing works and how it all will turn out.&lt;/p&gt;
2966&lt;p&gt;I have also created a small Gnome extension which is still very unstable, but
2967when/if this whole experiment pays of I will share on Github.&lt;/p&gt;
2968</content:encoded>
2969 </item>
2970
2971
2972
2973 <item>
2974 <title>Fix bind warning in .profile on login in Ubuntu</title>
2975 <link>https://mitjafelicijan.com/bind-warning-on-login-in-ubuntu.html</link>
2976 <pubDate>Tue, 08 Sep 2020 12:00:00 &#43;0200</pubDate>
2977 <guid>https://mitjafelicijan.com/bind-warning-on-login-in-ubuntu.html</guid>
2978 <description>Recently I moved back to bash as mydefault shell.</description>
2979 <content:encoded>&lt;p&gt;Recently I moved back to &lt;a href=&#34;https://www.gnu.org/software/bash/&#34;&gt;bash&lt;/a&gt; as my
2980default shell. I was previously using &lt;a href=&#34;https://fishshell.com/&#34;&gt;fish&lt;/a&gt; and got
2981used to the cool features it has. But, regardless of that, I wanted to move to a
2982more standard shell because I was hopping back and forth with exporting
2983variables and stuff like that which got pretty annoying.&lt;/p&gt;
2984&lt;p&gt;So I embarked on a mission to make &lt;a href=&#34;https://www.gnu.org/software/bash/&#34;&gt;bash&lt;/a&gt;
2985more like &lt;a href=&#34;https://fishshell.com/&#34;&gt;fish&lt;/a&gt; and in the process found that I really
2986missed autosuggest with TAB on changing directories.&lt;/p&gt;
2987&lt;p&gt;I found a nice alternative that emulates &lt;a href=&#34;http://zsh.sourceforge.net/&#34;&gt;zsh&lt;/a&gt; like
2988autosuggestion and autocomplete so I added the following to my &lt;code&gt;.bashrc&lt;/code&gt; file.&lt;/p&gt;
2989&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;TAB:menu-complete&amp;#34;&lt;/span&gt;
2990&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;set show-all-if-ambiguous on&amp;#34;&lt;/span&gt;
2991&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;set completion-ignore-case on&amp;#34;&lt;/span&gt;
2992&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;set menu-complete-display-prefix on&amp;#34;&lt;/span&gt;
2993&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&amp;#34;\e[Z&amp;#34;:menu-complete-backward&amp;#39;&lt;/span&gt;
2994&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I haven&#39;t noticed anything wrong with this and all was working fine until I
2995restarted my machine and then I got this error.&lt;/p&gt;
2996&lt;p&gt;&lt;img src=&#34;/assets/profile-bind-error/error.jpg&#34; alt=&#34;Profile bind error&#34; /&gt;&lt;/p&gt;
2997&lt;p&gt;When I pressed OK, I got into the &lt;a href=&#34;https://wiki.gnome.org/Projects/GnomeShell&#34;&gt;Gnome
2998shell&lt;/a&gt; and all was working fine, but
2999the error was still bugging me. I started looking for the reason why this is
3000happening and found a solution to this error on &lt;a href=&#34;https://superuser.com/a/892682&#34;&gt;Remote SSH Commands - bash bind
3001warning: line editing not enabled&lt;/a&gt;.&lt;/p&gt;
3002&lt;p&gt;So I added a simple &lt;code&gt;if [ -t 1 ]&lt;/code&gt; around &lt;code&gt;bind&lt;/code&gt; statements to avoid running
3003commands that presume the session is interactive when it isn&#39;t.&lt;/p&gt;
3004&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; [ -t 1 ]; &lt;span style=&#34;color:#00f&#34;&gt;then&lt;/span&gt;
3005&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;TAB:menu-complete&amp;#34;&lt;/span&gt;
3006&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;set show-all-if-ambiguous on&amp;#34;&lt;/span&gt;
3007&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;set completion-ignore-case on&amp;#34;&lt;/span&gt;
3008&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;set menu-complete-display-prefix on&amp;#34;&lt;/span&gt;
3009&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bind &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&amp;#34;\e[Z&amp;#34;:menu-complete-backward&amp;#39;&lt;/span&gt;
3010&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;fi&lt;/span&gt;
3011&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After logging out and back in the problem was gone.&lt;/p&gt;
3012</content:encoded>
3013 </item>
3014
3015
3016
3017 <item>
3018 <title>Getting started with MicroPython and ESP8266</title>
3019 <link>https://mitjafelicijan.com/esp8266-and-micropython-guide.html</link>
3020 <pubDate>Sun, 06 Sep 2020 12:00:00 &#43;0200</pubDate>
3021 <guid>https://mitjafelicijan.com/esp8266-and-micropython-guide.html</guid>
3022 <description>IntroductionA while ago I bought someESP8266 andESP32 dev boards to playaround with and I finally found a project to try it out.</description>
3023 <content:encoded>&lt;h2 id=&#34;introduction&#34;&gt;Introduction&lt;/h2&gt;
3024&lt;p&gt;A while ago I bought some
3025&lt;a href=&#34;https://www.espressif.com/en/products/socs/esp8266&#34;&gt;ESP8266&lt;/a&gt; and
3026&lt;a href=&#34;https://www.espressif.com/en/products/socs/esp32&#34;&gt;ESP32&lt;/a&gt; dev boards to play
3027around with and I finally found a project to try it out.&lt;/p&gt;
3028&lt;p&gt;For my project, I used &lt;a href=&#34;https://www.espressif.com/en/products/socs/esp32&#34;&gt;ESP32&lt;/a&gt;
3029but I could easily choose
3030&lt;a href=&#34;https://www.espressif.com/en/products/socs/esp8266&#34;&gt;ESP8266&lt;/a&gt;. This guide
3031contains which tools I use and how I prepared my workspace to code for
3032&lt;a href=&#34;https://www.espressif.com/en/products/socs/esp8266&#34;&gt;ESP8266&lt;/a&gt;.&lt;/p&gt;
3033&lt;p&gt;&lt;img src=&#34;/assets/esp8366-micropython/boards.jpg&#34; alt=&#34;ESP8266 and ESP32 boards&#34; /&gt;&lt;/p&gt;
3034&lt;p&gt;This guide covers:&lt;/p&gt;
3035&lt;ul&gt;
3036&lt;li&gt;flashing SOC&lt;/li&gt;
3037&lt;li&gt;install proper tooling&lt;/li&gt;
3038&lt;li&gt;deploying a simple script&lt;/li&gt;
3039&lt;/ul&gt;
3040&lt;blockquote&gt;
3041&lt;p&gt;Make sure that you are using &lt;strong&gt;a good USB cable&lt;/strong&gt;. I had some problems with
3042mine and once I replaced it everything started to work.&lt;/p&gt;
3043&lt;/blockquote&gt;
3044&lt;h2 id=&#34;flashing-the-soc&#34;&gt;Flashing the SOC&lt;/h2&gt;
3045&lt;p&gt;Plug your ESP8266 to USB port and check if the device was recognized with
3046executing &lt;code&gt;dmesg | grep ch341-uart&lt;/code&gt;.&lt;/p&gt;
3047&lt;p&gt;Then check if the device is available under &lt;code&gt;/dev/&lt;/code&gt; by running &lt;code&gt;ls /dev/ttyUSB*&lt;/code&gt;.&lt;/p&gt;
3048&lt;blockquote&gt;
3049&lt;p&gt;&lt;strong&gt;Linux users&lt;/strong&gt;: if a device is not available be sure you are in &lt;code&gt;dialout&lt;/code&gt;
3050group. You can check this by executing &lt;code&gt;groups $USER&lt;/code&gt;. You can add a user to
3051&lt;code&gt;dialout&lt;/code&gt; group with &lt;code&gt;sudo adduser $USER dialout&lt;/code&gt;.&lt;/p&gt;
3052&lt;/blockquote&gt;
3053&lt;p&gt;After these conditions are meet go to the navigate to
3054&lt;a href=&#34;https://micropython.org/download/esp8266/&#34;&gt;https://micropython.org/download/esp8266/&lt;/a&gt;
3055and download &lt;code&gt;esp8266-20200902-v1.13.bin&lt;/code&gt;.&lt;/p&gt;
3056&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;mkdir esp8266-test
3057&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cd esp8266-test
3058&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3059&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;wget https://micropython.org/resources/firmware/esp8266-20200902-v1.13.bin
3060&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After obtaining firmware we will need some tooling to flash the firmware to the
3061board.&lt;/p&gt;
3062&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo pip3 install esptool
3063&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;You can read more about &lt;code&gt;esptool&lt;/code&gt; at
3064&lt;a href=&#34;https://github.com/espressif/esptool/&#34;&gt;https://github.com/espressif/esptool/&lt;/a&gt;.&lt;/p&gt;
3065&lt;p&gt;Before flashing the firmware we need to erase the flash on device. Substitute
3066&lt;code&gt;USB0&lt;/code&gt; with the device listed in output of &lt;code&gt;ls /dev/ttyUSB*&lt;/code&gt;.&lt;/p&gt;
3067&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;esptool.py --port /dev/ttyUSB0 erase_flash
3068&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;If flash was successfully erased it is now time to flash the new firmware to it.&lt;/p&gt;
3069&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect 0 esp8266-20200902-v1.13.bin
3070&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;If everything went ok you can try accessing MicroPython REPL with &lt;code&gt; screen /dev/ttyUSB0 115200&lt;/code&gt; or &lt;code&gt;picocom /dev/ttyUSB0 -b115200&lt;/code&gt;.&lt;/p&gt;
3071&lt;blockquote&gt;
3072&lt;p&gt;Sometimes you will need to press &lt;code&gt;ENTER&lt;/code&gt; in &lt;code&gt;screen&lt;/code&gt; or &lt;code&gt;picocom&lt;/code&gt; to access
3073REPL.&lt;/p&gt;
3074&lt;/blockquote&gt;
3075&lt;p&gt;When you are in REPL you can test if all is working properly following steps.&lt;/p&gt;
3076&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;gt; &lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; machine
3077&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;gt; machine.freq()
3078&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This should output a number representing a frequency of the CPU (mine was
3079&lt;code&gt;80000000&lt;/code&gt;).&lt;/p&gt;
3080&lt;p&gt;When you are in &lt;code&gt;screen&lt;/code&gt; or &lt;code&gt;picocom&lt;/code&gt; these can help you a bit.&lt;/p&gt;
3081&lt;table&gt;
3082&lt;thead&gt;
3083&lt;tr&gt;
3084&lt;th&gt;Key&lt;/th&gt;
3085&lt;th&gt;Command&lt;/th&gt;
3086&lt;/tr&gt;
3087&lt;/thead&gt;
3088&lt;tbody&gt;
3089&lt;tr&gt;
3090&lt;td&gt;CTRL&#43;d&lt;/td&gt;
3091&lt;td&gt;preforms soft reboot&lt;/td&gt;
3092&lt;/tr&gt;
3093&lt;tr&gt;
3094&lt;td&gt;CTRL&#43;a x&lt;/td&gt;
3095&lt;td&gt;exits picocom&lt;/td&gt;
3096&lt;/tr&gt;
3097&lt;tr&gt;
3098&lt;td&gt;CTRL&#43;a \&lt;/td&gt;
3099&lt;td&gt;exits screen&lt;/td&gt;
3100&lt;/tr&gt;
3101&lt;/tbody&gt;
3102&lt;/table&gt;
3103&lt;h2 id=&#34;install-better-tooling&#34;&gt;Install better tooling&lt;/h2&gt;
3104&lt;p&gt;Now, to make our lives a little bit easier there are couple of additional tools
3105that will make this whole experience a little more bearable.&lt;/p&gt;
3106&lt;p&gt;There are twq cool ways of uploading local files to SOC flash.&lt;/p&gt;
3107&lt;ul&gt;
3108&lt;li&gt;ampy → &lt;a href=&#34;https://github.com/scientifichackers/ampy&#34;&gt;https://github.com/scientifichackers/ampy&lt;/a&gt;&lt;/li&gt;
3109&lt;li&gt;rshell → &lt;a href=&#34;https://github.com/dhylands/rshell&#34;&gt;https://github.com/dhylands/rshell&lt;/a&gt;&lt;/li&gt;
3110&lt;/ul&gt;
3111&lt;h3 id=&#34;ampy&#34;&gt;ampy&lt;/h3&gt;
3112&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# installing ampy&lt;/span&gt;
3113&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo pip3 install adafruit-ampy
3114&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Listed below are some common commands I used.&lt;/p&gt;
3115&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3116&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# uploads file to flash&lt;/span&gt;
3117&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ampy --delay 2 --port /dev/ttyUSB0 put boot.py
3118&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3119&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# lists file on flash&lt;/span&gt;
3120&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ampy --delay 2 --port /dev/ttyUSB0 ls
3121&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3122&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# outputs contents of file on flash&lt;/span&gt;
3123&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ampy --delay 2 --port /dev/ttyUSB0 cat boot.py
3124&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;blockquote&gt;
3125&lt;p&gt;I added &lt;code&gt;delay&lt;/code&gt; of 2 seconds because I had problems with executing commands.&lt;/p&gt;
3126&lt;/blockquote&gt;
3127&lt;h3 id=&#34;rshell&#34;&gt;rshell&lt;/h3&gt;
3128&lt;p&gt;Even though &lt;code&gt;ampy&lt;/code&gt; is a cool tool I opted with &lt;code&gt;rshell&lt;/code&gt; in the end since it&#39;s
3129much more polished and feature rich.&lt;/p&gt;
3130&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# installing ampy&lt;/span&gt;
3131&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo pip3 install rshell
3132&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now that &lt;code&gt;rshell&lt;/code&gt; is installed we can connect to the board.&lt;/p&gt;
3133&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;rshell --buffer-size=30 -p /dev/ttyUSB0 -a
3134&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This will open a shell inside bash and from here you can execute multiple
3135commands. You can check what is supported with &lt;code&gt;help&lt;/code&gt; once you are inside of a
3136shell.&lt;/p&gt;
3137&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;m@turing ~/Junk/esp8266-test
3138&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ rshell --buffer-size=30 -p /dev/ttyUSB0 -a
3139&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3140&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Using buffer-size of 30
3141&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Connecting to /dev/ttyUSB0 (buffer-size 30)...
3142&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Trying to connect to REPL connected
3143&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Testing &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; ubinascii.unhexlify exists ... Y
3144&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Retrieving root directories ... /boot.py/
3145&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Setting time ... Sep 06, 2020 23:54:28
3146&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Evaluating board_name ... pyboard
3147&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Retrieving time epoch ... Jan 01, 2000
3148&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Welcome to rshell. Use Control-D (or the exit command) to exit rshell.
3149&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;/home/m/Junk/esp8266-test&amp;gt; help
3150&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3151&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Documented commands (type help &amp;lt;topic&amp;gt;):
3152&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;========================================
3153&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;args cat connect date edit filesize help mkdir rm shell
3154&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;boards cd cp echo exit filetype ls repl rsync
3155&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3156&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Use Control-D (or the exit command) to exit rshell.
3157&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;blockquote&gt;
3158&lt;p&gt;Inside a shell &lt;code&gt;ls&lt;/code&gt; will display list of files on your machine. To get list
3159of files on flash folder &lt;code&gt;/pyboard&lt;/code&gt; is remapped inside the shell. To list files
3160on flash you must perform &lt;code&gt;ls /pyboard&lt;/code&gt;.&lt;/p&gt;
3161&lt;/blockquote&gt;
3162&lt;h4 id=&#34;moving-files-to-flash&#34;&gt;Moving files to flash&lt;/h4&gt;
3163&lt;p&gt;To avoid copying files all the time I used &lt;code&gt;rsync&lt;/code&gt; function from the inside of
3164&lt;code&gt;rshell&lt;/code&gt;.&lt;/p&gt;
3165&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;rsync . /pyboard
3166&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h4 id=&#34;executing-scripts&#34;&gt;Executing scripts&lt;/h4&gt;
3167&lt;p&gt;It is a pain to continuously reboot the device to trigger &lt;code&gt;/pyboard/boot.py&lt;/code&gt; and
3168there is a better way of testing local scripts on remote device.&lt;/p&gt;
3169&lt;p&gt;Lets assume we have &lt;code&gt;src/freq.py&lt;/code&gt; file that displays CPU frequency of a remote
3170device.&lt;/p&gt;
3171&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# src/freq.py&lt;/span&gt;
3172&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3173&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; machine
3174&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(machine.freq())
3175&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now lets upload this and execute it.&lt;/p&gt;
3176&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# syncs files to remove device&lt;/span&gt;
3177&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;rsync ./src /pyboard
3178&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3179&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# goes into REPL&lt;/span&gt;
3180&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;repl
3181&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3182&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# we import file by importing it without .py extension and this will run the script&lt;/span&gt;
3183&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;gt; import freq
3184&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3185&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# CTRL&#43;x will exit REPL&lt;/span&gt;
3186&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;additional-resources&#34;&gt;Additional resources&lt;/h2&gt;
3187&lt;ul&gt;
3188&lt;li&gt;&lt;a href=&#34;https://randomnerdtutorials.com/getting-started-micropython-esp32-esp8266/&#34;&gt;https://randomnerdtutorials.com/getting-started-micropython-esp32-esp8266/&lt;/a&gt;&lt;/li&gt;
3189&lt;li&gt;&lt;a href=&#34;http://docs.micropython.org/en/latest/esp8266/quickref.html&#34;&gt;http://docs.micropython.org/en/latest/esp8266/quickref.html&lt;/a&gt;&lt;/li&gt;
3190&lt;/ul&gt;
3191</content:encoded>
3192 </item>
3193
3194
3195
3196 <item>
3197 <title>Disable mouse wake from suspend with systemd service</title>
3198 <link>https://mitjafelicijan.com/disable-mouse-wake-from-suspend-with-systemd-service.html</link>
3199 <pubDate>Sat, 15 Aug 2020 12:00:00 &#43;0200</pubDate>
3200 <guid>https://mitjafelicijan.com/disable-mouse-wake-from-suspend-with-systemd-service.html</guid>
3201 <description>I recently bought ThinkPadX220 just as ajoke on eBay to test Linux distributions and play around with things and notdestroy my main machine.</description>
3202 <content:encoded>&lt;p&gt;I recently bought &lt;a href=&#34;https://www.laptopmag.com/reviews/laptops/lenovo-thinkpad-x220&#34;&gt;ThinkPad
3203X220&lt;/a&gt; just as a
3204joke on eBay to test Linux distributions and play around with things and not
3205destroy my main machine. Little to my knowledge I felt in love with it. Man,
3206they really made awesome machines back then.&lt;/p&gt;
3207&lt;p&gt;After changing disk that came with it to SSD and installing Ubuntu to test if 
3208everything works I noticed that even after a single touch of my external mouse
3209the system would wake up from sleep even though the lid was shut down.&lt;/p&gt;
3210&lt;p&gt;I wouldn&#39;t even noticed it if laptop didn&#39;t have &lt;a href=&#34;https://support.lenovo.com/lk/en/solutions/~/media/Images/ContentImages/p/pd025386_x1_status_03.ashx?w=426&amp;amp;h=262&#34;&gt;LED
3211sleep indicator&lt;/a&gt;.
3212I already had a bad experience with Linux and it&#39;s power management. I had a
3213&lt;a href=&#34;https://www.pcmag.com/reviews/dell-inspiron-15-7537&#34;&gt;Dell Inspiron 7537&lt;/a&gt; laptop
3214with a touchscreen and while traveling it decided to wake up and started cooking
3215in my backpack to the point that the digitizer responsible for touch actually
3216glue off and the whole screen got wrecked. So, I am a bit touchy about this.&lt;/p&gt;
3217&lt;p&gt;I went on solution hunting and to my surprise there is no easy way to disable
3218specific devices to perform wake up. Why is this not under the power management 
3219tab in setting is really strange.&lt;/p&gt;
3220&lt;p&gt;After googling for a solution I found &lt;a href=&#34;https://codetrips.com/2020/03/18/ubuntu-disable-mouse-wake-from-suspend/&#34;&gt;this nice article describing the
3221solution&lt;/a&gt;
3222that worked for me. The only problem with this solution was that he added his
3223solution to &lt;code&gt;.bashrc&lt;/code&gt; and this triggers &lt;code&gt;sudo&lt;/code&gt; that asks for a password each
3224time new terminal is opened, which get annoying quickly since I open a lot of
3225terminals all the time.&lt;/p&gt;
3226&lt;p&gt;I followed his instructions and got to solution &lt;code&gt;sudo sh -c &amp;quot;echo &#39;disabled&#39; &amp;gt; /sys/bus/usb/devices/2-1.1/power/wakeup&amp;quot;&lt;/code&gt;.&lt;/p&gt;
3227&lt;p&gt;I created a system service file &lt;code&gt;sudo nano /etc/systemd/system/disable-mouse-wakeup.service&lt;/code&gt; and removed &lt;code&gt;sudo&lt;/code&gt; and
3228replaced &lt;code&gt;sh&lt;/code&gt; with &lt;code&gt;/usr/bin/sh&lt;/code&gt; and pasted all that in &lt;code&gt;ExecStart&lt;/code&gt;.&lt;/p&gt;
3229&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;[Unit]&lt;/span&gt;
3230&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Description=&lt;span style=&#34;color:#a31515&#34;&gt;Disables wakeup on mouse event&lt;/span&gt;
3231&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;After=&lt;span style=&#34;color:#a31515&#34;&gt;network.target&lt;/span&gt;
3232&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;StartLimitIntervalSec=&lt;span style=&#34;color:#a31515&#34;&gt;0&lt;/span&gt;
3233&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3234&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;[Service]&lt;/span&gt;
3235&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Type=&lt;span style=&#34;color:#a31515&#34;&gt;simple&lt;/span&gt;
3236&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Restart=&lt;span style=&#34;color:#a31515&#34;&gt;always&lt;/span&gt;
3237&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;RestartSec=&lt;span style=&#34;color:#a31515&#34;&gt;1&lt;/span&gt;
3238&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;User=&lt;span style=&#34;color:#a31515&#34;&gt;root&lt;/span&gt;
3239&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ExecStart=&lt;span style=&#34;color:#a31515&#34;&gt;/usr/bin/sh -c &amp;#34;echo &amp;#39;disabled&amp;#39; &amp;gt; /sys/bus/usb/devices/2-1.1/power/wakeup&amp;#34;&lt;/span&gt;
3240&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3241&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;[Install]&lt;/span&gt;
3242&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;WantedBy=&lt;span style=&#34;color:#a31515&#34;&gt;multi-user.target&lt;/span&gt;
3243&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After that I enabled, started and checked status of service.&lt;/p&gt;
3244&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo systemctl enable disable-mouse-wakeup.service
3245&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo systemctl start disable-mouse-wakeup.service
3246&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sudo systemctl status disable-mouse-wakeup.service
3247&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This will permanently disable that device from wakeing up you computer on boot.
3248If you have many devices you would like to surpress from waking up your machine
3249I would create a shell script and call that instead of direclty doing it in
3250service file.&lt;/p&gt;
3251</content:encoded>
3252 </item>
3253
3254
3255
3256 <item>
3257 <title>Remote work and how it affects the daily lives of people</title>
3258 <link>https://mitjafelicijan.com/remote-work.html</link>
3259 <pubDate>Tue, 05 May 2020 12:00:00 &#43;0200</pubDate>
3260 <guid>https://mitjafelicijan.com/remote-work.html</guid>
3261 <description>I have been working remotely for the past 5 years.</description>
3262 <content:encoded>&lt;p&gt;I have been working remotely for the past 5 years. I love it. Love the freedom
3263and make your schedule thingy.&lt;/p&gt;
3264&lt;h2 id=&#34;you-work-more-not-less&#34;&gt;You work more not less&lt;/h2&gt;
3265&lt;p&gt;I&#39;ve heard from people things like: &amp;quot;Oh, you are so lucky, working from home,
3266having all the free time you want&amp;quot;. It was obvious they had no clue what means
3267working remotely. They had this romantic idea of remote work. You can watch TV
3268whenever you like, you can go outside for a picnic if you want and stuff like
3269that.&lt;/p&gt;
3270&lt;p&gt;This may be true if you work a day or two in a week from home. But if you go
3271completely remote all these changes completely. I take some time to acclimate
3272but then you start feeling the consequences of going fully remote. And it&#39;s not
3273all rainbows and unicorns. Rather the opposite.&lt;/p&gt;
3274&lt;h2 id=&#34;feeling-lost&#34;&gt;Feeling lost&lt;/h2&gt;
3275&lt;p&gt;At first, I remembered I felt lost. I was not used to this kind of environment.
3276It felt disoriented and a part of you that is used to procrastinate turns on.
3277You start thinking of a workday as a whole day. And soon this idea of &amp;quot;I can do
3278this later&amp;quot; starts creeping in. Well, I have the whole day ahead of me. I can do
3279this a bit later.&lt;/p&gt;
3280&lt;h2 id=&#34;hyper-performance&#34;&gt;Hyper-performance&lt;/h2&gt;
3281&lt;p&gt;As a direct result, you become more focused on your work since you don&#39;t have
3282all the interruptions common in the workplace. And you can quickly get used to
3283this hyper-performance. But this mode requires also a lot of peace and quiet.&lt;/p&gt;
3284&lt;p&gt;And here we come to the ugly parts of all this. &lt;strong&gt;People rarely have the
3285self-control&lt;/strong&gt; to not waste other people&#39;s time. It is paralyzing when people
3286start calling you, sending you chat messages, etc. The thing is, that when I
3287achieve this hyper-performance mode I am completely embroiled in the problem I
3288am solving and this kind of interruptions mess with your head. I need an hour at
3289least to get back in the zone. Sometimes not achieving the same focus the whole
3290day.&lt;/p&gt;
3291&lt;p&gt;I know that life is not how you want it to be and takes its route but from what
3292I&#39;ve learned this kind of interruptions can be avoided in 90% of the case easily
3293just by closing any chat programs and putting your phone in a drawer.&lt;/p&gt;
3294&lt;h2 id=&#34;suggestion-to-all-the-new-remote-workers&#34;&gt;Suggestion to all the new remote workers&lt;/h2&gt;
3295&lt;ul&gt;
3296&lt;li&gt;Stop wasting other people&#39;s time. You don&#39;t bother people at their desks in
3297the office either.&lt;/li&gt;
3298&lt;li&gt;Do not replace daily chats in the hallways with instant messaging software.
3299It will only interrupt people. Nothing good will come of it.&lt;/li&gt;
3300&lt;li&gt;Set your working hours and try to not allow it to bleed outside these
3301boundaries and maintain your routine.&lt;/li&gt;
3302&lt;li&gt;Be prepared that hours will be longer regardless of your good intentions and
3303your well thought of routine.&lt;/li&gt;
3304&lt;li&gt;Try to be hyper-focused and do only one thing at the time. Multitasking is the
3305enemy of progress.&lt;/li&gt;
3306&lt;li&gt;Avoid long meetings and if possible eliminate them. Rather take time to write
3307them out and allow others to respond in their own time. Meetings are usually a
3308large waste of time and most of the people attending them are there just
3309because the manager said so.&lt;/li&gt;
3310&lt;li&gt;The software will not solve your problems. And throwing money at problems
3311neither.&lt;/li&gt;
3312&lt;li&gt;If you are in a managerial position don&#39;t supervise any single minute of
3313workers. They are probably giving you more hours anyways. Track progress
3314weekly not daily. You hired them and give them the benefit of the doubt that
3315they will deliver what you agreed upon.&lt;/li&gt;
3316&lt;/ul&gt;
3317</content:encoded>
3318 </item>
3319
3320
3321
3322 <item>
3323 <title>My love and hate relationship with Node.js</title>
3324 <link>https://mitjafelicijan.com/my-love-and-hate-relationship-with-nodejs.html</link>
3325 <pubDate>Mon, 30 Mar 2020 12:00:00 &#43;0200</pubDate>
3326 <guid>https://mitjafelicijan.com/my-love-and-hate-relationship-with-nodejs.html</guid>
3327 <description>Previous project I was working on was being coded inGolang.</description>
3328 <content:encoded>&lt;p&gt;Previous project I was working on was being coded in
3329&lt;a href=&#34;https://golang.org/&#34;&gt;Golang&lt;/a&gt;. Also was my first project using it. And damn,
3330that was an awesome experience. The whole thing is just superb. From how errors
3331are handled. The C-like way you handle compiling. The way the language is
3332structured making it incredibly versatile and easy to learn.&lt;/p&gt;
3333&lt;p&gt;It may cause some pain for somebody that is not used of using interfaces to map
3334JSON and doing the recompilation all the time. But we have tools like
3335&lt;a href=&#34;http://eradman.com/entrproject/&#34;&gt;entr&lt;/a&gt; and
3336&lt;a href=&#34;https://www.gnu.org/software/make/&#34;&gt;make&lt;/a&gt; to fix that.&lt;/p&gt;
3337&lt;p&gt;But we are not here to talk about my undying love for &lt;strong&gt;Golang&lt;/strong&gt;. Only in some
3338way we probably should. It is an excellent example of how modern language should
3339be designed. And because I have used it extensively in the last couple of years
3340this probably taints my views of other languages. And is doing me a great
3341disservice. Nevertheless, here we are.&lt;/p&gt;
3342&lt;p&gt;About two years ago I started flirting with &lt;a href=&#34;https://nodejs.org/en/&#34;&gt;Node.js&lt;/a&gt;
3343for a project I started working on. What I wanted was to have things written in
3344a language that is widely used, and we could get additional developers for. As
3345much as &lt;strong&gt;Golang&lt;/strong&gt; is amazing it&#39;s really hard to get developers for it. Even
3346now. And after playing around with it for a week I felt in love with the speed
3347of iteration and massive package ecosystem. Do you want SSO? You got it! Do you
3348want some esoteric library for something? There is a strong chance somebody
3349wrote it. It is so extensive that you find yourself evaluating packages based on
3350&lt;strong&gt;GitHub stars&lt;/strong&gt; and number of contributors. You get swallowed by the vanity
3351metrics and that potentially will become the downfall of Node.js.&lt;/p&gt;
3352&lt;p&gt;Because of the sheer amount of choice I often got anxiety when choosing
3353libraries. Will I choose the correct one? Is this library something that will be
3354supported for a foreseeable future or not? I am used of using libraries that are
3355being in development for 10 years plus (Python, C) and that gave me some sort of
3356comfort. And it is probably unfair to Node.js and community to expect same
3357dedication.&lt;/p&gt;
3358&lt;p&gt;Moving forward ... Work started and things were great. &lt;strong&gt;Speed of iteration was
3359insane&lt;/strong&gt;. For some feature that I would need a day in Golang only took me hour
3360or two. I became lazy! Using packages all over the place. Falling into the same
3361trap as others. Packages on top of packages. And &lt;a href=&#34;https://www.npmjs.com/&#34;&gt;npm&lt;/a&gt;
3362didn&#39;t help at all. The way that the package manager works is just
3363horrendous. And not allowing to have node_modules outside the project is also
3364the stupidest idea ever.&lt;/p&gt;
3365&lt;p&gt;So at that point I started feeling the technical debt that comes with Node.js
3366and the whole ecosystem. What nobody tells you is that &lt;strong&gt;structuring large
3367Node.js apps&lt;/strong&gt; is more problematic than one would think. And going microservice
3368for every single thing is also a bad idea. The amount of networking you
3369introduce with that approach always ends up being a pain in the ass. And I don&#39;t
3370even want to go into system administration here. The overhead is
3371insane. Package-lock.json made many days feel like living hell for me. And I
3372would eat the cost of all this if it meant for better development
3373experience. Well, it didn&#39;t.&lt;/p&gt;
3374&lt;p&gt;The &lt;strong&gt;lack of Typescript&lt;/strong&gt; support in the interpreter is still mind boggling to
3375me. Why haven&#39;t they added native support yet for this is beyond me?! That would
3376have solved so many problems. Lack of type safety became a problem somewhere in
3377the middle of the project where the codebase was sufficiently large enough to
3378present problems. We started adding arguments to functions and there was &lt;strong&gt;no
3379way to implicitly define argument types&lt;/strong&gt;. And because at that point there were
3380a lot of functions, it became impossible to know what each one accepts,
3381development became more and more trial and error based.&lt;/p&gt;
3382&lt;p&gt;I tried &lt;strong&gt;implementing Typescript&lt;/strong&gt;, but that would present a large refactor
3383that we were not willing to do at that point. The benefits were not enough. I
3384also tried &lt;a href=&#34;https://flow.org/&#34;&gt;Flow - static type checker&lt;/a&gt; but implementation
3385was also horrible. What Typescript and Flow forces you is to have src folder and
3386then &lt;strong&gt;transpile&lt;/strong&gt; your code into dist folder and run it with node. WTH is that
3387all about. Why can&#39;t this be done in memory or some virtual file system? Why? I
3388see no reason why this couldn&#39;t be done like this. But it is what it is. I
3389abandoned all hope for static type checking.&lt;/p&gt;
3390&lt;p&gt;One of the problems that resulted from not having interfaces or types was
3391inability to model out our data from &lt;strong&gt;Elasticsearch&lt;/strong&gt;. I could have done a
3392&lt;strong&gt;pedestrian implementation&lt;/strong&gt; of it, but there must be a better way of doing
3393this without resorting to some hack basically. Or maybe I haven&#39;t found a
3394solution, which is also a possibility. I have looked, though. No juice!&lt;/p&gt;
3395&lt;p&gt;&lt;strong&gt;Error handling?&lt;/strong&gt; Is that a joke?&lt;/p&gt;
3396&lt;p&gt;Thank god for &lt;strong&gt;await/async&lt;/strong&gt;. Without it, I would have probably just abandoned
3397the whole thing and went with something else like Python. That&#39;s all I am going
3398to say about this :)&lt;/p&gt;
3399&lt;p&gt;I started asking myself a question if Node.js is actually ready to be used in a
3400&lt;strong&gt;large scale applications&lt;/strong&gt;? And this was a totally wrong question. What I
3401should have been asking myself was, how to use Node.js in large scale
3402application. And you don&#39;t get this in &lt;strong&gt;marketing material&lt;/strong&gt; for Express or Koa
3403etc. They never tell you this. Making Node.js scale on infrastructure or in
3404codebase is really &lt;strong&gt;more of an art than a science&lt;/strong&gt;. And just like with the
3405whole JavaScript ecosystem:&lt;/p&gt;
3406&lt;ul&gt;
3407&lt;li&gt;impossible to master,&lt;/li&gt;
3408&lt;li&gt;half of your time you work on your tooling,&lt;/li&gt;
3409&lt;li&gt;just accept transpilers that convert one code into another (holly smokes),&lt;/li&gt;
3410&lt;li&gt;error handling is a joke,&lt;/li&gt;
3411&lt;li&gt;standards? What standards?&lt;/li&gt;
3412&lt;/ul&gt;
3413&lt;p&gt;But on the other hand. As I did, you will also learn to love it. Learn to use it
3414quickly and do impossible things in crazy limited time.&lt;/p&gt;
3415&lt;p&gt;I hate to admit it. But I love Node.js. Dammit, I love it :)&lt;/p&gt;
3416&lt;p&gt;2023 Update: I hate Node.js!&lt;/p&gt;
3417</content:encoded>
3418 </item>
3419
3420
3421
3422 <item>
3423 <title>The strange case of Elasticsearch allocation failure</title>
3424 <link>https://mitjafelicijan.com/the-strange-case-of-elasticsearch-allocation-failure.html</link>
3425 <pubDate>Sun, 29 Mar 2020 12:00:00 &#43;0200</pubDate>
3426 <guid>https://mitjafelicijan.com/the-strange-case-of-elasticsearch-allocation-failure.html</guid>
3427 <description>I&amp;#39;ve been using Elasticsearch in production for 5 years now and never had asingle problem with it.</description>
3428 <content:encoded>&lt;p&gt;I&#39;ve been using Elasticsearch in production for 5 years now and never had a
3429single problem with it. Hell, never even known there could be a problem. Just
3430worked. All this time. The first node that I deployed is still being used in
3431production, never updated, upgraded, touched in anyway.&lt;/p&gt;
3432&lt;p&gt;All this bliss came to an abrupt end this Friday when I got notification that
3433Elasticsearch cluster went warm. Well, warm is not that bad right? Wrong!
3434Quickly after that I got another email which sent chills down my spine. Cluster
3435is now red. RED! Now, shit really hit the fan!&lt;/p&gt;
3436&lt;p&gt;I tried googling what could be the problem and after executing allocation
3437function noticed that some shards were unassigned and 5 attempts were already
3438made (which is BTW to my luck the maximum) and that meant I am basically fucked.
3439They also applied that one should wait for cluster to re-balance itself. So, I
3440waited. One hour, two hours, several hours. Nothing, still RED.&lt;/p&gt;
3441&lt;p&gt;The strangest thing about it all was, that queries were still being fulfilled.
3442Data was coming out. On the outside it looked like nothing was wrong but
3443everybody that would look at the cluster would know immediately that something
3444was very very wrong and we were living on borrowed time here.&lt;/p&gt;
3445&lt;blockquote&gt;
3446&lt;p&gt;&lt;strong&gt;Please, DO NOT do what I did.&lt;/strong&gt; Seriously! Please ask someone on official
3447forums or if you know an expert please consult him. There could be million of
3448reasons and these solution fit my problem. Maybe in your case it would
3449disastrous. I had all the data backed up and even if I would fail spectacularly
3450I would be able to restore the data. It would be a huge pain and I would loose
3451couple of days but I had a plan B.&lt;/p&gt;
3452&lt;/blockquote&gt;
3453&lt;p&gt;Executing allocation and told me what the problem was but no clear solution yet.&lt;/p&gt;
3454&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;GET /_cat/allocation?format=json
3455&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I got a message that &lt;code&gt;ALLOCATION_FAILED&lt;/code&gt; with additional info &lt;code&gt;failed to create shard, failure ioexception[failed to obtain in-memory shard lock]&lt;/code&gt;. Well
3456splendid! I must also say that our cluster is capable more than enough to handle
3457the traffic. Also JVM memory pressure never was an issue. So what happened
3458really then?&lt;/p&gt;
3459&lt;p&gt;I tried also re-routing failed ones with no success due to AWS restrictions on
3460having managed Elasticsearch cluster (they lock some of the functions).&lt;/p&gt;
3461&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;POST /_cluster/reroute?retry_failed=true
3462&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I got a message that significantly reduced my options.&lt;/p&gt;
3463&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
3464&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;Message&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Your request: &amp;#39;/_cluster/reroute&amp;#39; is not allowed.&amp;#34;&lt;/span&gt;
3465&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
3466&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After that I went on a hunt again. I won&#39;t bother you with all the details
3467because hours/days went by until I was finally able to re-index the problematic
3468index and hoped for the best. Until that moment even re-indexing was giving me
3469errors.&lt;/p&gt;
3470&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;POST _reindex
3471&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
3472&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;source&amp;#34;: {
3473&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;index&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;myindex&amp;#34;&lt;/span&gt;
3474&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
3475&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;dest&amp;#34;: {
3476&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;index&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;myindex-new&amp;#34;&lt;/span&gt;
3477&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
3478&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
3479&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;I needed to do this multiple times to get all the documents re-indexed. Then I
3480dropped the original one with the following command.&lt;/p&gt;
3481&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;DELETE /myindex
3482&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And re-indexed again new one in the original one (well by name only).&lt;/p&gt;
3483&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;POST _reindex
3484&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
3485&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;source&amp;#34;: {
3486&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;index&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;myindex-new&amp;#34;&lt;/span&gt;
3487&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
3488&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;dest&amp;#34;: {
3489&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;index&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;myindex&amp;#34;&lt;/span&gt;
3490&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
3491&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
3492&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;On the surface it looks like all is working but I have a long road in front of
3493me to get all the things working again. Cluster now shows that it is in Green
3494mode but I am also getting a notification that the cluster has processing status
3495which could mean million of things.&lt;/p&gt;
3496&lt;p&gt;Godspeed!&lt;/p&gt;
3497</content:encoded>
3498 </item>
3499
3500
3501
3502 <item>
3503 <title>Create placeholder images with sharp Node.js image processing library</title>
3504 <link>https://mitjafelicijan.com/create-placeholder-images-with-sharp.html</link>
3505 <pubDate>Fri, 27 Mar 2020 12:00:00 &#43;0200</pubDate>
3506 <guid>https://mitjafelicijan.com/create-placeholder-images-with-sharp.html</guid>
3507 <description>I have been searching for a solution to pre-generate some placeholder images forimage server I needed to develop that resizes images on S3.</description>
3508 <content:encoded>&lt;p&gt;I have been searching for a solution to pre-generate some placeholder images for
3509image server I needed to develop that resizes images on S3. I though this would
3510be a 15min job and quickly found out how very mistaken I was.&lt;/p&gt;
3511&lt;p&gt;Even though Node.js is not really the best way to do this kind of things (surely
3512something written in C or Rust or even Golang would be the correct way to do
3513this but we didn&#39;t need the speed in our case) I found an excellent library
3514&lt;a href=&#34;https://github.com/lovell/sharp&#34;&gt;sharp - High performance Node.js image
3515processing&lt;/a&gt;.&lt;/p&gt;
3516&lt;p&gt;Getting things running was a breeze.&lt;/p&gt;
3517&lt;h2 id=&#34;fetch-image-from-s3-and-save-resized&#34;&gt;Fetch image from S3 and save resized&lt;/h2&gt;
3518&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; sharp = require(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;sharp&amp;#39;&lt;/span&gt;);
3519&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; aws = require(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;aws-sdk&amp;#39;&lt;/span&gt;);
3520&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3521&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; x,y = 100;
3522&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; s3 = &lt;span style=&#34;color:#00f&#34;&gt;new&lt;/span&gt; aws.S3({});
3523&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3524&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;aws.config.update({
3525&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; secretAccessKey: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;secretAccessKey&amp;#39;&lt;/span&gt;,
3526&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; accessKeyId: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;accessKeyId&amp;#39;&lt;/span&gt;,
3527&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; region: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;region&amp;#39;&lt;/span&gt;
3528&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3529&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3530&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; originalImage = &lt;span style=&#34;color:#00f&#34;&gt;await&lt;/span&gt; s3.getObject({
3531&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Bucket: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;some-bucket-name&amp;#39;&lt;/span&gt;,
3532&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Key: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;image.jpg&amp;#39;&lt;/span&gt;,
3533&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}).promise();
3534&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3535&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; resizedImage = &lt;span style=&#34;color:#00f&#34;&gt;await&lt;/span&gt; sharp(originalImage.Body)
3536&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .resize(x, y)
3537&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .jpeg({ progressive: &lt;span style=&#34;color:#00f&#34;&gt;true&lt;/span&gt; })
3538&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .toBuffer();
3539&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3540&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;s3.putObject({
3541&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Bucket: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;some-bucket-name&amp;#39;&lt;/span&gt;,
3542&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Key: &lt;span style=&#34;color:#a31515&#34;&gt;`optimized/&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;x&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;x&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;y&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;/image.jpg`&lt;/span&gt;,
3543&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Body: resizedImage,
3544&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ContentType: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;image/jpeg&amp;#39;&lt;/span&gt;,
3545&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ACL: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;public-read&amp;#39;&lt;/span&gt;
3546&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}).promise();
3547&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;All this code was wrapped inside a web service with some additional security
3548checks and defensive coding to detect if key is missing on S3.&lt;/p&gt;
3549&lt;p&gt;And at that point I needed to return placeholder images as a response in case
3550key is missing or x,y are not allowed by the server etc. I could have created
3551PNG in Gimp and just serve them but I wanted to respect aspect ratio and I
3552didn&#39;t want to return some mangled images.&lt;/p&gt;
3553&lt;blockquote&gt;
3554&lt;p&gt;Main problem with finding a clean solution I could copy and paste and change a
3555bit was a task. API is changing constantly and there weren&#39;t clear examples or
3556I was unable to find them.&lt;/p&gt;
3557&lt;/blockquote&gt;
3558&lt;h2 id=&#34;generating-placeholder-images-using-svg&#34;&gt;Generating placeholder images using SVG&lt;/h2&gt;
3559&lt;p&gt;What I ended up was using SVG to generate text and created image with sharp and
3560used composition to combine both layers. Response returned by this function is a
3561buffer you can use to either upload to S3 or save to local file.&lt;/p&gt;
3562&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; generatePlaceholderImageWithText = &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (width, height, message) =&amp;gt; {
3563&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; overlay = &lt;span style=&#34;color:#a31515&#34;&gt;`&amp;lt;svg width=&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;width - 20&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34; height=&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;height - 20&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;gt;
3564&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; &amp;lt;text x=&amp;#34;50%&amp;#34; y=&amp;#34;50%&amp;#34; font-family=&amp;#34;sans-serif&amp;#34; font-size=&amp;#34;16&amp;#34; text-anchor=&amp;#34;middle&amp;#34;&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;message&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;lt;/text&amp;gt;
3565&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; &amp;lt;/svg&amp;gt;`&lt;/span&gt;;
3566&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3567&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;await&lt;/span&gt; sharp({
3568&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; create: {
3569&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; width: width,
3570&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; height: height,
3571&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; channels: 4,
3572&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; background: { r: 230, g: 230, b: 230, alpha: 1 }
3573&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
3574&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; })
3575&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .composite([{
3576&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; input: Buffer.from(overlay),
3577&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; gravity: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;center&amp;#39;&lt;/span&gt;,
3578&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }])
3579&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .jpeg()
3580&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .toBuffer();
3581&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
3582&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;That is about it. Nothing more to it. You can change the color of the image by
3583changing &lt;code&gt;background&lt;/code&gt; and if you want to change text styling you can adapt SVG
3584to your needs.&lt;/p&gt;
3585&lt;blockquote&gt;
3586&lt;p&gt;Also be careful about the length of the text. This function positions text at
3587the center and adds &lt;code&gt;20px&lt;/code&gt; padding on all sides. If text is longer than the
3588image it will get cut.&lt;/p&gt;
3589&lt;/blockquote&gt;
3590</content:encoded>
3591 </item>
3592
3593
3594
3595 <item>
3596 <title>Simple Server-Sent Events based PubSub Server</title>
3597 <link>https://mitjafelicijan.com/simple-server-sent-events-based-pubsub-server.html</link>
3598 <pubDate>Sun, 22 Mar 2020 12:00:00 &#43;0200</pubDate>
3599 <guid>https://mitjafelicijan.com/simple-server-sent-events-based-pubsub-server.html</guid>
3600 <description>Before we continue .</description>
3601 <content:encoded>&lt;h2 id=&#34;before-we-continue-&#34;&gt;Before we continue ...&lt;/h2&gt;
3602&lt;p&gt;Publisher Subscriber model is nothing new and there are many amazing solutions
3603out there, so writing a new one would be a waste of time if other solutions
3604wouldn&#39;t have quite complex install procedures and weren&#39;t so hard to maintain.
3605But to be fair, comparing this simple server with something like
3606&lt;a href=&#34;https://kafka.apache.org/&#34;&gt;Kafka&lt;/a&gt; or &lt;a href=&#34;https://www.rabbitmq.com/&#34;&gt;RabbitMQ&lt;/a&gt; is
3607laughable at the least. Those solutions are enterprise grade and have many
3608mechanisms there to ensure messages aren&#39;t lost and much more. Regardless of
3609these drawbacks, this method has been tested on a large website and worked until
3610now without any problems. So now, that we got that cleared up, let&#39;s continue.&lt;/p&gt;
3611&lt;p&gt;&lt;em&gt;&lt;strong&gt;Wiki definition:&lt;/strong&gt; Publish/subscribe messaging, or pub/sub messaging, is a
3612form of asynchronous service-to-service communication used in serverless and
3613microservices architectures. In a pub/sub model, any message published to a
3614topic is immediately received by all the subscribers to the topic.&lt;/em&gt;&lt;/p&gt;
3615&lt;h2 id=&#34;general-goals&#34;&gt;General goals&lt;/h2&gt;
3616&lt;ul&gt;
3617&lt;li&gt;provide a simple server that relays messages to all the connected clients,&lt;/li&gt;
3618&lt;li&gt;messages can be posted on specific topics,&lt;/li&gt;
3619&lt;li&gt;messages get sent via &lt;a href=&#34;https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events&#34;&gt;Server-Sent
3620Events&lt;/a&gt;
3621to all the subscribers.&lt;/li&gt;
3622&lt;/ul&gt;
3623&lt;h2 id=&#34;how-exactly-does-the-pubsub-model-work&#34;&gt;How exactly does the pub/sub model work?&lt;/h2&gt;
3624&lt;p&gt;The easiest way to explain this is with diagram bellow. Basic function is
3625simple. We have subscribers that receive messages, and we have publishers that
3626create and post messages. Similar model is also well know pattern that works on
3627a premise of consumers and producers, and they take similar roles.&lt;/p&gt;
3628&lt;p&gt;&lt;img src=&#34;/assets/simple-pubsub-server/pubsub-overview.png&#34; alt=&#34;How PubSub works&#34; /&gt;&lt;/p&gt;
3629&lt;p&gt;&lt;strong&gt;These are some naive characteristics we want to achieve:&lt;/strong&gt;&lt;/p&gt;
3630&lt;ul&gt;
3631&lt;li&gt;producer is publishing messages to subscribe topic,&lt;/li&gt;
3632&lt;li&gt;consumer is receiving messages from subscribed topic,&lt;/li&gt;
3633&lt;li&gt;servers is also known as Broker,&lt;/li&gt;
3634&lt;li&gt;broker does not store messages or tracks success,&lt;/li&gt;
3635&lt;li&gt;broker uses
3636&lt;a href=&#34;https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)&#34;&gt;FIFO&lt;/a&gt; method
3637for delivering messages,&lt;/li&gt;
3638&lt;li&gt;if consumer wants to receive messages from a topic, producer and consumer
3639topics must match,&lt;/li&gt;
3640&lt;li&gt;consumer can subscribe to multiple topics,&lt;/li&gt;
3641&lt;li&gt;producer can publish to multiple topics,&lt;/li&gt;
3642&lt;li&gt;each message has a messageId.&lt;/li&gt;
3643&lt;/ul&gt;
3644&lt;p&gt;&lt;strong&gt;Known drawbacks:&lt;/strong&gt;&lt;/p&gt;
3645&lt;ul&gt;
3646&lt;li&gt;messages will not be stored in a persistent queue or unreceived messages like
3647&lt;a href=&#34;https://en.wikipedia.org/wiki/Dead_letter_queue&#34;&gt;DeadLetterQueue&lt;/a&gt; so old
3648messages could be lost on server restart,&lt;/li&gt;
3649&lt;li&gt;&lt;a href=&#34;https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events&#34;&gt;Server-Sent
3650Events&lt;/a&gt;
3651opens a long-running connection between the client and the server so make sure
3652if your setup is load balanced that the load balancer in this case can have
3653long opened connection,&lt;/li&gt;
3654&lt;li&gt;no system moderation due to the dynamic nature of creating queues.&lt;/li&gt;
3655&lt;/ul&gt;
3656&lt;h2 id=&#34;server-sent-events&#34;&gt;Server-Sent Events&lt;/h2&gt;
3657&lt;p&gt;Read more about it on &lt;a href=&#34;https://html.spec.whatwg.org/multipage/server-sent-events.html&#34;&gt;official specification
3658page&lt;/a&gt;.&lt;/p&gt;
3659&lt;h3 id=&#34;current-browser-support&#34;&gt;Current browser support&lt;/h3&gt;
3660&lt;p&gt;&lt;img src=&#34;/assets/simple-pubsub-server/caniuse.png&#34; alt=&#34;Browser support&#34; /&gt;&lt;/p&gt;
3661&lt;p&gt;Check
3662&lt;a href=&#34;https://caniuse.com/#feat=eventsource&#34;&gt;https://caniuse.com/#feat=eventsource&lt;/a&gt;
3663for latest information about browser support.&lt;/p&gt;
3664&lt;h3 id=&#34;known-issues&#34;&gt;Known issues&lt;/h3&gt;
3665&lt;ul&gt;
3666&lt;li&gt;Firefox 52 and below do not support EventSource in web/shared workers&lt;/li&gt;
3667&lt;li&gt;In Firefox prior to version 36 server-sent events do not reconnect
3668automatically in case of a connection interrupt (bug)&lt;/li&gt;
3669&lt;li&gt;Reportedly, CORS in EventSource is currently supported in Firefox 10&#43;, Opera
367012&#43;, Chrome 26&#43;, Safari 7.0&#43;.&lt;/li&gt;
3671&lt;li&gt;Antivirus software may block the event streaming data chunks.&lt;/li&gt;
3672&lt;/ul&gt;
3673&lt;p&gt;Source: &lt;a href=&#34;https://caniuse.com/#feat=eventsource&#34;&gt;https://caniuse.com/#feat=eventsource&lt;/a&gt;&lt;/p&gt;
3674&lt;h3 id=&#34;message-format&#34;&gt;Message format&lt;/h3&gt;
3675&lt;p&gt;The simplest message that can be sent is only with data attribute:&lt;/p&gt;
3676&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;data: this is a simple message
3677&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;blank line&amp;gt;
3678&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;You can send message IDs to be used if the connection is dropped:&lt;/p&gt;
3679&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;id: 33
3680&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;data: this is line one
3681&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;data: this is line two
3682&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;blank line&amp;gt;
3683&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And you can specify your own event types (the above messages will all trigger
3684the message event):&lt;/p&gt;
3685&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;id: 36
3686&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;event: price
3687&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;data: 103.34
3688&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;blank line&amp;gt;
3689&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;server-requirements&#34;&gt;Server requirements&lt;/h3&gt;
3690&lt;p&gt;The important thing is how you send headers and which headers are sent by the
3691server that triggers browser to threat response as a EventStream.&lt;/p&gt;
3692&lt;p&gt;Headers responsible for this are:&lt;/p&gt;
3693&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Content-Type: text/event-stream
3694&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Cache-Control: no-cache
3695&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Connection: keep-alive
3696&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;debugging-with-google-chrome&#34;&gt;Debugging with Google Chrome&lt;/h3&gt;
3697&lt;p&gt;Google Chrome provides build-in debugging and exploration tool for &lt;a href=&#34;https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events&#34;&gt;Server-Sent
3698Events&lt;/a&gt;
3699which is quite nice and available from Developer Tools under Network tab.&lt;/p&gt;
3700&lt;blockquote&gt;
3701&lt;p&gt;You can debug only client side events that get received and not the server
3702ones. For debugging server events add &lt;code&gt;console.log&lt;/code&gt; to &lt;code&gt;server.js&lt;/code&gt; code and
3703print out events.&lt;/p&gt;
3704&lt;/blockquote&gt;
3705&lt;p&gt;&lt;img src=&#34;/assets/simple-pubsub-server/chrome-debugging.png&#34; alt=&#34;Google Chrome Developer Tools EventStream&#34; /&gt;&lt;/p&gt;
3706&lt;h2 id=&#34;server-implementation&#34;&gt;Server implementation&lt;/h2&gt;
3707&lt;p&gt;For the sake of this example we will use &lt;a href=&#34;https://nodejs.org/en/&#34;&gt;Node.js&lt;/a&gt; with
3708&lt;a href=&#34;https://expressjs.com&#34;&gt;Express&lt;/a&gt; as our router since this is the easiest way to
3709get started and we will use already written SSE library for node
3710&lt;a href=&#34;https://www.npmjs.com/package/sse-pubsub&#34;&gt;sse-pubsub&lt;/a&gt; so we don&#39;t reinvent the
3711wheel.&lt;/p&gt;
3712&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;npm init --yes
3713&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3714&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;npm install express
3715&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;npm install body-parser
3716&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;npm install sse-pubsub
3717&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Basic implementation of a server (&lt;code&gt;server.js&lt;/code&gt;):&lt;/p&gt;
3718&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; express = require(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;express&amp;#39;&lt;/span&gt;);
3719&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; bodyParser = require(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;body-parser&amp;#39;&lt;/span&gt;);
3720&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; SSETopic = require(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;sse-pubsub&amp;#39;&lt;/span&gt;);
3721&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3722&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; app = express();
3723&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; port = process.env.PORT || 4000;
3724&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3725&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// topics container
3726&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; sseTopics = {};
3727&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3728&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app.use(bodyParser.json());
3729&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3730&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// open for all cors
3731&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.all(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;*&amp;#39;&lt;/span&gt;, (req, res, next) =&amp;gt; {
3732&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Access-Control-Allow-Origin&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;*&amp;#39;&lt;/span&gt;);
3733&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Access-Control-Allow-Headers&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;X-Requested-With, Content-Type&amp;#39;&lt;/span&gt;);
3734&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; next();
3735&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3736&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3737&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// preflight request error fix
3738&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.options(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;*&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (req, res) =&amp;gt; {
3739&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Access-Control-Allow-Origin&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;*&amp;#39;&lt;/span&gt;);
3740&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Access-Control-Allow-Headers&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;X-Requested-With, Content-Type&amp;#39;&lt;/span&gt;);
3741&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.send(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;OK&amp;#39;&lt;/span&gt;);
3742&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3743&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3744&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// serve the event streams
3745&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.get(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;/stream/:topic&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (req, res, next) =&amp;gt; {
3746&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; topic = req.params.topic;
3747&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3748&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (!(topic &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; sseTopics)) {
3749&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; sseTopics[topic] = &lt;span style=&#34;color:#00f&#34;&gt;new&lt;/span&gt; SSETopic({
3750&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; pingInterval: 0,
3751&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; maxStreamDuration: 15000,
3752&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3753&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
3754&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3755&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// subscribing client to topic
3756&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; sseTopics[topic].subscribe(req, res);
3757&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3758&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3759&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// accepts new messages into topic
3760&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.post(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;/publish&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (req, res) =&amp;gt; {
3761&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;let&lt;/span&gt; body = req.body;
3762&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;let&lt;/span&gt; status = 200;
3763&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3764&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; console.log(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Incoming message:&amp;#39;&lt;/span&gt;, req.body);
3765&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3766&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (
3767&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; body.hasOwnProperty(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;topic&amp;#39;&lt;/span&gt;) &amp;amp;&amp;amp;
3768&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; body.hasOwnProperty(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;event&amp;#39;&lt;/span&gt;) &amp;amp;&amp;amp;
3769&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; body.hasOwnProperty(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;message&amp;#39;&lt;/span&gt;)
3770&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ) {
3771&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; topic = req.body.topic;
3772&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; event = req.body.event;
3773&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; message = req.body.message;
3774&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3775&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (topic &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; sseTopics) {
3776&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// sends message to all the subscribers
3777&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; sseTopics[topic].publish(message, event);
3778&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
3779&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; } &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt; {
3780&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; status = 400;
3781&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
3782&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3783&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.status(status).send({
3784&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; status,
3785&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3786&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3787&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3788&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// returns JSON object of all opened topics
3789&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.get(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;/status&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (req, res) =&amp;gt; {
3790&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.send(sseTopics);
3791&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3792&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3793&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// health-check endpoint
3794&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.get(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;/&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (req, res) =&amp;gt; {
3795&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.send(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;OK&amp;#39;&lt;/span&gt;);
3796&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3797&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3798&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// return a 404 if no routes match
3799&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.use((req, res, next) =&amp;gt; {
3800&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.set(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Cache-Control&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;private, no-store&amp;#39;&lt;/span&gt;);
3801&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res.status(404).end(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Not found&amp;#39;&lt;/span&gt;);
3802&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3803&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3804&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// starts the server
3805&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;app.listen(port, () =&amp;gt; {
3806&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; console.log(&lt;span style=&#34;color:#a31515&#34;&gt;`PubSub server running on http://localhost:&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;port&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt;);
3807&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;});
3808&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;our-custom-message-format&#34;&gt;Our custom message format&lt;/h3&gt;
3809&lt;p&gt;Each message posted on a server must be in a specific format that out server
3810accepts. Having structure like this allows us to have multiple separated type of
3811events on each topic.&lt;/p&gt;
3812&lt;p&gt;With this we can separate streams and only receive events that belong to the
3813topic.&lt;/p&gt;
3814&lt;p&gt;One example would be, that we have index page and we want to receive messages
3815about new upvotes or new subscribers but we don&#39;t want to follow events for
3816other pages. This reduces clutter and overall network. And structure is much
3817nicer and maintanable.&lt;/p&gt;
3818&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
3819&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;topic&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sample-topic&amp;#34;&lt;/span&gt;,
3820&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;event&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sample-event&amp;#34;&lt;/span&gt;,
3821&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;message&amp;#34;: { &amp;#34;name&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;John&amp;#34;&lt;/span&gt; }
3822&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
3823&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;publisher-and-subscriber-clients&#34;&gt;Publisher and subscriber clients&lt;/h2&gt;
3824&lt;h3 id=&#34;publisher-and-subscriber-in-action&#34;&gt;Publisher and subscriber in action&lt;/h3&gt;
3825&lt;p&gt;&lt;video src=&#34;/assets/simple-pubsub-server/clients.m4v&#34; controls&gt;&lt;/video&gt;&lt;/p&gt;
3826&lt;p&gt;You can download &lt;a href=&#34;../simple-pubsub-server/sse-pubsub-server.zip&#34;&gt;the code&lt;/a&gt; and
3827follow along.&lt;/p&gt;
3828&lt;h3 id=&#34;publisher&#34;&gt;Publisher&lt;/h3&gt;
3829&lt;p&gt;As talked about above publisher is the one that send messages to the
3830broker/server. Message inside the payload can be whatever you want (string,
3831object, array). I would however personally avoid send large chunks of data like
3832blobs and such.&lt;/p&gt;
3833&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;!DOCTYPE html&amp;gt;&lt;/span&gt;
3834&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;html lang=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;en&amp;#34;&lt;/span&gt;&amp;gt;
3835&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3836&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;head&amp;gt;
3837&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;meta charset=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;UTF-8&amp;#34;&lt;/span&gt;&amp;gt;
3838&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;meta name=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;viewport&amp;#34;&lt;/span&gt; content=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;width=device-width, initial-scale=1.0&amp;#34;&lt;/span&gt;&amp;gt;
3839&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;title&amp;gt;Publisher&amp;lt;/title&amp;gt;
3840&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/head&amp;gt;
3841&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3842&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;body&amp;gt;
3843&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3844&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;h1&amp;gt;Publisher&amp;lt;/h1&amp;gt;
3845&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3846&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;fieldset&amp;gt;
3847&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3848&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;label&amp;gt;Server:&amp;lt;/label&amp;gt;
3849&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;input type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;text&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;server&amp;#34;&lt;/span&gt; value=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;http://localhost:4000&amp;#34;&lt;/span&gt;&amp;gt;
3850&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3851&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3852&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;label&amp;gt;Topic:&amp;lt;/label&amp;gt;
3853&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;input type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;text&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;topic&amp;#34;&lt;/span&gt; value=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sample-topic&amp;#34;&lt;/span&gt;&amp;gt;
3854&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3855&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3856&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;label&amp;gt;Event:&amp;lt;/label&amp;gt;
3857&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;input type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;text&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;event&amp;#34;&lt;/span&gt; value=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sample-event&amp;#34;&lt;/span&gt;&amp;gt;
3858&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3859&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3860&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;label&amp;gt;Message:&amp;lt;/label&amp;gt;
3861&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;input type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;text&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;message&amp;#34;&lt;/span&gt; value=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;{&amp;#34;name&amp;#34;: &amp;#34;John&amp;#34;}&amp;#39;&lt;/span&gt;&amp;gt;
3862&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3863&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3864&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;button type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;button&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;button&amp;#34;&lt;/span&gt;&amp;gt;Publish message to topic&amp;lt;/button&amp;gt;
3865&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3866&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/fieldset&amp;gt;
3867&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3868&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;script&amp;gt;
3869&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3870&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; button = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#button&amp;#39;&lt;/span&gt;);
3871&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; server = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#server&amp;#39;&lt;/span&gt;);
3872&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; topic = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#topic&amp;#39;&lt;/span&gt;);
3873&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; event = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#event&amp;#39;&lt;/span&gt;);
3874&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; message = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#message&amp;#39;&lt;/span&gt;);
3875&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3876&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; button.addEventListener(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;click&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (evt) =&amp;gt; {
3877&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; req = &lt;span style=&#34;color:#00f&#34;&gt;await&lt;/span&gt; fetch(&lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;server.value&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;/publish`&lt;/span&gt;, {
3878&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; method: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;post&amp;#39;&lt;/span&gt;,
3879&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; headers: {
3880&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Accept&amp;#39;&lt;/span&gt;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;application/json&amp;#39;&lt;/span&gt;,
3881&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Content-Type&amp;#39;&lt;/span&gt;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;application/json&amp;#39;&lt;/span&gt;,
3882&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
3883&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; body: JSON.stringify({
3884&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; topic: topic.value,
3885&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; event: event.value,
3886&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; message: JSON.parse(message.value),
3887&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }),
3888&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3889&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3890&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; res = &lt;span style=&#34;color:#00f&#34;&gt;await&lt;/span&gt; req.json();
3891&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; console.log(res);
3892&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3893&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3894&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/script&amp;gt;
3895&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3896&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/body&amp;gt;
3897&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3898&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;/html&amp;gt;
3899&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;subscriber&#34;&gt;Subscriber&lt;/h3&gt;
3900&lt;p&gt;Subscriber is responsible for receiving new messages that come from server via
3901publisher. The code bellow is very rudimentary but works and follows the
3902implementation guidelines for EventSource.&lt;/p&gt;
3903&lt;p&gt;You can use either Developer Tools Console to see incoming messages or you can
3904defer to Debugging with Google Chrome section above to see all EventStream
3905messages.&lt;/p&gt;
3906&lt;blockquote&gt;
3907&lt;p&gt;Don&#39;t be alarmed if the subscriber gets disconnected from the server every so
3908often. The code we have here resets connection every 15s but it automatically
3909get reconnected and fetches all messages up to last received message id. This
3910setting can be adjusted in &lt;code&gt;server.js&lt;/code&gt; file; search for the
3911&lt;code&gt;maxStreamDuration&lt;/code&gt; variable.&lt;/p&gt;
3912&lt;/blockquote&gt;
3913&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;!DOCTYPE html&amp;gt;&lt;/span&gt;
3914&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;html lang=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;en&amp;#34;&lt;/span&gt;&amp;gt;
3915&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3916&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;head&amp;gt;
3917&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;meta charset=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;UTF-8&amp;#34;&lt;/span&gt;&amp;gt;
3918&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;meta name=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;viewport&amp;#34;&lt;/span&gt; content=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;width=device-width, initial-scale=1.0&amp;#34;&lt;/span&gt;&amp;gt;
3919&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;title&amp;gt;Subscriber&amp;lt;/title&amp;gt;
3920&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;link rel=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;stylesheet&amp;#34;&lt;/span&gt; href=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;style.css&amp;#34;&lt;/span&gt;&amp;gt;
3921&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/head&amp;gt;
3922&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3923&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;body&amp;gt;
3924&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3925&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;h1&amp;gt;Subscriber&amp;lt;/h1&amp;gt;
3926&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3927&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;fieldset&amp;gt;
3928&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3929&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;label&amp;gt;Server:&amp;lt;/label&amp;gt;
3930&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;input type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;text&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;server&amp;#34;&lt;/span&gt; value=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;http://localhost:4000&amp;#34;&lt;/span&gt;&amp;gt;
3931&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3932&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3933&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;label&amp;gt;Topic:&amp;lt;/label&amp;gt;
3934&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;input type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;text&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;topic&amp;#34;&lt;/span&gt; value=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sample-topic&amp;#34;&lt;/span&gt;&amp;gt;
3935&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3936&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3937&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;label&amp;gt;Event:&amp;lt;/label&amp;gt;
3938&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;input type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;text&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;event&amp;#34;&lt;/span&gt; value=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sample-event&amp;#34;&lt;/span&gt;&amp;gt;
3939&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3940&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;p&amp;gt;
3941&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;button type=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;button&amp;#34;&lt;/span&gt; id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;button&amp;#34;&lt;/span&gt;&amp;gt;Subscribe to topic&amp;lt;/button&amp;gt;
3942&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/p&amp;gt;
3943&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/fieldset&amp;gt;
3944&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3945&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;script&amp;gt;
3946&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3947&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; button = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#button&amp;#39;&lt;/span&gt;);
3948&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; server = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#server&amp;#39;&lt;/span&gt;);
3949&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; topic = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#topic&amp;#39;&lt;/span&gt;);
3950&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;const&lt;/span&gt; event = document.querySelector(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;#event&amp;#39;&lt;/span&gt;);
3951&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3952&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; button.addEventListener(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;click&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;async&lt;/span&gt; (evt) =&amp;gt; {
3953&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3954&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;let&lt;/span&gt; es = &lt;span style=&#34;color:#00f&#34;&gt;new&lt;/span&gt; EventSource(&lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;server.value&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;/stream/&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;${&lt;/span&gt;topic.value&lt;span style=&#34;color:#a31515&#34;&gt;}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;`&lt;/span&gt;);
3955&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3956&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; es.addEventListener(event.value, &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt; (evt) {
3957&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; console.log(&lt;span style=&#34;color:#a31515&#34;&gt;`incoming message`&lt;/span&gt;, JSON.parse(evt.data));
3958&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3959&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3960&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; es.addEventListener(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;open&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt; (evt) {
3961&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; console.log(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;connected&amp;#39;&lt;/span&gt;, evt);
3962&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3963&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3964&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; es.addEventListener(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;error&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt; (evt) {
3965&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; console.log(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;error&amp;#39;&lt;/span&gt;, evt);
3966&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3967&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3968&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
3969&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3970&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/script&amp;gt;
3971&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3972&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/body&amp;gt;
3973&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
3974&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;/html&amp;gt;
3975&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;reading-further&#34;&gt;Reading further&lt;/h2&gt;
3976&lt;ul&gt;
3977&lt;li&gt;&lt;a href=&#34;https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events&#34;&gt;Using server-sent events&lt;/a&gt;&lt;/li&gt;
3978&lt;li&gt;&lt;a href=&#34;https://www.smashingmagazine.com/2018/02/sse-websockets-data-flow-http2/&#34;&gt;Using SSE Instead Of WebSockets For Unidirectional Data Flow Over HTTP/2&lt;/a&gt;&lt;/li&gt;
3979&lt;li&gt;&lt;a href=&#34;https://apifriends.com/api-streaming/server-sent-events/&#34;&gt;What is Server-Sent Events?&lt;/a&gt;&lt;/li&gt;
3980&lt;li&gt;&lt;a href=&#34;https://tools.ietf.org/id/draft-xie-bidirectional-messaging-01.html&#34;&gt;An HTTP/2 extension for bidirectional messaging communication&lt;/a&gt;&lt;/li&gt;
3981&lt;li&gt;&lt;a href=&#34;https://developers.google.com/web/fundamentals/performance/http2&#34;&gt;Introduction to HTTP/2&lt;/a&gt;&lt;/li&gt;
3982&lt;li&gt;&lt;a href=&#34;https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API&#34;&gt;The WebSocket API (WebSockets)&lt;/a&gt;&lt;/li&gt;
3983&lt;/ul&gt;
3984</content:encoded>
3985 </item>
3986
3987
3988
3989 <item>
3990 <title>Using sentiment analysis for clickbait detection in RSS feeds</title>
3991 <link>https://mitjafelicijan.com/using-sentiment-analysis-for-clickbait-detection-in-rss-feeds.html</link>
3992 <pubDate>Sat, 19 Oct 2019 12:00:00 &#43;0200</pubDate>
3993 <guid>https://mitjafelicijan.com/using-sentiment-analysis-for-clickbait-detection-in-rss-feeds.html</guid>
3994 <description>Initial thoughtsOne of the things that interested me for a while now is if major wellestablished news sites use click bait titles to drive additional traffic totheir sites and generate additional impressions.</description>
3995 <content:encoded>&lt;h2 id=&#34;initial-thoughts&#34;&gt;Initial thoughts&lt;/h2&gt;
3996&lt;p&gt;One of the things that interested me for a while now is if major well
3997established news sites use click bait titles to drive additional traffic to
3998their sites and generate additional impressions.&lt;/p&gt;
3999&lt;p&gt;Goal is to see how article titles and actual content of article differ from each
4000other and see if titles are clickbaited.&lt;/p&gt;
4001&lt;h2 id=&#34;preparing-and-cleaning-data&#34;&gt;Preparing and cleaning data&lt;/h2&gt;
4002&lt;p&gt;For this example I opted to just use RSS feed from a new website and decided to
4003go with &lt;a href=&#34;https://www.theguardian.com&#34;&gt;The Guardian&lt;/a&gt; World news. While this gets
4004us limited data (~40) articles and also description (actual content) is trimmed
4005this really doesn&#39;t reflect the actual article contents.&lt;/p&gt;
4006&lt;p&gt;To get better content I could use web scraping and use RSS as link list and
4007fetch contents directly from website, but for this simple example this will
4008suffice.&lt;/p&gt;
4009&lt;p&gt;There are couple of requirements we need to install before we continue:&lt;/p&gt;
4010&lt;ul&gt;
4011&lt;li&gt;&lt;code&gt;pip3 install feedparser&lt;/code&gt; (parses RSS feed from url)&lt;/li&gt;
4012&lt;li&gt;&lt;code&gt;pip3 install vaderSentiment&lt;/code&gt; (does sentiment polarity analysis)&lt;/li&gt;
4013&lt;li&gt;&lt;code&gt;pip3 install matplotlib&lt;/code&gt; (plots chart of results)&lt;/li&gt;
4014&lt;/ul&gt;
4015&lt;p&gt;So first we need to fetch RSS data and sanitize HTML content from description.&lt;/p&gt;
4016&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; re
4017&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; feedparser
4018&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4019&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;feed_url = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;https://www.theguardian.com/world/rss&amp;#34;&lt;/span&gt;
4020&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;feed = feedparser.parse(feed_url)
4021&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4022&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# sanitize html&lt;/span&gt;
4023&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; item &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; feed.entries:
4024&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; item.description = re.sub(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&amp;lt;[^&amp;lt;]&#43;?&amp;gt;&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;&amp;#39;&lt;/span&gt;, item.description)
4025&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;perform-sentiment-analysis&#34;&gt;Perform sentiment analysis&lt;/h2&gt;
4026&lt;p&gt;Since we now have cleaned up data in our &lt;code&gt;feed.entries&lt;/code&gt; object we can start with
4027performing sentiment analysis.&lt;/p&gt;
4028&lt;p&gt;There are many sentiment analysis libraries available that range from rule-based
4029sentiment analysis up to machine learning supported analysis. To keep things
4030simple I decided to use rule-based analysis library
4031&lt;a href=&#34;https://github.com/cjhutto/vaderSentiment&#34;&gt;vaderSentiment&lt;/a&gt; from
4032&lt;a href=&#34;https://github.com/cjhutto&#34;&gt;C.J. Hutto&lt;/a&gt;. Really nice library and quite easy to
4033use.&lt;/p&gt;
4034&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; vaderSentiment.vaderSentiment &lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; SentimentIntensityAnalyzer
4035&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;analyser = SentimentIntensityAnalyzer()
4036&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4037&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sentiment_results = []
4038&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; item &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; feed.entries:
4039&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; sentiment_title = analyser.polarity_scores(item.title)
4040&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; sentiment_description = analyser.polarity_scores(item.description)
4041&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; sentiment_results.append([sentiment_title[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;compound&amp;#39;&lt;/span&gt;], sentiment_description[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;compound&amp;#39;&lt;/span&gt;]])
4042&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now that we have this data in a shape that is compatible with matplotlib we can
4043plot results to see the difference between title and description sentiment of an
4044article.&lt;/p&gt;
4045&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; matplotlib.pyplot &lt;span style=&#34;color:#00f&#34;&gt;as&lt;/span&gt; plt
4046&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4047&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;plt.rcParams[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;figure.figsize&amp;#39;&lt;/span&gt;] = (15, 3)
4048&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;plt.plot(sentiment_results, drawstyle=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;steps&amp;#39;&lt;/span&gt;)
4049&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;plt.title(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;Sentiment analysis relationship between title and description (Guardian World News)&amp;#39;&lt;/span&gt;)
4050&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;plt.legend([&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;title&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;description&amp;#39;&lt;/span&gt;])
4051&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;plt.show()
4052&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;results-and-assets&#34;&gt;Results and assets&lt;/h2&gt;
4053&lt;ol&gt;
4054&lt;li&gt;Because of the small sample size further conclusions are impossible to make.&lt;/li&gt;
4055&lt;li&gt;Rule-based approach may not be the best way of doing this. By using deep
4056learning we would be able to get better insights.&lt;/li&gt;
4057&lt;li&gt;&lt;strong&gt;Next step would be to&lt;/strong&gt; periodically fetch RSS items and store them over a
4058longer period of time and then perform analysis again and use either machine
4059learning or deep learning on top of it.&lt;/li&gt;
4060&lt;/ol&gt;
4061&lt;p&gt;&lt;img src=&#34;/assets/sentiment-analysis/guardian-sa-title-desc-relationship.png&#34; alt=&#34;Relationship between title and description&#34; /&gt;&lt;/p&gt;
4062&lt;p&gt;Figure above displays difference between title and description sentiment for
4063specific RSS feed item. 1 means positive and -1 means negative sentiment.&lt;/p&gt;
4064&lt;p&gt;&lt;a href=&#34;/assets/sentiment-analysis/sentiment-analysis.ipynb&#34;&gt;» Download Jupyter Notebook&lt;/a&gt;&lt;/p&gt;
4065&lt;h2 id=&#34;going-further&#34;&gt;Going further&lt;/h2&gt;
4066&lt;ul&gt;
4067&lt;li&gt;&lt;a href=&#34;https://github.com/bswiss/news_mood&#34;&gt;Twitter Sentiment Analysis by Bryan Schwierzke&lt;/a&gt;&lt;/li&gt;
4068&lt;li&gt;&lt;a href=&#34;https://github.com/thisandagain/sentiment&#34;&gt;AFINN-based sentiment analysis for Node.js by Andrew Sliwinski&lt;/a&gt;&lt;/li&gt;
4069&lt;li&gt;&lt;a href=&#34;https://github.com/adeshpande3/LSTM-Sentiment-Analysis&#34;&gt;Sentiment Analysis with LSTMs in Tensorflow by Adit Deshpande&lt;/a&gt;&lt;/li&gt;
4070&lt;li&gt;&lt;a href=&#34;https://github.com/abdulfatir/twitter-sentiment-analysis&#34;&gt;Sentiment analysis on tweets using Naive Bayes, SVM, CNN, LSTM, etc. by Abdul Fatir&lt;/a&gt;&lt;/li&gt;
4071&lt;/ul&gt;
4072</content:encoded>
4073 </item>
4074
4075
4076
4077 <item>
4078 <title>Simplifying and reducing clutter in my life and work</title>
4079 <link>https://mitjafelicijan.com/simplifying-and-reducing-clutter.html</link>
4080 <pubDate>Mon, 14 Oct 2019 12:00:00 &#43;0200</pubDate>
4081 <guid>https://mitjafelicijan.com/simplifying-and-reducing-clutter.html</guid>
4082 <description>I recently moved my main working machine back from Hachintosh to Linux.</description>
4083 <content:encoded>&lt;p&gt;I recently moved my main working machine back from Hachintosh to Linux. Well the
4084experiment was interesting and I have done some great work on macOS but it was
4085time to move back.&lt;/p&gt;
4086&lt;p&gt;I actually really missed Linux. The simplicity of &lt;code&gt;apt-get&lt;/code&gt; or just the amount
4087of software that exists for Linux should be a no-brainer. I spent most of my
4088time on macOS finding solutions to make things work. Using
4089&lt;a href=&#34;https://brew.sh/&#34;&gt;Brew&lt;/a&gt; was just a horrible experience and far from package
4090managers of Linux. At least they managed to get that &lt;code&gt;sudo&lt;/code&gt; debacle sorted.&lt;/p&gt;
4091&lt;p&gt;Not all was bad. macOS in general was a perfectly good environment. Things like
4092Docker and tooling like this worked without any hiccups. My normal tools like
4093coding IDE worked flawlessly and the whole look and feel is just superb. I have
4094been using MacBook Air for couple of years so I was used to the system but never
4095as a daily driver.&lt;/p&gt;
4096&lt;p&gt;One of the things I did after I installed Linux back on my machine was cleaning
4097up my Dropbox folder. I have everything on Dropbox. Even projects folder. I
4098write code for living so my whole life revolves around couple of megs of code
4099(with assets). So it&#39;s not like I have huge files on my machine. I don&#39;t have
4100movies or music or pictures on my PC. All of that stuff is in cloud. I use
4101Google music and I have Netflix account which is more than enough for me.&lt;/p&gt;
4102&lt;p&gt;I also went and deleted some of the repositories on my Github account. I have
4103deleted more code than deployed. People find this strange but for me deleting
4104something feels so cathartic and also forces me to write better code next time
4105around when I am faced with similar problem. That was a huge relief if I am
4106being totally honest.&lt;/p&gt;
4107&lt;p&gt;Next step was to do something with my webpage. I have been using some scripts I
4108wrote a while ago to generate static pages from markdown source posts. I kept on
4109adding and adding stuff on top of it and it became a source of a
4110frustration. And this is just a simple blog and I was using gulp and npm.
4111Anyways after couple of hours of searching and testing static generators I found
4112an interesting one
4113&lt;a href=&#34;https://github.com/piranha/gostatic&#34;&gt;https://github.com/piranha/gostatic&lt;/a&gt; and I
4114just decided to use this one. It was the only one that had a simple templating
4115engine, not that I really need one. But others had this convoluted way of trying
4116to solve everything and at the end just required quite bigger learning curve I
4117was ready to go with. So I deleted couple of old posts, simplified HTML, trashed
4118most of the CSS and went with
4119&lt;a href=&#34;https://motherfuckingwebsite.com/&#34;&gt;https://motherfuckingwebsite.com/&lt;/a&gt;
4120aesthetics. Yeah, the previous site was more visually stimulating but all I
4121really care is the content at this point. And Times New Roman font is kind of
4122awesome.&lt;/p&gt;
4123&lt;p&gt;I stopped working on most of the projects in the past couple of months because
4124the overhead was just too insane. There comes a point when you stretch yourself
4125too much and then you stop progressing and with that comes dissatisfaction.&lt;/p&gt;
4126&lt;p&gt;So that&#39;s about it. Moving forward minimal style.&lt;/p&gt;
4127</content:encoded>
4128 </item>
4129
4130
4131
4132 <item>
4133 <title>Encoding binary data into DNA sequence</title>
4134 <link>https://mitjafelicijan.com/encoding-binary-data-into-dna-sequence.html</link>
4135 <pubDate>Thu, 03 Jan 2019 12:00:00 &#43;0200</pubDate>
4136 <guid>https://mitjafelicijan.com/encoding-binary-data-into-dna-sequence.html</guid>
4137 <description>Initial thoughtsImagine a world where you could go outside and take a leaf from a tree and putit through your personal DNA sequencer and get data like music, videos orcomputer programs from it.</description>
4138 <content:encoded>&lt;h2 id=&#34;initial-thoughts&#34;&gt;Initial thoughts&lt;/h2&gt;
4139&lt;p&gt;Imagine a world where you could go outside and take a leaf from a tree and put
4140it through your personal DNA sequencer and get data like music, videos or
4141computer programs from it. Well, this is all possible now. It was not done on a
4142large scale because it is quite expensive to create DNA strands but it&#39;s
4143possible.&lt;/p&gt;
4144&lt;p&gt;Encoding data into DNA sequence is relatively simple process once you understand
4145the relationship between binary data and nucleotides and scientists have been
4146making large leaps in this field in order to provide viable long-term storage
4147solution for our data that would potentially survive our specie if case of
4148global disaster. We could imprint all the world&#39;s knowledge into plants and
4149ensure the survival of our knowledge.&lt;/p&gt;
4150&lt;p&gt;More optimistic usage for this technology would be easier storage of ever
4151growing data we produce every day. Once machines for sequencing DNA become fast
4152enough and cheaper this could mean the next evolution of storing data and
4153abandoning classical hard and solid state drives in data warehouses.&lt;/p&gt;
4154&lt;p&gt;As we currently stand this is still not viable but it is quite an amazing and
4155cool technology.&lt;/p&gt;
4156&lt;p&gt;My interests in this field are purely in encoding processes and experimental
4157testing mainly because I don&#39;t have the access to this expensive machines. My
4158initial goal was to create a toolkit that can be used by everybody to encode
4159their data into a proper DNA sequence.&lt;/p&gt;
4160&lt;h2 id=&#34;glossary&#34;&gt;Glossary&lt;/h2&gt;
4161&lt;p&gt;&lt;strong&gt;deoxyribose&lt;/strong&gt; A five-carbon sugar molecule with a hydrogen atom rather than a
4162hydroxyl group in the 2′ position; the sugar component of DNA nucleotides.&lt;/p&gt;
4163&lt;p&gt;&lt;strong&gt;double helix&lt;/strong&gt; The molecular shape of DNA in which two strands of nucleotides
4164wind around each other in a spiral shape.&lt;/p&gt;
4165&lt;p&gt;&lt;strong&gt;nitrogenous base&lt;/strong&gt; A nitrogen-containing molecule that acts as a base; often
4166referring to one of the purine or pyrimidine components of nucleic acids.&lt;/p&gt;
4167&lt;p&gt;&lt;strong&gt;phosphate group&lt;/strong&gt; A molecular group consisting of a central phosphorus atom
4168bound to four oxygen atoms.&lt;/p&gt;
4169&lt;p&gt;&lt;strong&gt;RGB&lt;/strong&gt; The RGB color model is an additive color model in which red, green and
4170blue light are added together in various ways to reproduce a broad array of
4171colors.&lt;/p&gt;
4172&lt;p&gt;&lt;strong&gt;GCC&lt;/strong&gt; The GNU Compiler Collection is a compiler system produced by the GNU
4173Project supporting various programming languages.&lt;/p&gt;
4174&lt;h2 id=&#34;data-encoding&#34;&gt;Data encoding&lt;/h2&gt;
4175&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; Encoding involves the use of a code to change original data into a
4176form that can be used by an external process.&lt;/p&gt;
4177&lt;p&gt;Encoding is the process of converting data into a format required for a number
4178of information processing needs, including:&lt;/p&gt;
4179&lt;ul&gt;
4180&lt;li&gt;Program compiling and execution&lt;/li&gt;
4181&lt;li&gt;Data transmission, storage and compression/decompression&lt;/li&gt;
4182&lt;li&gt;Application data processing, such as file conversion&lt;/li&gt;
4183&lt;/ul&gt;
4184&lt;p&gt;Encoding can have two meanings:&lt;/p&gt;
4185&lt;ul&gt;
4186&lt;li&gt;In computer technology, encoding is the process of applying a specific code,
4187such as letters, symbols and numbers, to data for conversion into an
4188equivalent cipher.&lt;/li&gt;
4189&lt;li&gt;In electronics, encoding refers to analog to digital conversion.&lt;/li&gt;
4190&lt;/ul&gt;
4191&lt;h2 id=&#34;quick-history-of-dna&#34;&gt;Quick history of DNA&lt;/h2&gt;
4192&lt;ul&gt;
4193&lt;li&gt;&lt;strong&gt;1869&lt;/strong&gt; - Friedrich Miescher identifies &amp;quot;nuclein&amp;quot;.&lt;/li&gt;
4194&lt;li&gt;&lt;strong&gt;1900s&lt;/strong&gt; - The Eugenics Movement.&lt;/li&gt;
4195&lt;li&gt;&lt;strong&gt;1900&lt;/strong&gt; – Mendel&#39;s theories are rediscovered by researchers.&lt;/li&gt;
4196&lt;li&gt;&lt;strong&gt;1944&lt;/strong&gt; - Oswald Avery identifies DNA as the &#39;transforming principle&#39;.&lt;/li&gt;
4197&lt;li&gt;&lt;strong&gt;1952&lt;/strong&gt; - Rosalind Franklin photographs crystallized DNA fibres.&lt;/li&gt;
4198&lt;li&gt;&lt;strong&gt;1953&lt;/strong&gt; - James Watson and Francis Crick discover the double helix structure of DNA.&lt;/li&gt;
4199&lt;li&gt;&lt;strong&gt;1965&lt;/strong&gt; - Marshall Nirenberg is the first person to sequence the bases in each codon.&lt;/li&gt;
4200&lt;li&gt;&lt;strong&gt;1983&lt;/strong&gt; - Huntington&#39;s disease is the first mapped genetic disease.&lt;/li&gt;
4201&lt;li&gt;&lt;strong&gt;1990&lt;/strong&gt; - The Human Genome Project begins.&lt;/li&gt;
4202&lt;li&gt;&lt;strong&gt;1995&lt;/strong&gt; - Haemophilus Influenzae is the first bacterium genome sequenced.&lt;/li&gt;
4203&lt;li&gt;&lt;strong&gt;1996&lt;/strong&gt; - Dolly the sheep is cloned.&lt;/li&gt;
4204&lt;li&gt;&lt;strong&gt;1999&lt;/strong&gt; - First human chromosome is decoded.&lt;/li&gt;
4205&lt;li&gt;&lt;strong&gt;2000&lt;/strong&gt; – Genetic code of the fruit fly is decoded.&lt;/li&gt;
4206&lt;li&gt;&lt;strong&gt;2002&lt;/strong&gt; – Mouse is the first mammal to have its genome decoded.&lt;/li&gt;
4207&lt;li&gt;&lt;strong&gt;2003&lt;/strong&gt; – The Human Genome Project is completed.&lt;/li&gt;
4208&lt;li&gt;&lt;strong&gt;2013&lt;/strong&gt; – DNA Worldwide and Eurofins Forensic discover identical twins have differences in their genetic makeup.&lt;/li&gt;
4209&lt;/ul&gt;
4210&lt;h2 id=&#34;what-is-dna&#34;&gt;What is DNA?&lt;/h2&gt;
4211&lt;p&gt;Deoxyribonucleic acid, a self-replicating material which is &lt;strong&gt;present in nearly
4212all living organisms&lt;/strong&gt; as the main constituent of chromosomes. It is the
4213&lt;strong&gt;carrier of genetic information&lt;/strong&gt;.&lt;/p&gt;
4214&lt;blockquote&gt;
4215&lt;p&gt;The nitrogen in our DNA, the calcium in our teeth, the iron in our blood,
4216the carbon in our apple pies were made in the interiors of collapsing stars.
4217We are made of starstuff.
4218&lt;strong&gt;-- Carl Sagan, Cosmos&lt;/strong&gt;&lt;/p&gt;
4219&lt;/blockquote&gt;
4220&lt;p&gt;The nucleotide in DNA consists of a sugar (deoxyribose), one of four bases
4221(cytosine (C), thymine (T), adenine (A), guanine (G)), and a phosphate.
4222Cytosine and thymine are pyrimidine bases, while adenine and guanine are purine
4223bases. The sugar and the base together are called a nucleoside.&lt;/p&gt;
4224&lt;p&gt;&lt;img src=&#34;/assets/dna-sequence/dna-basics.jpg&#34; alt=&#34;DNA&#34; /&gt;&lt;/p&gt;
4225&lt;p&gt;&lt;em&gt;DNA (a) forms a double stranded helix, and (b) adenine pairs with thymine and
4226cytosine pairs with guanine. (credit a: modification of work by Jerome Walker,
4227Dennis Myts)&lt;/em&gt;&lt;/p&gt;
4228&lt;h2 id=&#34;encode-binary-data-into-dna-sequence&#34;&gt;Encode binary data into DNA sequence&lt;/h2&gt;
4229&lt;p&gt;As an input file you can use any file you want:&lt;/p&gt;
4230&lt;ul&gt;
4231&lt;li&gt;ASCII files,&lt;/li&gt;
4232&lt;li&gt;Compiled programs,&lt;/li&gt;
4233&lt;li&gt;Multimedia files (MP3, MP4, MVK, etc),&lt;/li&gt;
4234&lt;li&gt;Images,&lt;/li&gt;
4235&lt;li&gt;Database files,&lt;/li&gt;
4236&lt;li&gt;etc.&lt;/li&gt;
4237&lt;/ul&gt;
4238&lt;p&gt;Note: If you would copy all the bytes from RAM to file or pipe data to file you
4239could encode also this data as long as you provide file pointer to the encoder.&lt;/p&gt;
4240&lt;h3 id=&#34;basic-encoding&#34;&gt;Basic Encoding&lt;/h3&gt;
4241&lt;p&gt;As already mentioned, the Basic Encoding is based on a simple mapping. Since DNA
4242is composed of 4 nucleotides (Adenine, Cytosine, Guanine, Thymine; usually
4243referred using the first letter). Using this technique we can encode&lt;/p&gt;
4244&lt;p&gt;$$ log_2(4) = log_2(2^2) = 2 bits $$&lt;/p&gt;
4245&lt;p&gt;using a single nucleotide. In this way, we are able to use the 4 bases that
4246compose the DNA strand to encode each byte of data.&lt;/p&gt;
4247&lt;table&gt;
4248&lt;thead&gt;
4249&lt;tr&gt;
4250&lt;th&gt;Two bits&lt;/th&gt;
4251&lt;th&gt;Nucleotides&lt;/th&gt;
4252&lt;/tr&gt;
4253&lt;/thead&gt;
4254&lt;tbody&gt;
4255&lt;tr&gt;
4256&lt;td&gt;00&lt;/td&gt;
4257&lt;td&gt;&lt;strong&gt;A&lt;/strong&gt; (Adenine)&lt;/td&gt;
4258&lt;/tr&gt;
4259&lt;tr&gt;
4260&lt;td&gt;10&lt;/td&gt;
4261&lt;td&gt;&lt;strong&gt;G&lt;/strong&gt; (Guanine)&lt;/td&gt;
4262&lt;/tr&gt;
4263&lt;tr&gt;
4264&lt;td&gt;01&lt;/td&gt;
4265&lt;td&gt;&lt;strong&gt;C&lt;/strong&gt; (Cytosine)&lt;/td&gt;
4266&lt;/tr&gt;
4267&lt;tr&gt;
4268&lt;td&gt;11&lt;/td&gt;
4269&lt;td&gt;&lt;strong&gt;T&lt;/strong&gt; (Thymine)&lt;/td&gt;
4270&lt;/tr&gt;
4271&lt;/tbody&gt;
4272&lt;/table&gt;
4273&lt;p&gt;With this in mind we can simply encode any data by using two-bit to Nucleotides
4274conversion.&lt;/p&gt;
4275&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{ Algorithm 1: Naive byte array to DNA encode }
4276&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;procedure EncodeToDNASequence(f) string
4277&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;begin
4278&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; enc string
4279&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;not&lt;/span&gt; eof(f) do
4280&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; c byte := buffer[0] { Read 1 byte &lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; buffer }
4281&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bin integer := sprintf(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;08b&amp;#39;&lt;/span&gt;, c) { Convert to string binary }
4282&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; e &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; range[0, 2, 4, 6] do
4283&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; e[0] == 48 &lt;span style=&#34;color:#00f&#34;&gt;and&lt;/span&gt; e[1] == 48 then { 0x00 - A (Adenine) }
4284&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; enc &#43;= &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;A&amp;#39;&lt;/span&gt;
4285&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; e[0] == 48 &lt;span style=&#34;color:#00f&#34;&gt;and&lt;/span&gt; e[1] == 49 then { 0x01 - G (Guanine) }
4286&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; enc &#43;= &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;G&amp;#39;&lt;/span&gt;
4287&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; e[0] == 49 &lt;span style=&#34;color:#00f&#34;&gt;and&lt;/span&gt; e[1] == 48 then { 0x10 - C (Cytosine) }
4288&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; enc &#43;= &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;C&amp;#39;&lt;/span&gt;
4289&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; e[0] == 49 &lt;span style=&#34;color:#00f&#34;&gt;and&lt;/span&gt; e[1] == 49 then { 0x11 - T (Thymine) }
4290&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; enc &#43;= &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;T&amp;#39;&lt;/span&gt;
4291&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; enc { Return DNA sequence }
4292&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;end
4293&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Another encoding would be &lt;strong&gt;Goldman encoding&lt;/strong&gt;. Using this encoding helps with
4294Nonsense mutation (amino acids replaced by a stop codon) that occurs and is the
4295most problematic during translation because it leads to truncated amino acid
4296sequences, which in turn results in truncated proteins.&lt;/p&gt;
4297&lt;p&gt;&lt;a href=&#34;https://www.youtube.com/watch?v=a4PiGWNsIEU&#34;&gt;Where to store big data? In DNA: Nick Goldman at TEDxPrague&lt;/a&gt;&lt;/p&gt;
4298&lt;h3 id=&#34;fasta-file-format&#34;&gt;FASTA file format&lt;/h3&gt;
4299&lt;p&gt;In bioinformatics, FASTA format is a text-based format for representing either
4300nucleotide sequences or peptide sequences, in which nucleotides or amino acids
4301are represented using single-letter codes. The format also allows for sequence
4302names and comments to precede the sequences. The format originates from the
4303FASTA software package, but has now become a standard in the field of
4304bioinformatics.&lt;/p&gt;
4305&lt;p&gt;The first line in a FASTA file started either with a &amp;quot;&amp;gt;&amp;quot; (greater-than) symbol
4306or, less frequently, a &amp;quot;;&amp;quot; (semicolon) was taken as a comment. Subsequent lines
4307starting with a semicolon would be ignored by software. Since the only comment
4308used was the first, it quickly became used to hold a summary description of the
4309sequence, often starting with a unique library accession number, and with time
4310it has become commonplace to always use &amp;quot;&amp;gt;&amp;quot; for the first line and to not use
4311&amp;quot;;&amp;quot; comments (which would otherwise be ignored).&lt;/p&gt;
4312&lt;pre&gt;&lt;code&gt;;LCBO - Prolactin precursor - Bovine
4313; a sample sequence in FASTA format
4314MDSKGSSQKGSRLLLLLVVSNLLLCQGVVSTPVCPNGPGNCQVSLRDLFDRAVMVSHYIHDLSS
4315EMFNEFDKRYAQGKGFITMALNSCHTSSLPTPEDKEQAQQTHHEVLMSLILGLLRSWNDPLYHL
4316VTEVRGMKGAPDAILSRAIEIEEENKRLLEGMEMIFGQVIPGAKETEPYPVWSGLPSLQTKDED
4317ARYSAFYNLLHCLRRDSSKIDTYLKLLNCRIIYNNNC*
4318
4319&amp;gt;MCHU - Calmodulin - Human, rabbit, bovine, rat, and chicken
4320ADQLTEEQIAEFKEAFSLFDKDGDGTITTKELGTVMRSLGQNPTEAELQDMINEVDADGNGTID
4321FPEFLTMMARKMKDTDSEEEIREAFRVFDKDGNGYISAAELRHVMTNLGEKLTDEEVDEMIREA
4322DIDGDGQVNYEEFVQMMTAK*
4323
4324&amp;gt;gi|5524211|gb|AAD44166.1| cytochrome b [Elephas maximus maximus]
4325LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV
4326EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG
4327LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL
4328GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX
4329IENY
4330&lt;/code&gt;&lt;/pre&gt;
4331&lt;p&gt;FASTA format was extended by &lt;a href=&#34;https://en.wikipedia.org/wiki/FASTQ_format&#34;&gt;FASTQ&lt;/a&gt;
4332format from the &lt;a href=&#34;https://www.sanger.ac.uk/&#34;&gt;Sanger Centre&lt;/a&gt; in Cambridge.&lt;/p&gt;
4333&lt;h3 id=&#34;png-encoded-dna-sequence&#34;&gt;PNG encoded DNA sequence&lt;/h3&gt;
4334&lt;table&gt;
4335&lt;thead&gt;
4336&lt;tr&gt;
4337&lt;th&gt;Nucleotides&lt;/th&gt;
4338&lt;th&gt;RGB&lt;/th&gt;
4339&lt;th&gt;Color name&lt;/th&gt;
4340&lt;/tr&gt;
4341&lt;/thead&gt;
4342&lt;tbody&gt;
4343&lt;tr&gt;
4344&lt;td&gt;A ➞ Adenine&lt;/td&gt;
4345&lt;td&gt;(0,0,255)&lt;/td&gt;
4346&lt;td&gt;Blue&lt;/td&gt;
4347&lt;/tr&gt;
4348&lt;tr&gt;
4349&lt;td&gt;G ➞ Guanine&lt;/td&gt;
4350&lt;td&gt;(0,100,0)&lt;/td&gt;
4351&lt;td&gt;Green&lt;/td&gt;
4352&lt;/tr&gt;
4353&lt;tr&gt;
4354&lt;td&gt;C ➞ Cytosine&lt;/td&gt;
4355&lt;td&gt;(255,0,0)&lt;/td&gt;
4356&lt;td&gt;Red&lt;/td&gt;
4357&lt;/tr&gt;
4358&lt;tr&gt;
4359&lt;td&gt;T ➞ Thymine&lt;/td&gt;
4360&lt;td&gt;(255,255,0)&lt;/td&gt;
4361&lt;td&gt;Yellow&lt;/td&gt;
4362&lt;/tr&gt;
4363&lt;/tbody&gt;
4364&lt;/table&gt;
4365&lt;p&gt;With this in mind we can create a simple algorithm to create PNG representation
4366of a DNA sequence.&lt;/p&gt;
4367&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{ Algorithm 2: Naive DNA to PNG encode &lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; FASTA file }
4368&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;procedure EncodeDNASequenceToPNG(f)
4369&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;begin
4370&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i image
4371&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;not&lt;/span&gt; eof(f) do
4372&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; c char := buffer[0] { Read 1 char &lt;span style=&#34;color:#00f&#34;&gt;from&lt;/span&gt; buffer }
4373&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; case c of
4374&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;A&amp;#39;&lt;/span&gt;: color := RGB(0, 0, 255) { Blue }
4375&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;G&amp;#39;&lt;/span&gt;: color := RGB(0, 100, 0) { Green }
4376&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;C&amp;#39;&lt;/span&gt;: color := RGB(255, 0, 0) { Red }
4377&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;T&amp;#39;&lt;/span&gt;: color := RGB(255, 255, 0) { Yellow }
4378&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; drawRect(i, [x, y], color)
4379&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; save(i) { Save PNG image }
4380&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;end
4381&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;encoding-text-file-in-practice&#34;&gt;Encoding text file in practice&lt;/h2&gt;
4382&lt;p&gt;In this example we will take a simple text file as our input stream for
4383encoding. This file will have a quote from Niels Bohr and saved as txt file.&lt;/p&gt;
4384&lt;blockquote&gt;
4385&lt;p&gt;How wonderful that we have met with a paradox. Now we have some hope of
4386making progress.
4387― Niels Bohr&lt;/p&gt;
4388&lt;/blockquote&gt;
4389&lt;p&gt;First we encode text file into FASTA file.&lt;/p&gt;
4390&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;./dnae-encode -i quote.txt -o quote.fa
4391&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:38:29 Gathering input file stats
4392&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:38:29 Starting encoding ...
4393&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; 106 B / 106 B [==================================] 100.00% 0s
4394&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:38:29 Saving to FASTA file ...
4395&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:38:29 Output FASTA file length is 438 B
4396&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:38:29 Process took 987.263µs
4397&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:38:29 Done ...
4398&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Output of &lt;code&gt;quote.fa&lt;/code&gt; file contains the encoded DNA sequence in ASCII format.&lt;/p&gt;
4399&lt;pre&gt;&lt;code&gt;&amp;gt;SEQ1
4400GACAGCTTGTGTACAAGTGTGCTTGCTCGCGAGCGGGTACGCGCGTGGGCTAACAAGTGA
4401GCCAGCAGGTGAACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGCTGGCGGGTGA
4402ACAAGTGTGCCGGTGAGCCAACAAGCAGACAAGTAAGCAGGTACGCAGGCGAGCTTGTCA
4403ACTCACAAGATCGCTTGTGTACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGTAT
4404GCTTGCTGGCGGACAAGCCAGCTTGTAAGCGGACAAGCTTGCGCACAAGCTGGCAGGCCT
4405GCCGGCTCGCGTACAAATTCACAAGTAAGTACGCTTGCGTGTACGCGGGTATGTATACTC
4406AACCTCACCAAACGGGACAAGATCGCCGGCGGGCTAGTATACAAGAACGCTTGCCAGTAC
4407AACC
4408&lt;/code&gt;&lt;/pre&gt;
4409&lt;p&gt;Then we encode FASTA file from previous operation to encode this data into PNG.&lt;/p&gt;
4410&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;./dnae-png -i quote.fa -o quote.png
4411&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:40:09 Gathering input file stats ...
4412&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:40:09 Deconstructing FASTA file ...
4413&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:40:09 Compositing image file ...
4414&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; 424 / 424 [==================================] 100.00% 0s
4415&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:40:09 Saving output file ...
4416&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:40:09 Output image file length is 1.1 kB
4417&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:40:09 Process took 19.036117ms
4418&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;2019/01/10 00:40:09 Done ...
4419&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After encoding into PNG format this file looks like this.&lt;/p&gt;
4420&lt;p&gt;&lt;img src=&#34;/assets/dna-sequence/quote.png&#34; alt=&#34;Encoded Quote in PNG format&#34; /&gt;&lt;/p&gt;
4421&lt;p&gt;The larger the input stream is the larger the PNG file would be.&lt;/p&gt;
4422&lt;p&gt;Compiled basic Hello World C program with
4423&lt;a href=&#34;https://www.gnu.org/software/gcc/&#34;&gt;GCC&lt;/a&gt; would &lt;a href=&#34;/assets/dna-sequence/sample.png&#34;&gt;look
4424like&lt;/a&gt;.&lt;/p&gt;
4425&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// gcc -O3 -o sample sample.c
4426&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#include&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;stdio.h&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;
4427&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&lt;/span&gt;
4428&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;main() {
4429&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; printf(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hello, world!&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;);
4430&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; 0;
4431&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
4432&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;toolkit-for-encoding-data&#34;&gt;Toolkit for encoding data&lt;/h2&gt;
4433&lt;p&gt;I have created a toolkit with two main programs:&lt;/p&gt;
4434&lt;ul&gt;
4435&lt;li&gt;dnae-encode (encodes file into FASTA file)&lt;/li&gt;
4436&lt;li&gt;dnae-png (encodes FASTA file into PNG)&lt;/li&gt;
4437&lt;/ul&gt;
4438&lt;p&gt;Toolkit with full source code is available on
4439&lt;a href=&#34;https://github.com/mitjafelicijan/dna-encoding&#34;&gt;github.com/mitjafelicijan/dna-encoding&lt;/a&gt;.&lt;/p&gt;
4440&lt;h3 id=&#34;dnae-encode&#34;&gt;dnae-encode&lt;/h3&gt;
4441&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;gt; ./dnae-encode --help
4442&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;usage: dnae-encode --input=INPUT [&amp;lt;flags&amp;gt;]
4443&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4444&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;A command-line application that encodes file into DNA sequence.
4445&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4446&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Flags:
4447&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; --help Show context-sensitive help (also try --help-long and --help-man).
4448&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; -i, --input=INPUT Input file (ASCII or binary) which will be encoded into DNA sequence.
4449&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; -o, --output=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;out.fa&amp;#34;&lt;/span&gt; Output file which stores DNA sequence in FASTA format.
4450&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; -s, --sequence=SEQ1 The description line (defline) or header/identifier line, gives a name and/or a unique identifier &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; the sequence.
4451&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; -c, --columns=60 Row characters length (no more than 120 characters). Devices preallocate fixed line sizes in software.
4452&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; --version Show application version.
4453&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;dnae-png&#34;&gt;dnae-png&lt;/h3&gt;
4454&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;gt; ./dnae-png --help
4455&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;usage: dnae-png --input=INPUT [&amp;lt;flags&amp;gt;]
4456&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4457&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;A command-line application that encodes FASTA file into PNG image.
4458&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4459&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;Flags:
4460&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; --help Show context-sensitive help (also try --help-long and --help-man).
4461&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; -i, --input=INPUT Input FASTA file which will be encoded into PNG image.
4462&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; -o, --output=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;out.png&amp;#34;&lt;/span&gt; Output file in PNG format that represents DNA sequence in graphical way.
4463&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; -s, --size=10 Size of pairings of DNA bases on image in pixels (lower resolution lower file size).
4464&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; --version Show application version.
4465&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h2 id=&#34;benchmarks&#34;&gt;Benchmarks&lt;/h2&gt;
4466&lt;p&gt;First we generate some binary sample data with dd.&lt;/p&gt;
4467&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dd &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt;=&amp;lt;(openssl enc -aes-256-ctr -pass pass:&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;$(&lt;/span&gt;dd &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt;=/dev/urandom bs=128 count=1 2&amp;gt;/dev/null | base64&lt;span style=&#34;color:#00f&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt; -nosalt &amp;lt; /dev/zero) of=1KB.bin bs=1KB count=1 iflag=fullblock
4468&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Our freshly generated 1KB file looks something like this (its full of garbage
4469data as intended).&lt;/p&gt;
4470&lt;p&gt;&lt;img src=&#34;/assets/dna-sequence/sample-binary-file.png&#34; alt=&#34;Sample binary file 1KB&#34; /&gt;&lt;/p&gt;
4471&lt;p&gt;We create following binary files:&lt;/p&gt;
4472&lt;ul&gt;
4473&lt;li&gt;1KB.bin&lt;/li&gt;
4474&lt;li&gt;10KB.bin&lt;/li&gt;
4475&lt;li&gt;100KB.bin&lt;/li&gt;
4476&lt;li&gt;1MB.bin&lt;/li&gt;
4477&lt;li&gt;10MB.bin&lt;/li&gt;
4478&lt;li&gt;100MB.bin&lt;/li&gt;
4479&lt;/ul&gt;
4480&lt;p&gt;After this we create FASTA files for all the binary files by encoding them
4481into DNA sequence.&lt;/p&gt;
4482&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;./dnae-encode -i 100MB.bin -o 100MB.fa
4483&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Then we GZIP all the FASTA files to see how much the can be compressed.&lt;/p&gt;
4484&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;gzip -9 &amp;lt; 10MB.fa &amp;gt; 10MB.fa.gz
4485&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;a href=&#34;/dna-sequence/benchmarks.ods&#34;&gt;Download ODS file with benchmarks&lt;/a&gt;.&lt;/p&gt;
4486&lt;p&gt;&lt;img src=&#34;/assets/dna-sequence/chart-1.png&#34; alt=&#34;Sample binary file 1KB&#34; /&gt;&lt;/p&gt;
4487&lt;p&gt;&lt;img src=&#34;/assets/dna-sequence/chart-2.png&#34; alt=&#34;Sample binary file 1KB&#34; /&gt;&lt;/p&gt;
4488&lt;h2 id=&#34;references&#34;&gt;References&lt;/h2&gt;
4489&lt;ul&gt;
4490&lt;li&gt;&lt;a href=&#34;https://www.techopedia.com/definition/948/encoding&#34;&gt;https://www.techopedia.com/definition/948/encoding&lt;/a&gt;&lt;/li&gt;
4491&lt;li&gt;&lt;a href=&#34;https://www.dna-worldwide.com/resource/160/history-dna-timeline&#34;&gt;https://www.dna-worldwide.com/resource/160/history-dna-timeline&lt;/a&gt;&lt;/li&gt;
4492&lt;li&gt;&lt;a href=&#34;https://opentextbc.ca/biology/chapter/9-1-the-structure-of-dna/&#34;&gt;https://opentextbc.ca/biology/chapter/9-1-the-structure-of-dna/&lt;/a&gt;&lt;/li&gt;
4493&lt;li&gt;&lt;a href=&#34;https://arxiv.org/abs/1801.04774&#34;&gt;https://arxiv.org/abs/1801.04774&lt;/a&gt;&lt;/li&gt;
4494&lt;li&gt;&lt;a href=&#34;https://en.wikipedia.org/wiki/FASTA_format&#34;&gt;https://en.wikipedia.org/wiki/FASTA_format&lt;/a&gt;&lt;/li&gt;
4495&lt;/ul&gt;
4496</content:encoded>
4497 </item>
4498
4499
4500
4501 <item>
4502 <title>Using DigitalOcean Spaces Object Storage with FUSE</title>
4503 <link>https://mitjafelicijan.com/using-digitalocean-spaces-object-storage-with-fuse.html</link>
4504 <pubDate>Tue, 16 Jan 2018 12:00:00 &#43;0200</pubDate>
4505 <guid>https://mitjafelicijan.com/using-digitalocean-spaces-object-storage-with-fuse.html</guid>
4506 <description>Couple of months ago DigitalOcean introduced newproduct calledSpaces whichis Object Storage very similar to Amazon&amp;#39;s S3.</description>
4507 <content:encoded>&lt;p&gt;Couple of months ago &lt;a href=&#34;https://www.digitalocean.com&#34;&gt;DigitalOcean&lt;/a&gt; introduced new
4508product called
4509&lt;a href=&#34;https://blog.digitalocean.com/introducing-spaces-object-storage/&#34;&gt;Spaces&lt;/a&gt; which
4510is Object Storage very similar to Amazon&#39;s S3. This really peaked my interest,
4511because this was something I was missing and even the thought of going over the
4512internet for such functionality was in no interest to me. Also in fashion with
4513their previous pricing this also is very cheap and pricing page is a no-brainer
4514compared to AWS or GCE. &lt;a href=&#34;https://www.digitalocean.com/pricing/&#34;&gt;Prices are clearly and precisely defined and
4515outlined&lt;/a&gt;. You must love them for that
4516:)&lt;/p&gt;
4517&lt;h2 id=&#34;initial-requirements&#34;&gt;Initial requirements&lt;/h2&gt;
4518&lt;ul&gt;
4519&lt;li&gt;Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)&lt;/li&gt;
4520&lt;li&gt;Will the performance degrade over time and over different sizes of objects?
4521(tl;dr NO&amp;amp;YES)&lt;/li&gt;
4522&lt;li&gt;Can storage be mounted on multiple machines at the same time and be writable?
4523(tl;dr YES)&lt;/li&gt;
4524&lt;/ul&gt;
4525&lt;blockquote&gt;
4526&lt;p&gt;Let me be clear. This scripts I use are made just for benchmarking and are not
4527intended to be used in real-life situations. Besides that, I am looking into
4528using this approaches but adding caching service in front of it and then
4529dumping everything as an object to storage. This could potentially be some
4530interesting post of itself. But in case you would need real-time data without
4531eventual consistency please take this scripts as they are: not usable in such
4532situations.&lt;/p&gt;
4533&lt;/blockquote&gt;
4534&lt;h2 id=&#34;is-it-possible-to-use-them-as-a-mounted-drive-with-fuse&#34;&gt;Is it possible to use them as a mounted drive with FUSE?&lt;/h2&gt;
4535&lt;p&gt;Well, actually they can be used in such manor. Because they are similar to &lt;a href=&#34;https://aws.amazon.com/s3/&#34;&gt;AWS
4536S3&lt;/a&gt; many tools are available and you can find many
4537articles and &lt;a href=&#34;https://stackoverflow.com/search?q=s3&#43;fuse&#34;&gt;Stackoverflow items&lt;/a&gt;.&lt;/p&gt;
4538&lt;p&gt;To make this work you will need DigitalOcean account. If you don&#39;t have one you
4539will not be able to test this code. But if you have an account then you go and
4540&lt;a href=&#34;https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&amp;amp;region=ams3&amp;amp;distro=debian&amp;amp;distroImage=debian-9-x64&amp;amp;options=private_networking,install_agent&#34;&gt;create new
4541Droplet&lt;/a&gt;.
4542If you click on this link you will already have preselected Debian 9 with
4543smallest VM option.&lt;/p&gt;
4544&lt;ul&gt;
4545&lt;li&gt;Please be sure to add you SSH key, because we will login to this machine
4546remotely.&lt;/li&gt;
4547&lt;li&gt;If you change your region please remember which one you choose because we will
4548need this information when we try to mount space to our machine.&lt;/li&gt;
4549&lt;/ul&gt;
4550&lt;p&gt;Instuctions on how to use SSH keys and how to setup them are available in
4551article &lt;a href=&#34;https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets&#34;&gt;How To Use SSH Keys with DigitalOcean
4552Droplets&lt;/a&gt;.&lt;/p&gt;
4553&lt;p&gt;&lt;img src=&#34;/assets/do-fuse/fuse-droplets.png&#34; alt=&#34;DigitalOcean Droplets&#34; /&gt;&lt;/p&gt;
4554&lt;p&gt;After we created Droplet it&#39;s time to create new Space. This is done by clicking
4555on a button &lt;a href=&#34;https://cloud.digitalocean.com/spaces/new&#34;&gt;Create&lt;/a&gt; (right top
4556corner) and selecting Spaces. Choose pronounceable &lt;code&gt;Unique name&lt;/code&gt; because we
4557will use it in examples below. You can either choose Private or Public, it
4558doesn&#39;t matter in our case. And you can always change that in the future.&lt;/p&gt;
4559&lt;p&gt;When you have created new Space we should &lt;a href=&#34;https://cloud.digitalocean.com/settings/api/tokens&#34;&gt;generate Access
4560key&lt;/a&gt;. This link will guide
4561to the page when you can generate this key. After you create new one, please
4562save provided Key and Secret because Secret will not be shown again.&lt;/p&gt;
4563&lt;p&gt;&lt;img src=&#34;/assets/do-fuse/fuse-spaces.png&#34; alt=&#34;DigitalOcean Spaces&#34; /&gt;&lt;/p&gt;
4564&lt;p&gt;Now that we have new Space and Access key we should SSH into our machine.&lt;/p&gt;
4565&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# replace IP with the ip of your newly created droplet&lt;/span&gt;
4566&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;ssh root@IP
4567&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4568&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# this will install utilities for mounting storage objects as FUSE&lt;/span&gt;
4569&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;apt install s3fs
4570&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4571&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# we now need to provide credentials (access key we created earlier)&lt;/span&gt;
4572&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# replace KEY and SECRET with your own credentials but leave the colon between them&lt;/span&gt;
4573&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# we also need to set proper permissions&lt;/span&gt;
4574&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;echo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;KEY:SECRET&amp;#34;&lt;/span&gt; &amp;gt; .passwd-s3fs
4575&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;chmod 600 .passwd-s3fs
4576&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4577&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we mount space to our machine&lt;/span&gt;
4578&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# replace UNIQUE-NAME with the name you choose earlier&lt;/span&gt;
4579&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# if you choose different region for your space be careful about -ourl option (ams3)&lt;/span&gt;
4580&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp
4581&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4582&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we try to create a file&lt;/span&gt;
4583&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# once you mount it may take a couple of seconds to retrieve data&lt;/span&gt;
4584&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;echo &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Hello cruel world&amp;#34;&lt;/span&gt; &amp;gt; /mnt/hello.txt
4585&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;After all this you can return to your browser and go to &lt;a href=&#34;https://cloud.digitalocean.com/spaces&#34;&gt;DigitalOcean
4586Spaces&lt;/a&gt; and click on your created
4587space. If file hello.txt is present you have successfully mounted space to your
4588machine and wrote data to it.&lt;/p&gt;
4589&lt;p&gt;I choose the same region for my Droplet and my Space but you don&#39;t have to. You
4590can have different regions. What this actually does to performance I don&#39;t know.&lt;/p&gt;
4591&lt;p&gt;Additional information on FUSE:&lt;/p&gt;
4592&lt;ul&gt;
4593&lt;li&gt;&lt;a href=&#34;https://github.com/s3fs-fuse/s3fs-fuse&#34;&gt;Github project page for s3fs&lt;/a&gt;&lt;/li&gt;
4594&lt;li&gt;&lt;a href=&#34;https://en.wikipedia.org/wiki/Filesystem_in_Userspace&#34;&gt;FUSE - Filesystem in Userspace&lt;/a&gt;&lt;/li&gt;
4595&lt;/ul&gt;
4596&lt;h2 id=&#34;will-the-performance-degrade-over-time-and-over-different-sizes-of-objects&#34;&gt;Will the performance degrade over time and over different sizes of objects?&lt;/h2&gt;
4597&lt;p&gt;For this task I didn&#39;t want to just read and write text files or uploading
4598images. I actually wanted to figure out if using something like SQlite is viable
4599in this case.&lt;/p&gt;
4600&lt;h3 id=&#34;measurement-experiment-1-file-copy&#34;&gt;Measurement experiment 1: File copy&lt;/h3&gt;
4601&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# first we create some dummy files at different sizes&lt;/span&gt;
4602&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dd &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt;=/dev/zero of=10KB.dat bs=1024 count=10 &lt;span style=&#34;color:#008000&#34;&gt;#10KB&lt;/span&gt;
4603&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dd &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt;=/dev/zero of=100KB.dat bs=1024 count=100 &lt;span style=&#34;color:#008000&#34;&gt;#100KB&lt;/span&gt;
4604&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dd &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt;=/dev/zero of=1MB.dat bs=1024 count=1024 &lt;span style=&#34;color:#008000&#34;&gt;#1MB&lt;/span&gt;
4605&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dd &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt;=/dev/zero of=10MB.dat bs=1024 count=10240 &lt;span style=&#34;color:#008000&#34;&gt;#10MB&lt;/span&gt;
4606&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4607&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we set time command to only return real&lt;/span&gt;
4608&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;TIMEFORMAT=%R
4609&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4610&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now lets test it&lt;/span&gt;
4611&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;(time cp 10KB.dat /mnt/) |&amp;amp; tee -a 10KB.results.txt
4612&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4613&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# and now we automate&lt;/span&gt;
4614&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# this will perform the same operation 100 times&lt;/span&gt;
4615&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# this will output results into separated files based on objecty size&lt;/span&gt;
4616&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;n=0; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (( n&#43;&#43; &amp;lt; 100 )); &lt;span style=&#34;color:#00f&#34;&gt;do&lt;/span&gt; (time cp 10KB.dat /mnt/10KB.$n.dat) |&amp;amp; tee -a 10KB.results.txt; &lt;span style=&#34;color:#00f&#34;&gt;done&lt;/span&gt;
4617&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;n=0; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (( n&#43;&#43; &amp;lt; 100 )); &lt;span style=&#34;color:#00f&#34;&gt;do&lt;/span&gt; (time cp 100KB.dat /mnt/100KB.$n.dat) |&amp;amp; tee -a 100KB.results.txt; &lt;span style=&#34;color:#00f&#34;&gt;done&lt;/span&gt;
4618&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;n=0; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (( n&#43;&#43; &amp;lt; 100 )); &lt;span style=&#34;color:#00f&#34;&gt;do&lt;/span&gt; (time cp 1MB.dat /mnt/1MB.$n.dat) |&amp;amp; tee -a 1MB.results.txt; &lt;span style=&#34;color:#00f&#34;&gt;done&lt;/span&gt;
4619&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;n=0; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (( n&#43;&#43; &amp;lt; 100 )); &lt;span style=&#34;color:#00f&#34;&gt;do&lt;/span&gt; (time cp 10MB.dat /mnt/10MB.$n.dat) |&amp;amp; tee -a 10MB.results.txt; &lt;span style=&#34;color:#00f&#34;&gt;done&lt;/span&gt;
4620&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Files of size 100MB were not successfully transferred and ended up displaying
4621error (cp: failed to close &#39;/mnt/100MB.1.dat&#39;: Operation not permitted).&lt;/p&gt;
4622&lt;p&gt;As I suspected, object size is not really that important. Sadly I don&#39;t have the
4623time to test performance over periods of time. But if some of you would do it
4624please send me your data. I would be interested in seeing results.&lt;/p&gt;
4625&lt;p&gt;&lt;strong&gt;Here are plotted results&lt;/strong&gt;&lt;/p&gt;
4626&lt;p&gt;You can download &lt;a href=&#34;/assets/do-fuse/copy-benchmarks.tsv&#34;&gt;raw result here&lt;/a&gt;.
4627Measurements are in seconds.&lt;/p&gt;
4628&lt;script src=&#34;//cdn.plot.ly/plotly-latest.min.js&#34;&gt;&lt;/script&gt;
4629&lt;div id=&#34;copy-benchmarks&#34;&gt;&lt;/div&gt;
4630&lt;script&gt;
4631(function(){
4632 var request = new XMLHttpRequest();
4633 request.open(&#34;GET&#34;, &#34;/assets/do-fuse/copy-benchmarks.tsv&#34;, true);
4634 request.onload = function() {
4635 if (request.status &gt;= 200 &amp;&amp; request.status &lt; 400) {
4636 var payload = request.responseText.trim();
4637 var tsv = payload.split(&#34;\n&#34;);
4638 for (var i=0; i&lt;tsv.length; i&#43;&#43;) { tsv[i] = tsv[i].split(&#34;\t&#34;); }
4639 var traces = [];
4640 var headers = tsv[0];
4641 tsv.shift();
4642 Array.prototype.forEach.call(headers, function(el, idx) {
4643 var x = [];
4644 var y = [];
4645 for (var j=0; j&lt;tsv.length; j&#43;&#43;) {
4646 x.push(j);
4647 y.push(parseFloat(tsv[j][idx].replace(&#34;,&#34;, &#34;.&#34;)));
4648 }
4649 traces.push({ x: x, y: y, type: &#34;scatter&#34;, name: el, line: { width: 1, shape: &#34;spline&#34; } });
4650 });
4651 var copy = Plotly.newPlot(&#34;copy-benchmarks&#34;, traces, { legend: {&#34;orientation&#34;: &#34;h&#34;}, height: 400, margin: { l: 40, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: &#34;execution time in seconds&#34;, titlefont: { size: 12 } }, xaxis: { title: &#34;fn(i)&#34;, titlefont: { size: 12 } } });
4652 } else { }
4653 };
4654 request.onerror = function() { };
4655 request.send(null);
4656})();
4657&lt;/script&gt;
4658&lt;p&gt;As far as these tests show, performance is quite stable and can be predicted
4659which is fantastic. But this is a small test and spans only over couple of
4660hours. So you should not completely trust them.&lt;/p&gt;
4661&lt;h3 id=&#34;measurement-experiment-2-sqlite-performanse&#34;&gt;Measurement experiment 2: SQLite performanse&lt;/h3&gt;
4662&lt;p&gt;I was unable to use database file directly from mounted drive so this is a no-go
4663as I suspected. So I executed code below on a local disk just to get some
4664benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY,
4665FETCHALL, COMMIT for 1000 times to generate statistics. As you can see
4666performance of SQLite is quite amazing. You could then potentially just copy
4667file to mounted drive and be done with it.&lt;/p&gt;
4668&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; time
4669&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; sqlite3
4670&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; sys
4671&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4672&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; len(sys.argv) &amp;lt; 3:
4673&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT&amp;#34;&lt;/span&gt;)
4674&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; exit()
4675&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4676&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; data_iter(x):
4677&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; range(x):
4678&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;yield&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;m&amp;#34;&lt;/span&gt; &#43; str(i), &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;f&amp;#34;&lt;/span&gt; &#43; str(i*i)
4679&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4680&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;header_line = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%s&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%s&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%s&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%s&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%s&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt; % (&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;DROPTABLE&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;CREATETABLE&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;INSERTMANY&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;FETCHALL&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;COMMIT&amp;#34;&lt;/span&gt;)
4681&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;with&lt;/span&gt; open(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sqlite-benchmarks.tsv&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;w&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#00f&#34;&gt;as&lt;/span&gt; fp:
4682&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fp.write(header_line)
4683&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4684&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;start_time = time.time()
4685&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;conn = sqlite3.connect(sys.argv[1])
4686&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;c = conn.cursor()
4687&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;end_time = time.time()
4688&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;result_time = CONNECT = end_time - start_time
4689&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;CONNECT: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4690&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4691&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;start_time = time.time()
4692&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;c.execute(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PRAGMA journal_mode=WAL&amp;#34;&lt;/span&gt;)
4693&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;c.execute(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PRAGMA temp_store=MEMORY&amp;#34;&lt;/span&gt;)
4694&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;c.execute(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PRAGMA synchronous=OFF&amp;#34;&lt;/span&gt;)
4695&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;result_time = PRAGMA = end_time - start_time
4696&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;PRAGMA: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4697&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4698&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; range(int(sys.argv[3])):
4699&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;#&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%i&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt; % (i))
4700&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4701&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; start_time = time.time()
4702&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; c.execute(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;drop table if exists test&amp;#34;&lt;/span&gt;)
4703&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; end_time = time.time()
4704&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; result_time = DROPTABLE = end_time - start_time
4705&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;DROPTABLE: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4706&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4707&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; start_time = time.time()
4708&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; c.execute(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;create table if not exists test(a,b)&amp;#34;&lt;/span&gt;)
4709&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; end_time = time.time()
4710&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; result_time = CREATETABLE = end_time - start_time
4711&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;CREATETABLE: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4712&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4713&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; start_time = time.time()
4714&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; c.executemany(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;INSERT INTO test VALUES (?, ?)&amp;#34;&lt;/span&gt;, data_iter(int(sys.argv[2])))
4715&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; end_time = time.time()
4716&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; result_time = INSERTMANY = end_time - start_time
4717&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;INSERTMANY: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4718&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4719&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; start_time = time.time()
4720&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; c.execute(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;select count(*) from test&amp;#34;&lt;/span&gt;)
4721&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; res = c.fetchall()
4722&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; end_time = time.time()
4723&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; result_time = FETCHALL = end_time - start_time
4724&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;FETCHALL: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4725&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4726&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; start_time = time.time()
4727&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; conn.commit()
4728&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; end_time = time.time()
4729&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; result_time = COMMIT = end_time - start_time
4730&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;COMMIT: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4731&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4732&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print
4733&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; log_line = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%f&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%f&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%f&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%f&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\t&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%f&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt; % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT)
4734&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;with&lt;/span&gt; open(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sqlite-benchmarks.tsv&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;a&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#00f&#34;&gt;as&lt;/span&gt; fp:
4735&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fp.write(log_line)
4736&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4737&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;start_time = time.time()
4738&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;conn.close()
4739&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;end_time = time.time()
4740&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;result_time = CLOSE = end_time - start_time
4741&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;CLOSE: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%g&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; seconds&amp;#34;&lt;/span&gt; % (result_time))
4742&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;You can download &lt;a href=&#34;/assets/do-fuse/sqlite-benchmarks.tsv&#34;&gt;raw result here&lt;/a&gt;. And
4743again, these results are done on a local block storage and do not represent
4744capabilities of object storage. With my current approach and state of the test
4745code these can not be done. I would need to make Python code much more robust
4746and check locking etc.&lt;/p&gt;
4747&lt;div id=&#34;sqlite-benchmarks&#34;&gt;&lt;/div&gt;
4748&lt;script&gt;
4749(function(){
4750 var request = new XMLHttpRequest();
4751 request.open(&#34;GET&#34;, &#34;/assets/do-fuse/sqlite-benchmarks.tsv&#34;, true);
4752 request.onload = function() {
4753 if (request.status &gt;= 200 &amp;&amp; request.status &lt; 400) {
4754 var payload = request.responseText.trim();
4755 var tsv = payload.split(&#34;\n&#34;);
4756 for (var i=0; i&lt;tsv.length; i&#43;&#43;) { tsv[i] = tsv[i].split(&#34;\t&#34;); }
4757 var traces = [];
4758 var headers = tsv[0];
4759 tsv.shift();
4760 Array.prototype.forEach.call(headers, function(el, idx) {
4761 var x = [];
4762 var y = [];
4763 for (var j=0; j&lt;tsv.length; j&#43;&#43;) {
4764 x.push(j);
4765 y.push(parseFloat(tsv[j][idx].replace(&#34;,&#34;, &#34;.&#34;)));
4766 }
4767 traces.push({ x: x, y: y, type: &#34;scatter&#34;, name: el, line: { width: 1, shape: &#34;spline&#34; } });
4768 });
4769 var sqlite = Plotly.newPlot(&#34;sqlite-benchmarks&#34;, traces, { legend: {&#34;orientation&#34;: &#34;h&#34;}, height: 400, margin: { l: 50, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: &#34;execution time in seconds&#34;, titlefont: { size: 12 } } });
4770 } else { }
4771 };
4772 request.onerror = function() { };
4773 request.send(null);
4774})();
4775&lt;/script&gt;
4776&lt;h2 id=&#34;can-storage-be-mounted-on-multiple-machines-at-the-same-time-and-be-writable&#34;&gt;Can storage be mounted on multiple machines at the same time and be writable?&lt;/h2&gt;
4777&lt;p&gt;Well, this one didn&#39;t take long to test. And the answer is &lt;strong&gt;YES&lt;/strong&gt;. I mounted
4778space on both machines and measured same performance on both machines. But
4779because file is downloaded before write and then uploaded on complete there
4780could potentially be problems is another process is trying to access the same
4781file.&lt;/p&gt;
4782&lt;h2 id=&#34;observations-and-conslusion&#34;&gt;Observations and conslusion&lt;/h2&gt;
4783&lt;p&gt;Using Spaces in this way makes it easier to access and manage files. But besides
4784that you would need to write additional code to make this one play nice with you
4785applications.&lt;/p&gt;
4786&lt;p&gt;Nevertheless, this was extremely simple to setup and use and this is just
4787another excellent product in DigitalOcean product line. I found this exercise
4788very valuable and am thinking about implementing some sort of mechanism for
4789SQLite, so data can be stored on Spaces and accessed by many VM&#39;s. For a project
4790where data doesn&#39;t need to be accessible in real-time and can have couple of
4791minutes old data this would be very interesting. If any of you find this
4792proposal interesting please write in a comment box below or shoot me an email
4793and I will keep you posted.&lt;/p&gt;
4794</content:encoded>
4795 </item>
4796
4797
4798
4799 <item>
4800 <title>Simple IOT application supported by real-time monitoring and data history</title>
4801 <link>https://mitjafelicijan.com/simple-iot-application.html</link>
4802 <pubDate>Fri, 11 Aug 2017 12:00:00 &#43;0200</pubDate>
4803 <guid>https://mitjafelicijan.com/simple-iot-application.html</guid>
4804 <description>Initial thoughtsI have been developing these kind of application for the better part of my last5 years and people keep asking me how to approach developing such applicationand I will give a try explaining it here.</description>
4805 <content:encoded>&lt;h2 id=&#34;initial-thoughts&#34;&gt;Initial thoughts&lt;/h2&gt;
4806&lt;p&gt;I have been developing these kind of application for the better part of my last
48075 years and people keep asking me how to approach developing such application
4808and I will give a try explaining it here.&lt;/p&gt;
4809&lt;p&gt;IOT applications are really no different than any other kind of applications.
4810We have data that needs to be collected and visualized in some form of tables or
4811charts. The main difference here is that most of the times these data is
4812collected by some kind of device foreign to developer that mainly operates in
4813web domain. But fear not, it&#39;s not that different than writing some JavaScript.&lt;/p&gt;
4814&lt;p&gt;There are many devices able to transmit data via wireless or wired network by
4815default but for the sake of example we will be using commonly known Arduino with
4816wireless module already on the board → &lt;a href=&#34;https://store.arduino.cc/arduino-mkr1000&#34;&gt;Arduino
4817MKR1000&lt;/a&gt;.&lt;/p&gt;
4818&lt;p&gt;In order to make this little project as accessible to others as possible I will
4819try to make it as inexpensive as possible. And by this I mean that I will avoid
4820using hosted virtual servers and will be using my own laptop as a server. But
4821you must buy Arduino MKR1000 to follow steps below. But if you would want to
4822deploy this software I would suggest using
4823&lt;a href=&#34;https://www.digitalocean.com&#34;&gt;DigitalOcean&lt;/a&gt; → smallest VPS is only per month
4824making this one of the most affordable option out there. Please notice that this
4825software will not run on stock web hosting that only supports LAMP (Linux,
4826Apache, MySQL, and PHP).&lt;/p&gt;
4827&lt;p&gt;But before we begin please take notice that this is strictly experimental code
4828and not well optimized and there are much better ways in handling some aspects
4829of the application but that requires much deeper knowledge of technology that is
4830not needed for an example like this.&lt;/p&gt;
4831&lt;p&gt;&lt;strong&gt;Development steps&lt;/strong&gt;&lt;/p&gt;
4832&lt;ol&gt;
4833&lt;li&gt;Simple Python API that will receive and store incoming data.&lt;/li&gt;
4834&lt;li&gt;Prototype C&#43;&#43; code that will read &amp;quot;sensor data&amp;quot; and transmit it to API.&lt;/li&gt;
4835&lt;li&gt;Data visualization with charts → extends Python web application.&lt;/li&gt;
4836&lt;/ol&gt;
4837&lt;p&gt;Step 1. and 3. will share the same web application. One route will be dedicated
4838to API and another to serving HTML with chart.&lt;/p&gt;
4839&lt;p&gt;Schema below represents what we will try to achieve and how different parts
4840correlates to each other.&lt;/p&gt;
4841&lt;p&gt;&lt;img src=&#34;/assets/iot-application/simple-iot-application-overview.svg&#34; alt=&#34;Overview&#34; /&gt;&lt;/p&gt;
4842&lt;h2 id=&#34;simple-python-api&#34;&gt;Simple Python API&lt;/h2&gt;
4843&lt;p&gt;I have always been a fan of simplicity so we will be using &lt;a href=&#34;https://bottlepy.org/docs/dev/&#34;&gt;Bottle: Python Web
4844Framework&lt;/a&gt;. It is a single file web framework
4845that seriously simplifies working with routes, templating and has built-in web
4846server that satisfies our need in this case.&lt;/p&gt;
4847&lt;p&gt;First we need to install bottle package. This can be done by downloading
4848&lt;code&gt;bottle.py&lt;/code&gt; and placing it in the root of your application or by using pip
4849software &lt;code&gt;pip install bottle --user&lt;/code&gt;.&lt;/p&gt;
4850&lt;p&gt;If you are using Linux or MacOS then Python is already installed. If you will
4851try to test this on Windows please install &lt;a href=&#34;https://www.python.org/downloads/windows/&#34;&gt;Python for
4852Windows&lt;/a&gt;. There may be some problems
4853with path when you will try to launch &lt;code&gt;python webapp.py&lt;/code&gt; so please take care
4854of this before you continue.&lt;/p&gt;
4855&lt;h3 id=&#34;basic-web-application&#34;&gt;Basic web application&lt;/h3&gt;
4856&lt;p&gt;Most basic bottle application is quite simple. Paste code below in
4857&lt;code&gt;webapp.py&lt;/code&gt; file and save.&lt;/p&gt;
4858&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -*- coding: utf-8 -*-&lt;/span&gt;
4859&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4860&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; bottle
4861&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4862&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# initializing bottle app&lt;/span&gt;
4863&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app = bottle.Bottle()
4864&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4865&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# triggered when / is accessed from browser&lt;/span&gt;
4866&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# only accepts GET → no POST allowed&lt;/span&gt;
4867&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@app.route(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/&amp;#34;&lt;/span&gt;, method=[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;GET&amp;#34;&lt;/span&gt;])
4868&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; route_default():
4869&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;howdy from python&amp;#34;&lt;/span&gt;
4870&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4871&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# starting server on http://0.0.0.0:5000&lt;/span&gt;
4872&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; __name__ == &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;:
4873&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bottle.run(
4874&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; app = app,
4875&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; host = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;0.0.0.0&amp;#34;&lt;/span&gt;,
4876&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; port = 5000,
4877&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; debug = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
4878&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; reloader = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
4879&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; catchall = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
4880&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; )
4881&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;To run this simple application you should open command prompt or terminal on
4882your machine and go to the folder containing your file and type &lt;code&gt;python webapp.py&lt;/code&gt;. If everything goes ok then open your web browser and point it to
4883&lt;code&gt;http://0.0.0.0:5000&lt;/code&gt;.&lt;/p&gt;
4884&lt;p&gt;If you would like change the port of your application (like port 80) and not use
4885root to run your app this will present a problem. The TCP/IP port numbers below
48861024 are privileged ports → this is a security feature. So in order of
4887simplicity and security use a port number above 1024 like I have used port 5000.&lt;/p&gt;
4888&lt;p&gt;If this fails at any time please fix it before you continue, because nothing
4889below will work otherwise.&lt;/p&gt;
4890&lt;p&gt;We use 0.0.0.0 as default host so that this app is available over your local
4891network. If you find your local ip &lt;code&gt;ifconfig&lt;/code&gt; and try accessing this site
4892with your phone (if on same network/router as your machine) this should work as
4893well (example of such ip &lt;code&gt;http://192.168.1.15:5000&lt;/code&gt;). This is a must have
4894because Arduino will be accessing this application to send it&#39;s data.&lt;/p&gt;
4895&lt;h3 id=&#34;web-application-security&#34;&gt;Web application security&lt;/h3&gt;
4896&lt;p&gt;There is a lot to be said about security and is a topic of many books. Of course
4897all this can not be written here but to just establish some basic security → you
4898should always use SSL with your application. Some fantastic free certificates
4899are available by &lt;a href=&#34;https://letsencrypt.org&#34;&gt;Let&#39;s Encrypt - Free SSL/TLS
4900Certificates&lt;/a&gt;. With SSL certificate installed you
4901should then make use of HTTP headers and send your &amp;quot;API key&amp;quot; via a header. If
4902your key is send via header then this key is encrypted by SSL and send encrypted
4903over the network. Never send your api keys by GET parameter like
4904&lt;code&gt;http://example.com/?api_key=somekeyvalue&lt;/code&gt;. The problem that this kind of
4905sending presents is that this key is visible in logs and by network sniffers.&lt;/p&gt;
4906&lt;p&gt;There is a fantastic article describing some aspects about security: &lt;a href=&#34;https://www.keycdn.com/blog/web-application-security-best-practices/&#34;&gt;11 Web
4907Application Security Best
4908Practices&lt;/a&gt;. Please
4909check it out.&lt;/p&gt;
4910&lt;h3 id=&#34;simple-api-for-writing-data-points&#34;&gt;Simple API for writing data-points&lt;/h3&gt;
4911&lt;p&gt;We will now be using boilerplate code from example above and extend it to be
4912SQLite3 because it plays well with Python and can store quite large amount of
4913able to write data received by API to local storage. For example use I will use
4914data. I have been using it to collect gigabytes of data in a single database
4915without any corruption or problems → your experience may vary.&lt;/p&gt;
4916&lt;p&gt;To avoid learning SQLite I will be using &lt;a href=&#34;https://dataset.readthedocs.io/en/latest/index.html&#34;&gt;Dataset: databases for lazy
4917people&lt;/a&gt;. This package
4918abstracts SQL and simplifies writing and reading data from database. You should
4919install this package with pip software &lt;code&gt;pip install dataset --user&lt;/code&gt;.&lt;/p&gt;
4920&lt;p&gt;Because API will use POST method I will be testing if code works correctly by
4921using &lt;a href=&#34;https://chrome.google.com/webstore/detail/restlet-client-rest-api-t/aejoelaoggembcahagimdiliamlcdmfm&#34;&gt;Restlet Client for Google
4922Chrome&lt;/a&gt;.
4923This software also allows you to set headers → for basic security with API_KEY.&lt;/p&gt;
4924&lt;p&gt;To quickly generate passwords or API keys I usually use this nifty website
4925&lt;a href=&#34;https://randomkeygen.com/&#34;&gt;RandomKeygen&lt;/a&gt;.&lt;/p&gt;
4926&lt;p&gt;Copy and paste code below over your previous code in file &lt;code&gt;webapp.py&lt;/code&gt;.&lt;/p&gt;
4927&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -*- coding: utf-8 -*-&lt;/span&gt;
4928&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4929&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; time
4930&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; bottle
4931&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; random
4932&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; dataset
4933&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4934&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# initializing bottle app&lt;/span&gt;
4935&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app = bottle.Bottle()
4936&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4937&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# connects to sqlite database&lt;/span&gt;
4938&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# check_same_thread=False allows using it in multi-threaded mode&lt;/span&gt;
4939&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;dsn&amp;#34;&lt;/span&gt;] = dataset.connect(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sqlite:///data.db?check_same_thread=False&amp;#34;&lt;/span&gt;)
4940&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4941&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# api key that will be used in Arduino code&lt;/span&gt;
4942&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;api_key&amp;#34;&lt;/span&gt;] = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;JtF2aUE5SGHfVJBCG5SH&amp;#34;&lt;/span&gt;
4943&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4944&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# triggered when /api is accessed from browser&lt;/span&gt;
4945&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# only accepts POST → no GET allowed&lt;/span&gt;
4946&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@app.route(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/api&amp;#34;&lt;/span&gt;, method=[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;POST&amp;#34;&lt;/span&gt;])
4947&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; route_default():
4948&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; status = 400
4949&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ts = int(time.time()) &lt;span style=&#34;color:#008000&#34;&gt;# current timestamp&lt;/span&gt;
4950&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; value = bottle.request.body.read() &lt;span style=&#34;color:#008000&#34;&gt;# data from device&lt;/span&gt;
4951&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; api_key = bottle.request.get_header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Api_Key&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#008000&#34;&gt;# api key from header&lt;/span&gt;
4952&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4953&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# outputs to console received data for debug reason&lt;/span&gt;
4954&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; :: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;.format(value, api_key)
4955&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4956&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# if api_key is correct and value is present&lt;/span&gt;
4957&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# then writes attribute to point table&lt;/span&gt;
4958&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; api_key == app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;api_key&amp;#34;&lt;/span&gt;] &lt;span style=&#34;color:#00f&#34;&gt;and&lt;/span&gt; value:
4959&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;dsn&amp;#34;&lt;/span&gt;][&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;point&amp;#34;&lt;/span&gt;].insert(dict(ts=ts, value=value))
4960&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; status = 200
4961&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4962&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# we only need to return status&lt;/span&gt;
4963&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; bottle.HTTPResponse(status=status, body=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;#34;&lt;/span&gt;)
4964&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
4965&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# starting server on http://0.0.0.0:5000&lt;/span&gt;
4966&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; __name__ == &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;:
4967&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bottle.run(
4968&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; app = app,
4969&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; host = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;0.0.0.0&amp;#34;&lt;/span&gt;,
4970&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; port = 5000,
4971&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; debug = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
4972&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; reloader = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
4973&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; catchall = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
4974&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; )
4975&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;To run this simply go to folder containing python file and run &lt;code&gt;python webapp.py&lt;/code&gt; from terminal. If everything goes ok you should have simple API
4976available via POST method on /api route.&lt;/p&gt;
4977&lt;p&gt;After testing the service with Restlet Client you should be able to view your
4978data in a database file &lt;code&gt;data.db&lt;/code&gt;.&lt;/p&gt;
4979&lt;p&gt;&lt;img src=&#34;/assets/iot-application/iot-rest-example.png&#34; alt=&#34;REST settings example&#34; /&gt;&lt;/p&gt;
4980&lt;p&gt;You can also check the contents of new database file by using desktop client
4981for SQLite → &lt;a href=&#34;http://sqlitebrowser.org/&#34;&gt;DB Browser for SQLite&lt;/a&gt;.&lt;/p&gt;
4982&lt;p&gt;&lt;img src=&#34;/assets/iot-application/iot-sqlite-db.png&#34; alt=&#34;SQLite database example&#34; /&gt;&lt;/p&gt;
4983&lt;p&gt;Table structure is as simple as it can be. We have ts (timestamp) and value
4984(value from Arduino). As you can see timestamp is generated on API side. If you
4985would happen to have atomic clock on Arduino it would be then better to generate
4986and send timestamp with the value. This would be particularity useful if we
4987would be collecting sensor data at a higher frequency and then sending this data
4988in bulk to API.&lt;/p&gt;
4989&lt;p&gt;If you will deploy this app with uWSGI and multi-threaded, use DSN (Data Source
4990Name) url with &lt;code&gt;?check_same_thread=False&lt;/code&gt;.&lt;/p&gt;
4991&lt;p&gt;Ok, now that we have some sort of a working API with some basic security so
4992unwanted people can not post data to your database can we proceed further and
4993try to program Arduino to send data to API.&lt;/p&gt;
4994&lt;h2 id=&#34;sending-data-to-api-with-arduino-mkr1000&#34;&gt;Sending data to API with Arduino MKR1000&lt;/h2&gt;
4995&lt;p&gt;First of all you should have MKR1000 module and microUSB cable to proceed. If
4996you have ever done any work with Arduino you should know that you also need
4997&lt;a href=&#34;https://www.arduino.cc/en/Main/Software&#34;&gt;Arduino IDE&lt;/a&gt;. On provided link you
4998should be able to download and install IDE. Once that task is completed and you
4999have successfully run blink example you should proceed to the next step.&lt;/p&gt;
5000&lt;p&gt;In order to use wireless capabilities of MKR1000 you need to first install
5001&lt;a href=&#34;https://www.arduino.cc/en/Reference/WiFi101&#34;&gt;WiFi101 library&lt;/a&gt; in Arduino IDE.
5002Please check before you install, you may already have it installed.&lt;/p&gt;
5003&lt;p&gt;Code below is a working example that sends data to API. Before you try to test
5004your code make sure you have run Python web application. Then change settings
5005for wifi, api endpoint and api_key. If by some reason code bellow doesn&#39;t work
5006for you please leave a comment and I&#39;ll try to help.&lt;/p&gt;
5007&lt;p&gt;Once you have opened IDE and copied this code try to compile and upload it.
5008Then open &amp;quot;Serial monitor&amp;quot; to see if any output is presented by Arduino.&lt;/p&gt;
5009&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;#include&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;WiFi101.h&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;
5010&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&lt;/span&gt;
5011&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// wifi settings
5012&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; ssid[] = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;ssid-name&amp;#34;&lt;/span&gt;;
5013&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; pass[] = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;ssid-password&amp;#34;&lt;/span&gt;;
5014&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5015&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// api server enpoint
5016&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;char&lt;/span&gt; server[] = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;192.168.6.22&amp;#34;&lt;/span&gt;;
5017&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;int&lt;/span&gt; port = 5000;
5018&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5019&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// api key that must be the same as the one in Python code
5020&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;String api_key = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;JtF2aUE5SGHfVJBCG5SH&amp;#34;&lt;/span&gt;;
5021&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5022&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;// frequency data is sent in ms - every 5 seconds
5023&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;int&lt;/span&gt; timeout = 1000 * 5;
5024&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5025&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;int&lt;/span&gt; status = WL_IDLE_STATUS;
5026&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5027&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;void&lt;/span&gt; setup() {
5028&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5029&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// initialize serial and wait for port to open:
5030&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; Serial.begin(9600);
5031&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; delay(1000);
5032&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5033&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// check for the presence of the shield
5034&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (WiFi.status() == WL_NO_SHIELD) {
5035&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;WiFi shield not present&amp;#34;&lt;/span&gt;);
5036&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (true);
5037&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5038&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5039&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// attempt to connect to wifi network
5040&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;while&lt;/span&gt; (status != WL_CONNECTED) {
5041&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Attempting to connect to SSID: &amp;#34;&lt;/span&gt;);
5042&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.println(ssid);
5043&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; status = WiFi.begin(ssid, pass);
5044&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// wait 10 seconds for connection
5045&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; delay(10000);
5046&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5047&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5048&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// output wifi status to serial monitor
5049&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; Serial.print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;SSID: &amp;#34;&lt;/span&gt;);
5050&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.println(WiFi.SSID());
5051&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5052&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; IPAddress ip = WiFi.localIP();
5053&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;IP Address: &amp;#34;&lt;/span&gt;);
5054&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.println(ip);
5055&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5056&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#2b91af&#34;&gt;long&lt;/span&gt; rssi = WiFi.RSSI();
5057&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.print(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;signal strength (RSSI):&amp;#34;&lt;/span&gt;);
5058&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.print(rssi);
5059&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34; dBm&amp;#34;&lt;/span&gt;);
5060&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5061&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5062&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#2b91af&#34;&gt;void&lt;/span&gt; loop() {
5063&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; WiFiClient client;
5064&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5065&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; (client.connect(server, port)) {
5066&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5067&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// I use random number generator for this example
5068&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// but you can use analog or digital inputs from arduino
5069&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; String content = String(random(1000));
5070&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5071&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; client.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;POST /api HTTP/1.1&amp;#34;&lt;/span&gt;);
5072&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; client.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Connection: close&amp;#34;&lt;/span&gt;);
5073&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; client.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Api-Key: &amp;#34;&lt;/span&gt; &#43; api_key);
5074&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; client.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Content-Length: &amp;#34;&lt;/span&gt; &#43; String(content.length()));
5075&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; client.println();
5076&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; client.println(content);
5077&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5078&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; delay(100);
5079&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; client.stop();
5080&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Data sent successfully ...&amp;#34;&lt;/span&gt;);
5081&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5082&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; } &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt; {
5083&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; Serial.println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Problem sending data ...&amp;#34;&lt;/span&gt;);
5084&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5085&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5086&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// waits for x seconds and continue looping
5087&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; delay(timeout);
5088&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5089&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;As seen from example you can notice that Arduino is generating random integer
5090between [ 0 .. 1000 ]. You can easily replace this with a temperature sensor or
5091any other kind of sensor.&lt;/p&gt;
5092&lt;p&gt;Now that we have API under the hood and Arduino is sending demo data we can now
5093focus on data visualization.&lt;/p&gt;
5094&lt;h2 id=&#34;data-visualization&#34;&gt;Data visualization&lt;/h2&gt;
5095&lt;p&gt;Before we continue we should examine our project folder structure. Currently we
5096only have two files in our project:&lt;/p&gt;
5097&lt;p&gt;&lt;em&gt;simple-iot-app/&lt;/em&gt;&lt;/p&gt;
5098&lt;ul&gt;
5099&lt;li&gt;&lt;em&gt;webapp.py&lt;/em&gt;&lt;/li&gt;
5100&lt;li&gt;&lt;em&gt;data.db&lt;/em&gt;&lt;/li&gt;
5101&lt;/ul&gt;
5102&lt;p&gt;We will now add HTML template that will contain CSS and JavaScript code inline
5103for the simplicity reason. And for the bottle framework to be able to scan root
5104application folder for templates we will add &lt;code&gt;bottle.TEMPLATE_PATH.insert(0, &amp;quot;./&amp;quot;)&lt;/code&gt; in &lt;code&gt;webapp.py&lt;/code&gt;. By default bottle framework uses &lt;code&gt;views/&lt;/code&gt;
5105subfolder to store templates. This is not the ideal situation and if you will
5106use bottle to develop web applications you should use native behavior and store
5107templates in it&#39;s predefined folder. But for the sake of example we will
5108over-ride this. Be careful to fully replace your code with new code that is
5109provided below. Avoid partially replacing code in file :) Also new code for
5110reading data-points is provided in Python example below.&lt;/p&gt;
5111&lt;p&gt;First we add new route to our web application. It should be trigger when browser
5112hits root of application &lt;code&gt;http://0.0.0.0:5000/&lt;/code&gt;. This route will do nothing
5113more than render &lt;code&gt;frontend.html&lt;/code&gt; template. This is done by &lt;code&gt;return bottle.template(&amp;quot;frontend.html&amp;quot;)&lt;/code&gt;. Check code below to further examine how
5114exactly this is done.&lt;/p&gt;
5115&lt;p&gt;Now we will expand &lt;code&gt;/api&lt;/code&gt; route and use different methods to write or read
5116data-points. For writing data-point we will use POST method and for reading
5117points we will use GET method. GET method will return JSON object with latest
5118readings and historical data.&lt;/p&gt;
5119&lt;p&gt;There is a fantastic JavaScript library for plotting time-series charts called
5120&lt;a href=&#34;https://www.metricsgraphicsjs.org&#34;&gt;MetricsGraphics.js&lt;/a&gt; that is based on
5121&lt;a href=&#34;https://d3js.org/&#34;&gt;D3.js&lt;/a&gt; library for visualizing data.&lt;/p&gt;
5122&lt;p&gt;Data schema required by MetricsGraphics.js → to achieve this we need to
5123transform data from database into this format:&lt;/p&gt;
5124&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[
5125&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
5126&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;date&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;2017-08-11 01:07:20&amp;#34;&lt;/span&gt;,
5127&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;value&amp;#34;: 933
5128&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
5129&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; {
5130&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;date&amp;#34;: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;2017-08-11 01:07:30&amp;#34;&lt;/span&gt;,
5131&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;value&amp;#34;: 743
5132&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5133&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]
5134&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Web application is now complete and we only need &lt;code&gt;frontend.html&lt;/code&gt; that we
5135will develop now. If you would try to start web app now and go to root app this
5136will return error because we don&#39;t have frontend.html yet.&lt;/p&gt;
5137&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -*- coding: utf-8 -*-&lt;/span&gt;
5138&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5139&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; time
5140&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; bottle
5141&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; json
5142&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; datetime
5143&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; random
5144&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; dataset
5145&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5146&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# initializing bottle app&lt;/span&gt;
5147&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app = bottle.Bottle()
5148&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5149&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# adds root directory as template folder&lt;/span&gt;
5150&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;bottle.TEMPLATE_PATH.insert(0, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;./&amp;#34;&lt;/span&gt;)
5151&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5152&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# connects to sqlite database&lt;/span&gt;
5153&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# check_same_thread=False allows using it in multi-threaded mode&lt;/span&gt;
5154&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;db&amp;#34;&lt;/span&gt;] = dataset.connect(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sqlite:///data.db?check_same_thread=False&amp;#34;&lt;/span&gt;)
5155&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5156&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# api key that will be used in Arduino code&lt;/span&gt;
5157&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;api_key&amp;#34;&lt;/span&gt;] = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;JtF2aUE5SGHfVJBCG5SH&amp;#34;&lt;/span&gt;
5158&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5159&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# triggered when / is accessed from browser&lt;/span&gt;
5160&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# only accepts GET → no POST allowed&lt;/span&gt;
5161&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@app.route(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/&amp;#34;&lt;/span&gt;, method=[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;GET&amp;#34;&lt;/span&gt;])
5162&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; route_default():
5163&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; bottle.template(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;frontend.html&amp;#34;&lt;/span&gt;)
5164&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5165&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# triggered when /api is accessed from browser&lt;/span&gt;
5166&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# accepts POST and GET&lt;/span&gt;
5167&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@app.route(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/api&amp;#34;&lt;/span&gt;, method=[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;GET&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;POST&amp;#34;&lt;/span&gt;])
5168&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; route_default():
5169&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5170&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# if method is POST then we write datapoint&lt;/span&gt;
5171&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; bottle.request.method == &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;POST&amp;#34;&lt;/span&gt;:
5172&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; status = 400
5173&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; ts = int(time.time()) &lt;span style=&#34;color:#008000&#34;&gt;# current timestamp&lt;/span&gt;
5174&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; value = bottle.request.body.read() &lt;span style=&#34;color:#008000&#34;&gt;# data from device&lt;/span&gt;
5175&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; api_key = bottle.request.get_header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Api-Key&amp;#34;&lt;/span&gt;) &lt;span style=&#34;color:#008000&#34;&gt;# api key from header&lt;/span&gt;
5176&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5177&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# outputs to console recieved data for debug reason&lt;/span&gt;
5178&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; print &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;gt;&amp;gt;&amp;gt; &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; :: &lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;{}&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&lt;/span&gt;.format(value, api_key)
5179&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5180&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# if api_key is correct and value is present&lt;/span&gt;
5181&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# then writes attribute to point table&lt;/span&gt;
5182&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; api_key == app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;api_key&amp;#34;&lt;/span&gt;] &lt;span style=&#34;color:#00f&#34;&gt;and&lt;/span&gt; value:
5183&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;db&amp;#34;&lt;/span&gt;][&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;point&amp;#34;&lt;/span&gt;].insert(dict(ts=ts, value=value))
5184&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; status = 200
5185&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5186&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# we only need to return status&lt;/span&gt;
5187&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; bottle.HTTPResponse(status=status, body=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;&amp;#34;&lt;/span&gt;)
5188&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5189&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;# if method is GET then we read datapoint&lt;/span&gt;
5190&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;else&lt;/span&gt;:
5191&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; response = []
5192&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; datapoints = app.config[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;db&amp;#34;&lt;/span&gt;][&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;point&amp;#34;&lt;/span&gt;].all()
5193&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5194&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; point &lt;span style=&#34;color:#00f&#34;&gt;in&lt;/span&gt; datapoints:
5195&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; response.append({
5196&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;date&amp;#34;&lt;/span&gt;: datetime.datetime.fromtimestamp(int(point[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;ts&amp;#34;&lt;/span&gt;])).strftime(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;%Y-%m-&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt;%d&lt;/span&gt;&lt;span style=&#34;color:#a31515&#34;&gt; %H:%M:%S&amp;#34;&lt;/span&gt;),
5197&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;value&amp;#34;&lt;/span&gt;: point[&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;value&amp;#34;&lt;/span&gt;]
5198&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; })
5199&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5200&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bottle.response.content_type = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;application/json&amp;#34;&lt;/span&gt;
5201&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; json.dumps(response)
5202&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5203&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# starting server on http://0.0.0.0:5000&lt;/span&gt;
5204&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; __name__ == &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;__main__&amp;#34;&lt;/span&gt;:
5205&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bottle.run(
5206&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; app = app,
5207&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; host = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;0.0.0.0&amp;#34;&lt;/span&gt;,
5208&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; port = 5000,
5209&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; debug = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
5210&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; reloader = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
5211&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; catchall = &lt;span style=&#34;color:#00f&#34;&gt;True&lt;/span&gt;,
5212&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; )
5213&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;And now finally we can implement &lt;code&gt;frontend.html&lt;/code&gt;. Create file with this name
5214and copy code below. When you are done you can start web application. Steps for
5215this part are listed below the code.&lt;/p&gt;
5216&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;&amp;lt;!DOCTYPE html&amp;gt;&lt;/span&gt;
5217&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;html&amp;gt;
5218&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5219&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;head&amp;gt;
5220&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;meta charset=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;utf-8&amp;#34;&lt;/span&gt;&amp;gt;
5221&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;title&amp;gt;Simple IOT application&amp;lt;/title&amp;gt;
5222&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/head&amp;gt;
5223&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5224&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;body&amp;gt;
5225&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5226&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;h1&amp;gt;Simple IOT application&amp;lt;/h1&amp;gt;
5227&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5228&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;div class=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;chart-placeholder&amp;#34;&lt;/span&gt;&amp;gt;
5229&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;div id=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;chart&amp;#34;&lt;/span&gt;&amp;gt;&amp;lt;/div&amp;gt;
5230&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/div&amp;gt;
5231&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5232&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;&amp;lt;!-- application main script --&amp;gt;&lt;/span&gt;
5233&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;script src=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js&amp;#34;&lt;/span&gt;&amp;gt;&amp;lt;/script&amp;gt;
5234&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;script src=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;https://cdnjs.cloudflare.com/ajax/libs/d3/4.10.0/d3.min.js&amp;#34;&lt;/span&gt;&amp;gt;&amp;lt;/script&amp;gt;
5235&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;script src=&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;https://cdnjs.cloudflare.com/ajax/libs/metrics-graphics/2.11.0/metricsgraphics.min.js&amp;#34;&lt;/span&gt;&amp;gt;&amp;lt;/script&amp;gt;
5236&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;script&amp;gt;
5237&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt; fetch_and_render() {
5238&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; d3.json(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/api&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;(data) {
5239&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; data = MG.convert.date(data, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;date&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;%Y-%m-%d %H:%M:%S&amp;#34;&lt;/span&gt;);
5240&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; MG.data_graphic({
5241&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; data: data,
5242&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; chart_type: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;line&amp;#34;&lt;/span&gt;,
5243&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; full_width: &lt;span style=&#34;color:#00f&#34;&gt;true&lt;/span&gt;,
5244&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; height: 270,
5245&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; target: document.getElementById(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;chart&amp;#34;&lt;/span&gt;),
5246&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; x_accessor: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;date&amp;#34;&lt;/span&gt;,
5247&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; y_accessor: &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;value&amp;#34;&lt;/span&gt;
5248&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
5249&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; });
5250&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5251&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; window.onload = &lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;() {
5252&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// initial call for rendering
5253&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; fetch_and_render();
5254&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5255&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;// updates chart every 5 seconds
5256&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt; setInterval(&lt;span style=&#34;color:#00f&#34;&gt;function&lt;/span&gt;() {
5257&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fetch_and_render();
5258&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }, 5000);
5259&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5260&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/script&amp;gt;
5261&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5262&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;&amp;lt;!-- application styles --&amp;gt;&lt;/span&gt;
5263&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;style&amp;gt;
5264&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; body {
5265&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;font&lt;/span&gt;: 13&lt;span style=&#34;color:#2b91af&#34;&gt;px&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;sans-serif&lt;/span&gt;;
5266&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;padding&lt;/span&gt;: 20&lt;span style=&#34;color:#2b91af&#34;&gt;px&lt;/span&gt; 50&lt;span style=&#34;color:#2b91af&#34;&gt;px&lt;/span&gt;;
5267&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5268&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .&lt;span style=&#34;color:#2b91af&#34;&gt;chart-placeholder&lt;/span&gt; {
5269&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;border&lt;/span&gt;: 2&lt;span style=&#34;color:#2b91af&#34;&gt;px&lt;/span&gt; &lt;span style=&#34;color:#00f&#34;&gt;solid&lt;/span&gt; #ccc;
5270&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;width&lt;/span&gt;: 100&lt;span style=&#34;color:#2b91af&#34;&gt;%&lt;/span&gt;;
5271&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;user-select&lt;/span&gt;: &lt;span style=&#34;color:#00f&#34;&gt;none&lt;/span&gt;;
5272&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5273&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#008000&#34;&gt;/* chart styles */&lt;/span&gt;
5274&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .&lt;span style=&#34;color:#2b91af&#34;&gt;mg-line1-color&lt;/span&gt; {
5275&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; stroke: &lt;span style=&#34;color:#00f&#34;&gt;red&lt;/span&gt;;
5276&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; stroke-width: 2;
5277&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5278&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .&lt;span style=&#34;color:#2b91af&#34;&gt;mg-main-area&lt;/span&gt;, .&lt;span style=&#34;color:#2b91af&#34;&gt;mg-main-line&lt;/span&gt; {
5279&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fill: #fff;
5280&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5281&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; .&lt;span style=&#34;color:#2b91af&#34;&gt;mg-x-axis&lt;/span&gt; line, .&lt;span style=&#34;color:#2b91af&#34;&gt;mg-y-axis&lt;/span&gt; line {
5282&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; stroke: #b3b2b2;
5283&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; stroke-width: 1&lt;span style=&#34;color:#2b91af&#34;&gt;px&lt;/span&gt;;
5284&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5285&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/style&amp;gt;
5286&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5287&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;/body&amp;gt;
5288&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5289&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&amp;lt;/html&amp;gt;
5290&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Now the folder structure should look like:&lt;/p&gt;
5291&lt;p&gt;&lt;em&gt;simple-iot-app/&lt;/em&gt;&lt;/p&gt;
5292&lt;ul&gt;
5293&lt;li&gt;&lt;em&gt;webapp.py&lt;/em&gt;&lt;/li&gt;
5294&lt;li&gt;&lt;em&gt;data.db&lt;/em&gt;&lt;/li&gt;
5295&lt;li&gt;&lt;em&gt;frontend.html&lt;/em&gt;&lt;/li&gt;
5296&lt;/ul&gt;
5297&lt;p&gt;Ok, lets now start application and start feeding it data.&lt;/p&gt;
5298&lt;ol&gt;
5299&lt;li&gt;&lt;code&gt;python webapp.py&lt;/code&gt;&lt;/li&gt;
5300&lt;li&gt;connect Arduino MKR1000 to power source&lt;/li&gt;
5301&lt;li&gt;open browser and go to &lt;code&gt;http://0.0.0.0:5000&lt;/code&gt;&lt;/li&gt;
5302&lt;/ol&gt;
5303&lt;p&gt;If everything goes well you should be seeing new data-points rendered on chart
5304every 5 seconds.&lt;/p&gt;
5305&lt;p&gt;If you navigate to &lt;code&gt;http://0.0.0.0:5000&lt;/code&gt; you should see rendered chart as
5306shown on picture below.&lt;/p&gt;
5307&lt;p&gt;&lt;img src=&#34;/assets/iot-application/iot-app-output.png&#34; alt=&#34;Application output&#34; /&gt;&lt;/p&gt;
5308&lt;p&gt;Complete application with all the code is available for
5309&lt;a href=&#34;/assets/iot-application/simple-iot-application.zip&#34;&gt;download&lt;/a&gt;.&lt;/p&gt;
5310&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;
5311&lt;p&gt;I hope this clarifies some aspects of IOT application development. Of course
5312this is a minimal example and is far from what can be done in real life with
5313some further dive into other technologies.&lt;/p&gt;
5314&lt;p&gt;If you would like to continue exploring IOT world here are some interesting
5315resources for you to examine:&lt;/p&gt;
5316&lt;ul&gt;
5317&lt;li&gt;&lt;a href=&#34;https://www.allaboutcircuits.com/projects/reading-sensors-with-an-arduino/&#34;&gt;Reading Sensors with an Arduino&lt;/a&gt;&lt;/li&gt;
5318&lt;li&gt;&lt;a href=&#34;http://www.hivemq.com/blog/how-to-get-started-with-mqtt&#34;&gt;MQTT 101 – How to Get Started with the lightweight IoT Protocol&lt;/a&gt;&lt;/li&gt;
5319&lt;li&gt;&lt;a href=&#34;https://www.html5rocks.com/en/tutorials/eventsource/basics/&#34;&gt;Stream Updates with Server-Sent Events&lt;/a&gt;&lt;/li&gt;
5320&lt;li&gt;&lt;a href=&#34;http://www.tutorialspoint.com/internet_of_things/&#34;&gt;Internet of Things (IoT) Tutorials&lt;/a&gt;&lt;/li&gt;
5321&lt;/ul&gt;
5322&lt;p&gt;Any comment or additional ideas are welcomed in comments below.&lt;/p&gt;
5323</content:encoded>
5324 </item>
5325
5326
5327
5328 <item>
5329 <title>Profiling Python web applications with visual tools</title>
5330 <link>https://mitjafelicijan.com/profiling-python-web-applications-with-visual-tools.html</link>
5331 <pubDate>Fri, 21 Apr 2017 12:00:00 &#43;0200</pubDate>
5332 <guid>https://mitjafelicijan.com/profiling-python-web-applications-with-visual-tools.html</guid>
5333 <description>I have been profiling my software with KCachegrind for a long time now and I wasmissing this option when I am developing API&amp;#39;s or other web services.</description>
5334 <content:encoded>&lt;p&gt;I have been profiling my software with KCachegrind for a long time now and I was
5335missing this option when I am developing API&#39;s or other web services. I always
5336knew that this is possible but never really took the time and dive into it.&lt;/p&gt;
5337&lt;p&gt;Before we begin there are some requirements. We will need to:&lt;/p&gt;
5338&lt;ul&gt;
5339&lt;li&gt;implement &lt;a href=&#34;https://docs.python.org/2/library/profile.html#module-cProfile&#34;&gt;cProfile&lt;/a&gt; into our web app,&lt;/li&gt;
5340&lt;li&gt;convert output to &lt;a href=&#34;http://valgrind.org/docs/manual/cl-manual.html&#34;&gt;callgrind&lt;/a&gt; format with &lt;a href=&#34;https://pypi.python.org/pypi/pyprof2calltree/&#34;&gt;pyprof2calltree&lt;/a&gt;,&lt;/li&gt;
5341&lt;li&gt;visualize data with &lt;a href=&#34;http://kcachegrind.sourceforge.net/html/Home.html&#34;&gt;KCachegrind&lt;/a&gt; or &lt;a href=&#34;http://www.profilingviewer.com/&#34;&gt;Profiling Viewer&lt;/a&gt;.&lt;/li&gt;
5342&lt;/ul&gt;
5343&lt;p&gt;If you are using MacOS you should check out &lt;a href=&#34;http://www.profilingviewer.com/&#34;&gt;Profiling
5344Viewer&lt;/a&gt; or
5345&lt;a href=&#34;http://www.maccallgrind.com/&#34;&gt;MacCallGrind&lt;/a&gt;.&lt;/p&gt;
5346&lt;p&gt;&lt;img src=&#34;/assets/python-profiling/kcachegrind.png&#34; alt=&#34;KCachegrind&#34; /&gt;&lt;/p&gt;
5347&lt;p&gt;We will be dividing this post into two main categories:&lt;/p&gt;
5348&lt;ul&gt;
5349&lt;li&gt;writing simple web-service,&lt;/li&gt;
5350&lt;li&gt;visualize profile of this web-service.&lt;/li&gt;
5351&lt;/ul&gt;
5352&lt;h2 id=&#34;simple-web-service&#34;&gt;Simple web-service&lt;/h2&gt;
5353&lt;p&gt;Let&#39;s use virtualenv so we won&#39;t pollute our base system. If you don&#39;t have
5354virtualenv installed on your system you can install it with pip command.&lt;/p&gt;
5355&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# let&amp;#39;s install virtualenv globally&lt;/span&gt;
5356&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ sudo pip install virtualenv
5357&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5358&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# let&amp;#39;s also install pyprof2calltree globally&lt;/span&gt;
5359&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ sudo pip install pyprof2calltree
5360&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5361&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we create project&lt;/span&gt;
5362&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ mkdir demo-project
5363&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cd demo-project/
5364&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5365&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now let&amp;#39;s create folder where we will store profiles&lt;/span&gt;
5366&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ mkdir prof
5367&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5368&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we create empty virtualenv in venv/ folder&lt;/span&gt;
5369&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ virtualenv --no-site-packages venv
5370&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5371&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# we now need to activate virtualenv&lt;/span&gt;
5372&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ source venv/bin/activate
5373&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5374&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# you can check if virtualenv was correctly initialized by&lt;/span&gt;
5375&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# checking where your python interpreter is located&lt;/span&gt;
5376&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# if command bellow points to your created directory and not some&lt;/span&gt;
5377&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# system dir like /usr/bin/python then everything is fine&lt;/span&gt;
5378&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ which python
5379&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5380&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# we can check now if all is good ➜ if ok couple of&lt;/span&gt;
5381&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# lines will be displayed&lt;/span&gt;
5382&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ pip freeze
5383&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# appdirs==1.4.3&lt;/span&gt;
5384&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# packaging==16.8&lt;/span&gt;
5385&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# pyparsing==2.2.0&lt;/span&gt;
5386&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# six==1.10.0&lt;/span&gt;
5387&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5388&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we are ready to install bottlepy ➜ web micro-framework&lt;/span&gt;
5389&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ pip install bottle
5390&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5391&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# you can deactivate virtualenv but you will then go&lt;/span&gt;
5392&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# under system domain ➜ for now don&amp;#39;t deactivate&lt;/span&gt;
5393&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ deactivate
5394&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;We are now ready to write simple web service. Let&#39;s create file app.py and paste
5395code bellow in this newly created file.&lt;/p&gt;
5396&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# -*- coding: utf-8 -*-&lt;/span&gt;
5397&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5398&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; bottle
5399&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; random
5400&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; cProfile
5401&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5402&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;app = bottle.Bottle()
5403&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5404&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# this function is a decorator and encapsulates function&lt;/span&gt;
5405&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# and performs profiling and then saves it to subfolder&lt;/span&gt;
5406&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# prof/function-name.prof&lt;/span&gt;
5407&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# in our example only awesome_random_number function will&lt;/span&gt;
5408&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# be profiled because it has do_cprofile defined&lt;/span&gt;
5409&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; do_cprofile(func):
5410&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; profiled_func(*args, **kwargs):
5411&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; profile = cProfile.Profile()
5412&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;try&lt;/span&gt;:
5413&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; profile.enable()
5414&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; result = func(*args, **kwargs)
5415&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; profile.disable()
5416&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; result
5417&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;finally&lt;/span&gt;:
5418&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; profile.dump_stats(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;prof/&amp;#34;&lt;/span&gt; &#43; str(func.__name__) &#43; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;.prof&amp;#34;&lt;/span&gt;)
5419&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; profiled_func
5420&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5421&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5422&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# we use profiling over specific function with including&lt;/span&gt;
5423&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# @do_cprofile above function declaration&lt;/span&gt;
5424&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@app.route(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/&amp;#34;&lt;/span&gt;)
5425&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@do_cprofile
5426&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; awesome_random_number():
5427&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; awesome_random_number = random.randint(0, 100)
5428&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;awesome random number is &amp;#34;&lt;/span&gt; &#43; str(awesome_random_number)
5429&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5430&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;@app.route(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;/test&amp;#34;&lt;/span&gt;)
5431&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;def&lt;/span&gt; test():
5432&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;dummy test&amp;#34;&lt;/span&gt;
5433&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5434&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;if&lt;/span&gt; __name__ == &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#39;__main__&amp;#39;&lt;/span&gt;:
5435&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; bottle.run(
5436&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; app = app,
5437&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; host = &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;0.0.0.0&amp;#34;&lt;/span&gt;,
5438&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; port = 4000
5439&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; )
5440&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5441&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# run with &amp;#39;python app.py&amp;#39;&lt;/span&gt;
5442&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# open browser &amp;#39;http://0.0.0.0:4000&amp;#39;&lt;/span&gt;
5443&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;When browser hits awesome_random_number() function profile is created in prof/
5444subfolder.&lt;/p&gt;
5445&lt;h2 id=&#34;visualize-profile&#34;&gt;Visualize profile&lt;/h2&gt;
5446&lt;p&gt;Now let&#39;s create callgrind format from this cProfile output.&lt;/p&gt;
5447&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cd prof/
5448&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ pyprof2calltree -i awesome_random_number.prof
5449&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# this creates &amp;#39;awesome_random_number.prof.log&amp;#39; file in the same folder&lt;/span&gt;
5450&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This file can be opened with visualizing tools listed above. In this case we
5451will be using Profilling Viewer under MacOS. You can open image in new tab. As
5452you can see from this example there is hierarchy of execution order of your
5453code.&lt;/p&gt;
5454&lt;p&gt;&lt;img src=&#34;/assets/python-profiling/profiling-viewer.png&#34; alt=&#34;Profilling Viewer&#34; /&gt;&lt;/p&gt;
5455&lt;blockquote&gt;
5456&lt;p&gt;Make sure you convert output of the cProfile output every time you want to
5457refresh and take a look at your possible optimizations because cProfile updates
5458.prof file every time browser hits the function.&lt;/p&gt;
5459&lt;/blockquote&gt;
5460&lt;p&gt;This is just a simple example but when you are developing real-life applications
5461this can be very illuminating, especially to see which parts of your code are
5462bottlenecks and need to be optimized.&lt;/p&gt;
5463&lt;h2 id=&#34;update-2017-04-22&#34;&gt;Update 2017-04-22&lt;/h2&gt;
5464&lt;p&gt;Reddit user &lt;a href=&#34;https://www.reddit.com/user/mvt&#34;&gt;mvt&lt;/a&gt; also recommended this awesome
5465web based profile visualizer &lt;a href=&#34;https://jiffyclub.github.io/snakeviz/&#34;&gt;SnakeViz&lt;/a&gt;
5466that directly takes output from
5467&lt;a href=&#34;https://docs.python.org/2/library/profile.html#module-cProfile&#34;&gt;cProfile&lt;/a&gt;
5468module.&lt;/p&gt;
5469&lt;div class=&#34;reddit-embed&#34; data-embed-media=&#34;www.redditmedia.com&#34; data-embed-parent=&#34;false&#34; data-embed-live=&#34;false&#34; data-embed-uuid=&#34;583880c1-002e-41ed-a373-020a0ef2cff9&#34; data-embed-created=&#34;2017-04-22T19:46:54.810Z&#34;&gt;&lt;a href=&#34;https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/&#34;&gt;Comment&lt;/a&gt; from discussion &lt;a href=&#34;https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/&#34;&gt;Profiling Python web applications with visual tools&lt;/a&gt;.&lt;/div&gt;&lt;script async src=&#34;https://www.redditstatic.com/comment-embed.js&#34;&gt;&lt;/script&gt;
5470&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# let&amp;#39;s install it globally as well&lt;/span&gt;
5471&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ sudo pip install snakeviz
5472&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5473&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now let&amp;#39;s visualize&lt;/span&gt;
5474&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ cd prof/
5475&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ snakeviz awesome_random_number.prof
5476&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# this automatically opens browser window and&lt;/span&gt;
5477&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# shows visualized profile&lt;/span&gt;
5478&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;img src=&#34;/assets/python-profiling/snakeviz.png&#34; alt=&#34;SnakeViz&#34; /&gt;&lt;/p&gt;
5479&lt;p&gt;Reddit user &lt;a href=&#34;https://www.reddit.com/user/ccharles&#34;&gt;ccharles&lt;/a&gt; suggested a better
5480way for installing pip software by targeting user level instead of using sudo.&lt;/p&gt;
5481&lt;div class=&#34;reddit-embed&#34; data-embed-media=&#34;www.redditmedia.com&#34; data-embed-parent=&#34;false&#34; data-embed-live=&#34;false&#34; data-embed-uuid=&#34;f4f0459e-684d-441e-bebe-eb49b2f0a31d&#34; data-embed-created=&#34;2017-04-22T19:46:10.874Z&#34;&gt;&lt;a href=&#34;https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/&#34;&gt;Comment&lt;/a&gt; from discussion &lt;a href=&#34;https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/&#34;&gt;Profiling Python web applications with visual tools&lt;/a&gt;.&lt;/div&gt;&lt;script async src=&#34;https://www.redditstatic.com/comment-embed.js&#34;&gt;&lt;/script&gt;
5482&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we need to add this path to our $PATH variable&lt;/span&gt;
5483&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# we do this my adding this line at the end of your&lt;/span&gt;
5484&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# ~/.bashrc file&lt;/span&gt;
5485&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;PATH=$PATH:$HOME/.local/bin/
5486&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5487&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# in order to use this new configuration you can close&lt;/span&gt;
5488&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# and reopen terminal or reload .bashrc file&lt;/span&gt;
5489&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ source ~/.bashrc
5490&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5491&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now let&amp;#39;s test if new directory is present in $PATH&lt;/span&gt;
5492&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ echo $PATH
5493&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5494&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# now we can install on user level by adding --user&lt;/span&gt;
5495&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# without use of sudo&lt;/span&gt;
5496&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ pip install snakeviz --user
5497&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Or as suggested by &lt;a href=&#34;https://www.reddit.com/user/mvt&#34;&gt;mvt&lt;/a&gt; you can
5498use &lt;a href=&#34;https://github.com/mitsuhiko/pipsi&#34;&gt;pipsi&lt;/a&gt;.&lt;/p&gt;
5499</content:encoded>
5500 </item>
5501
5502
5503
5504 <item>
5505 <title>What I&#39;ve learned developing ad server</title>
5506 <link>https://mitjafelicijan.com/what-i-ve-learned-developing-ad-server.html</link>
5507 <pubDate>Mon, 17 Apr 2017 12:00:00 &#43;0200</pubDate>
5508 <guid>https://mitjafelicijan.com/what-i-ve-learned-developing-ad-server.html</guid>
5509 <description>For the past year and half I have been developing native advertising server thatcontextually matches ads and displays them in different template forms onvariety of websites.</description>
5510 <content:encoded>&lt;p&gt;For the past year and half I have been developing native advertising server that
5511contextually matches ads and displays them in different template forms on
5512variety of websites. This project grew from serving thousands of ads per day to
5513millions.&lt;/p&gt;
5514&lt;p&gt;The system is made from couple of core components:&lt;/p&gt;
5515&lt;ul&gt;
5516&lt;li&gt;API for serving ads,&lt;/li&gt;
5517&lt;li&gt;Utils - cronjobs and queue management tools,&lt;/li&gt;
5518&lt;li&gt;Dashboard UI.&lt;/li&gt;
5519&lt;/ul&gt;
5520&lt;p&gt;Initial release was using &lt;a href=&#34;https://www.mongodb.com/&#34;&gt;MongoDB&lt;/a&gt; for full-text
5521search but was later replaced by &lt;a href=&#34;https://www.elastic.co/&#34;&gt;Elasticsearch&lt;/a&gt; for
5522better CPU utilization and better search performance. This provided us with many
5523amazing functionalities of &lt;a href=&#34;https://www.elastic.co/&#34;&gt;Elasticsearch&lt;/a&gt;. You should
5524check it out if you do any search related operations.&lt;/p&gt;
5525&lt;p&gt;Because the premise of the server is to provide native ad experience, they are
5526rendered on the client side via simple templating engine. This ensures that ads
5527can be displayed number of different ways based on the visual style of the
5528page. And this makes JavaScript client library quite complex.&lt;/p&gt;
5529&lt;p&gt;So now that you know basic information about the product lets get into the
5530lessons we learned.&lt;/p&gt;
5531&lt;h2 id=&#34;aggregate-everything&#34;&gt;Aggregate everything&lt;/h2&gt;
5532&lt;p&gt;After beta version was released everything (impressions, clicks, etc) was
5533written in nanosecond resolution in the database. At that time we were using
5534&lt;a href=&#34;https://www.postgresql.org/&#34;&gt;PostgreSQL&lt;/a&gt; and database quickly grew way above
5535200GB in disk space. And that was problematic. Statistics took disturbingly long
5536time to aggregate. Also using indexes on stats table in database was no help
5537after we reached 500 million datapoints.&lt;/p&gt;
5538&lt;blockquote&gt;
5539&lt;p&gt;There is a marketing product information and there is real life experience.
5540And the tend to be quite the opposite.&lt;/p&gt;
5541&lt;/blockquote&gt;
5542&lt;p&gt;This was the reason that now everything is aggregated on daily basis and this
5543data is then fed to Elastic in form of daily summary. With this we achieved we
5544can now track many more dimensions such as zone, channel and platform
5545information. And with this information we can now adapt occurrences of ads on
5546specific places more precisely.&lt;/p&gt;
5547&lt;p&gt;We have also adapted &lt;a href=&#34;https://redis.io/&#34;&gt;Redis&lt;/a&gt; as a full-time citizen in our
5548stack. Because Redis also stores information on a local disk we have some sort
5549of backup if server would accidentally suffer some failure.&lt;/p&gt;
5550&lt;p&gt;All the real-time statistics for ad serving and redirecting is presented as
5551counters in Redis instance and daily extracted and pushed to Elastic.&lt;/p&gt;
5552&lt;h2 id=&#34;measure-everything&#34;&gt;Measure everything&lt;/h2&gt;
5553&lt;p&gt;The thing about software is that we really don&#39;t know how well it is performing
5554under load until such load is presented. When testing locally everything is fine
5555but when on production things tend to fall apart.&lt;/p&gt;
5556&lt;p&gt;As a solution for this we are measuring everything we can. Function execution
5557time (by encapsulating functions with timers), server performance (cpu, memory,
5558disk, etc), Nginx and &lt;a href=&#34;https://uwsgi-docs.readthedocs.io/&#34;&gt;uWSGI&lt;/a&gt; performance.
5559We sacrifice a bit of performance for the sake of this information. And we store
5560all this information for later analysis.&lt;/p&gt;
5561&lt;p&gt;&lt;strong&gt;Example of function execution time&lt;/strong&gt;&lt;/p&gt;
5562&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;{
5563&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;get_final_filtered_ads&amp;#34;: {
5564&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;counter&amp;#34;: 1931250,
5565&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;avg&amp;#34;: 0.0066143431,
5566&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;elapsed&amp;#34;: 12773.9500310003
5567&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
5568&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;store_keywords_statistics&amp;#34;: {
5569&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;counter&amp;#34;: 1931011,
5570&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;avg&amp;#34;: 0.0004605267,
5571&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;elapsed&amp;#34;: 889.2821669996
5572&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
5573&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;match_by_context&amp;#34;: {
5574&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;counter&amp;#34;: 1931011,
5575&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;avg&amp;#34;: 0.0055960716,
5576&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;elapsed&amp;#34;: 10806.0758889999
5577&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
5578&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;match_by_high_performance&amp;#34;: {
5579&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;counter&amp;#34;: 262,
5580&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;avg&amp;#34;: 0.0152770229,
5581&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;elapsed&amp;#34;: 4.00258
5582&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; },
5583&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;store_impression_stats&amp;#34;: {
5584&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;counter&amp;#34;: 1931250,
5585&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;avg&amp;#34;: 0.0006189991,
5586&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;#34;elapsed&amp;#34;: 1195.4419869999
5587&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5588&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5589&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;We have also started profiling with &lt;a href=&#34;https://pymotw.com/2/profile/&#34;&gt;cProfile&lt;/a&gt;
5590and then visualizing with &lt;a href=&#34;http://kcachegrind.sourceforge.net/&#34;&gt;KCachegrind&lt;/a&gt;.
5591This provides much more detailed look into code execution.&lt;/p&gt;
5592&lt;h2 id=&#34;cache-control-is-your-friend&#34;&gt;Cache control is your friend&lt;/h2&gt;
5593&lt;p&gt;Because we use Javascript library for rendering ads we rely on this script
5594extensively and when in need we need to be able to change behavior of the script
5595quickly.&lt;/p&gt;
5596&lt;p&gt;In our case we can not simply replace javascript url in html code. It usually
5597takes a day or two for the guys who maintain sites to change code or add
5598?ver=xxx attribute. And this makes rapid deployment and testing very difficult
5599and time consuming. There is a limitation of how much you can test locally.&lt;/p&gt;
5600&lt;p&gt;We are now in the process of integrating &lt;a href=&#34;https://www.google.com/analytics/tag-manager/&#34;&gt;Google Tag
5601Manager&lt;/a&gt; but couple of websites
5602are developed on ASP.net platform that have some problems with tag manager. With
5603a solution below we are certain that we are serving latest version of the
5604script.&lt;/p&gt;
5605&lt;p&gt;And it only takes one mistake and users have the script cached and in case of
5606caching it for 1 year you probably know where the problem is.&lt;/p&gt;
5607&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# nginx ➜ /etc/nginx/sites-available/default
5608&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;&lt;/span&gt;&lt;span style=&#34;color:#00f&#34;&gt;location&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;/static/&lt;/span&gt; {
5609&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;alias&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;/path-to-static-content/&lt;/span&gt;;
5610&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;autoindex&lt;/span&gt; off;
5611&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;charset&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;utf-8&lt;/span&gt;;
5612&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;gzip&lt;/span&gt; on;
5613&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;gzip_types&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;text/plain&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;application/javascript&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;application/x-javascript&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;text/javascript&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;text/xml&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;text/css&lt;/span&gt;;
5614&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;location&lt;/span&gt; ~&lt;span style=&#34;color:#a31515&#34;&gt;*&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;\.(ico|gif|jpeg|jpg|png|woff|ttf|otf|svg|woff2|eot)&lt;/span&gt;$ {
5615&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;expires&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;1y&lt;/span&gt;;
5616&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;add_header&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;Pragma&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;public&lt;/span&gt;;
5617&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;add_header&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;Cache-Control&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;public&amp;#34;&lt;/span&gt;;
5618&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5619&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;location&lt;/span&gt; ~&lt;span style=&#34;color:#a31515&#34;&gt;*&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;\.(css|js|txt)&lt;/span&gt;$ {
5620&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;expires&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;3600s&lt;/span&gt;;
5621&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;add_header&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;Pragma&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;public&lt;/span&gt;;
5622&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;add_header&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;Cache-Control&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;public,&lt;/span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;must-revalidate&amp;#34;&lt;/span&gt;;
5623&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5624&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5625&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Also be careful when redirecting to url in your python code. We noticed that if
5626we didn&#39;t precisely setup cache control and expire headers in response we didn&#39;t
5627get the request on the server and therefore couldn&#39;t measure clicks. So when
5628redirecting do as follows and there will be no problems.&lt;/p&gt;
5629&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# python ➜ bottlepy web micro-framework&lt;/span&gt;
5630&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;response = bottle.HTTPResponse(status=302)
5631&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;response.set_header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Cache-Control&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;no-store, no-cache, must-revalidate&amp;#34;&lt;/span&gt;)
5632&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;response.set_header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Expires&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Thu, 01 Jan 1970 00:00:00 GMT&amp;#34;&lt;/span&gt;)
5633&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;response.set_header(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;Location&amp;#34;&lt;/span&gt;, url)
5634&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;return&lt;/span&gt; response
5635&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;blockquote&gt;
5636&lt;p&gt;Cache control in browsers is quite aggressive and you need to be precise to
5637avoid future problems. We learned that lesson the hard way.&lt;/p&gt;
5638&lt;/blockquote&gt;
5639&lt;h2 id=&#34;learn-nginx&#34;&gt;Learn NGINX&lt;/h2&gt;
5640&lt;p&gt;When deciding on a web server we went with Nginx as a reverse proxy for our
5641applications. We adapted micro-service oriented architecture early in the
5642project to ensure when we scale we can easily add additional servers to our
5643cluster. And Nginx was crucial to perform load balancing and static content
5644delivery.&lt;/p&gt;
5645&lt;p&gt;At first our config file was quite simple and later grew larger. After patching
5646and adding new settings I sat down and learned more about the guts of Nginx.
5647This proved to be very useful and we were able to squeeze much more out of our
5648setup. So I advise you to take your time and read through the
5649&lt;a href=&#34;https://nginx.org/en/docs/&#34;&gt;documentation&lt;/a&gt;. This saved us a lot of headache.
5650Googling for solutions only goes so far.&lt;/p&gt;
5651&lt;h2 id=&#34;use-redismemcached&#34;&gt;Use Redis/Memcached&lt;/h2&gt;
5652&lt;p&gt;As explained above we are using caching basically for everything. It is the
5653corner stone of our services. At first we were very careful about the quantity
5654of things we stored in &lt;a href=&#34;https://redis.io/&#34;&gt;Redis&lt;/a&gt;. But we later found out that
5655the memory footprint is very low even when storing large amount of data in it.&lt;/p&gt;
5656&lt;p&gt;So we gradually increased our usage to caching whole HTML outputs of dashboard.
5657This improved our performance in order of magnitude. And by using native TTL
5658support this goes hand in hand with our needs.&lt;/p&gt;
5659&lt;p&gt;The reason why we choose &lt;a href=&#34;https://redis.io/&#34;&gt;Redis&lt;/a&gt; over
5660&lt;a href=&#34;https://memcached.org/&#34;&gt;Memcached&lt;/a&gt; was the nature of scalability of Redis out
5661of the box. But all this can be achieved with Memcached.&lt;/p&gt;
5662&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;
5663&lt;p&gt;There are a lot more details that could have been written and every single topic
5664in here deserves it&#39;s own post but you probably got the idea about the problems
5665we faced.&lt;/p&gt;
5666</content:encoded>
5667 </item>
5668
5669
5670
5671 <item>
5672 <title>Golang profiling simplified</title>
5673 <link>https://mitjafelicijan.com/golang-profiling-simplified.html</link>
5674 <pubDate>Tue, 07 Mar 2017 12:00:00 &#43;0200</pubDate>
5675 <guid>https://mitjafelicijan.com/golang-profiling-simplified.html</guid>
5676 <description>Many posts have been written regarding profiling in Golang and I haven’t foundproper tutorial regarding this.</description>
5677 <content:encoded>&lt;p&gt;Many posts have been written regarding profiling in Golang and I haven’t found
5678proper tutorial regarding this. Almost all of them are missing some part of
5679important information and it gets pretty frustrating when you have a deadline
5680and are not finding simple distilled solution.&lt;/p&gt;
5681&lt;p&gt;Nevertheless, after searching and experimenting I have found a solution that
5682works for me and probably should also for you.&lt;/p&gt;
5683&lt;h2 id=&#34;where-are-my-pprof-files&#34;&gt;Where are my pprof files?&lt;/h2&gt;
5684&lt;p&gt;By default pprof files are generated in /tmp/ folder. You can override folder
5685where this files are generated programmatically in your golang code as we will
5686see below in example.&lt;/p&gt;
5687&lt;h2 id=&#34;why-is-my-cpu-profile-empty&#34;&gt;Why is my CPU profile empty?&lt;/h2&gt;
5688&lt;p&gt;I have found out that sometimes CPU profile is empty because program was not
5689executing long enough. Programs, that execute too quickly don’t produce pprof
5690file in my cases. Well, file is generated but only contains 4KB of information.&lt;/p&gt;
5691&lt;h2 id=&#34;profiling&#34;&gt;Profiling&lt;/h2&gt;
5692&lt;p&gt;As you can see from examples we are executing dummy_benchmark functions to
5693ensure some sort of execution. Memory profiling can be done without such a
5694“complex” function. But CPU profiling needs it.&lt;/p&gt;
5695&lt;p&gt;Both memory and CPU profiling examples are almost the same. Only parameters in
5696main function when calling profile.Start are different. When we set
5697profile.ProfilePath(“.”) we tell profiler to store pprof files in the same
5698folder as our program.&lt;/p&gt;
5699&lt;h3 id=&#34;memory-profiling&#34;&gt;Memory profiling&lt;/h3&gt;
5700&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;package&lt;/span&gt; main
5701&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5702&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; (
5703&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;fmt&amp;#34;&lt;/span&gt;
5704&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;time&amp;#34;&lt;/span&gt;
5705&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;github.com/pkg/profile&amp;#34;&lt;/span&gt;
5706&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;)
5707&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5708&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;func&lt;/span&gt; dummy_benchmark() {
5709&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5710&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fmt.Println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;first set ...&amp;#34;&lt;/span&gt;)
5711&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i := 0; i &amp;lt; 918231333; i&#43;&#43; {
5712&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i *= 2
5713&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i /= 2
5714&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5715&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5716&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;-time.After(time.Second*3)
5717&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5718&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fmt.Println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sencond set ...&amp;#34;&lt;/span&gt;)
5719&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i := 0; i &amp;lt; 9182312232; i&#43;&#43; {
5720&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i *= 2
5721&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i /= 2
5722&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5723&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5724&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5725&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;func&lt;/span&gt; main() {
5726&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;defer&lt;/span&gt; profile.Start(profile.MemProfile, profile.ProfilePath(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;.&amp;#34;&lt;/span&gt;), profile.NoShutdownHook).Stop()
5727&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; dummy_benchmark()
5728&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5729&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;cpu-profiling&#34;&gt;CPU profiling&lt;/h3&gt;
5730&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;package&lt;/span&gt; main
5731&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5732&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;import&lt;/span&gt; (
5733&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;fmt&amp;#34;&lt;/span&gt;
5734&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;time&amp;#34;&lt;/span&gt;
5735&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;github.com/pkg/profile&amp;#34;&lt;/span&gt;
5736&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;)
5737&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5738&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;func&lt;/span&gt; dummy_benchmark() {
5739&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5740&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fmt.Println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;first set ...&amp;#34;&lt;/span&gt;)
5741&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i := 0; i &amp;lt; 918231333; i&#43;&#43; {
5742&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i *= 2
5743&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i /= 2
5744&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5745&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5746&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &amp;lt;-time.After(time.Second*3)
5747&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5748&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; fmt.Println(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;sencond set ...&amp;#34;&lt;/span&gt;)
5749&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;for&lt;/span&gt; i := 0; i &amp;lt; 9182312232; i&#43;&#43; {
5750&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i *= 2
5751&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; i /= 2
5752&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; }
5753&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5754&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5755&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#00f&#34;&gt;func&lt;/span&gt; main() {
5756&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; &lt;span style=&#34;color:#00f&#34;&gt;defer&lt;/span&gt; profile.Start(profile.CPUProfile, profile.ProfilePath(&lt;span style=&#34;color:#a31515&#34;&gt;&amp;#34;.&amp;#34;&lt;/span&gt;), profile.NoShutdownHook).Stop()
5757&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt; dummy_benchmark()
5758&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}
5759&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;h3 id=&#34;generating-profiling-reports&#34;&gt;Generating profiling reports&lt;/h3&gt;
5760&lt;pre tabindex=&#34;0&#34; style=&#34;background-color:#fff;&#34;&gt;&lt;code&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# memory profiling&lt;/span&gt;
5761&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;go build mem.go
5762&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;./mem
5763&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;go tool pprof -pdf ./mem mem.pprof &amp;gt; mem.pdf
5764&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
5765&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#008000&#34;&gt;# cpu profiling&lt;/span&gt;
5766&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;go build cpu.go
5767&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;./cpu
5768&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;go tool pprof -pdf ./cpu cpu.pprof &amp;gt; cpu.pdf
5769&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;This will generate PDF document with visualized profile.&lt;/p&gt;
5770&lt;ul&gt;
5771&lt;li&gt;&lt;a href=&#34;/assets/go-profiling/golang-profiling-mem.pdf&#34;&gt;Memory PDF profile example&lt;/a&gt;&lt;/li&gt;
5772&lt;li&gt;&lt;a href=&#34;/assets/go-profiling/golang-profiling-cpu.pdf&#34;&gt;CPU PDF profile example&lt;/a&gt;&lt;/li&gt;
5773&lt;/ul&gt;
5774</content:encoded>
5775 </item>
5776
5777
5778
5779 <item>
5780 <title>Software development and my favorite pitfalls</title>
5781 <link>https://mitjafelicijan.com/software-development-pitfalls.html</link>
5782 <pubDate>Tue, 10 Nov 2015 12:00:00 &#43;0200</pubDate>
5783 <guid>https://mitjafelicijan.com/software-development-pitfalls.html</guid>
5784 <description>Over the years I had the privilege to work on some very excited projects both insoftware development field and also in electronics field and every experiencetaught me some invaluable lessons about how NOT TO approach development.</description>
5785 <content:encoded>&lt;p&gt;Over the years I had the privilege to work on some very excited projects both in
5786software development field and also in electronics field and every experience
5787taught me some invaluable lessons about how NOT TO approach development. And
5788through this post I will try to point out some absurd, outdated techniques I
5789find the most annoying and damaging during a development cycle. There will be
5790swearing because this topic really gets on my nerves and I never coherently
5791tried to explain them in writing. So if I get heated up, please bear with me.&lt;/p&gt;
5792&lt;p&gt;As new methods of project management are emerging, underlying processes still
5793stay old and outdated. This is mainly because we as people are unable to
5794completely shift away from these approaches.&lt;/p&gt;
5795&lt;p&gt;I was always struggling with communication, and many times that cost me a
5796relationship or two because I was not on the ball all the time. Through every
5797experience, I became more convinced that I am the problem and never ever doubted
5798that the problem may be that communication never evolved a single step from
5799emails. And if you think for a second, not many things have changed around this
5800topic. We just have different representations of email (message boards, chats,
5801project management tools). And I believe this is the real issue we are facing
5802now.&lt;/p&gt;
5803&lt;p&gt;There are many articles written about hyper connectivity and the effects that
5804are a direct result of it. But mainstream does nothing towards it. We are just
5805putting out fires, and we do nothing to prevent it. I am certain this will be a
5806major source of grief in coming years. And what we all can do to avoid this is
5807to change our mindset and experiment on our communication skills, development
5808approaches. We need to maximize possible output that a person can give. And to
5809achieve this we need to listen to them, encourage them. I know that not
5810everybody is a naturally born leader, but with enough practice and encouragement
5811they also can become active participants in leadership.&lt;/p&gt;
5812&lt;p&gt;There are many talks now about methodologies such as Scrum, Kanban, Cleanroom
5813and they all fucking piss me of :). These are all boxes that imprison people and
5814take away their freedom of thought. This is a straightforward mindfuck /
5815amputation of creativity.&lt;/p&gt;
5816&lt;p&gt;Let me list a couple of things that I find really destructive and bad for a
5817project and in a long run company.&lt;/p&gt;
5818&lt;h2 id=&#34;ping-emails&#34;&gt;Ping emails&lt;/h2&gt;
5819&lt;p&gt;Ping emails are emails you have to write as soon as you receive an email. Its
5820sole purpose is to inform the sender that you received their email, and you are
5821working on it. Its result is only to calm down the sender that their task is
5822being dealt with. It’s intent basically is, I did my job by sending you this
5823email, so I am on clear grounds. I categorize this email as fuck you email.
5824This is one of the most irritating types of emails I need to write. This is the
5825ultimate control freak show you can experience, and it gives the sender a false
5826feeling of control. Newsflash: We do not live in 1982 where there was a
5827possibility that email never reached the destination. I really hate this from
5828the bottom of my heart.&lt;/p&gt;
5829&lt;p&gt;They should be like: “Yes, I am fucking alive, and I am at your service my
5830leash!”. I guess if I would reply like this, I wouldn’t have to write any more
5831of this kind of messages.&lt;/p&gt;
5832&lt;h2 id=&#34;everybody-is-a-project-manager&#34;&gt;Everybody is a project manager&lt;/h2&gt;
5833&lt;p&gt;Well, this is a tough one. I noticed that as soon as you let people to give
5834their suggestions, you are basically screwed. There is a truth in the saying:
5835“Give low expectations and deliver little more than you promised.”.&lt;/p&gt;
5836&lt;p&gt;People tend to take a role of a manager as soon as they are presented with an
5837opportunity. And by getting angry at them, you only provoke yourself. They are
5838not at fault. You just need to tell them they are only giving suggestions and
5839not tasks at the beginning and everything will be alright. But if you give them
5840a feeling that they are in control, you will have immense problems explaining
5841why their features are not in current release.&lt;/p&gt;
5842&lt;p&gt;Project mission must be always leading project requirements and any deviation
5843from it will result in major project butchering. And by this, I mean that the
5844project will get its own path, and you will be left with half done software that
5845helps nobody. Clear mission goals and clean execution will allow you to develop
5846software will clear intent.&lt;/p&gt;
5847&lt;h2 id=&#34;we-are-never-wrong&#34;&gt;We are never wrong&lt;/h2&gt;
5848&lt;p&gt;I find this type of arrogance the worst. We must always conduct ourselves that
5849we are infallible and cannot make mistakes. As soon as a procedure or process is
5850established, there is no room for changes or improvements. This is the most
5851idiotic thing someone can say of think. I think that processes need to involve
5852and change over time. This is imperative and need to have in your organization
5853if you want to improve and develop company. We all need to grow balls and change
5854everything in order to adapt to current situations. Being a prisoner of
5855predefined processes kills creativity.&lt;/p&gt;
5856&lt;p&gt;I am constantly trying new software for project managing and communication. I
5857believe every team has its own dynamic, and it needs to be discovered
5858organically and naturally through many experiments. By putting the team in a
5859box, you are amputating their creativity and therefore minimizing their
5860potential. But if you talk to an executive, you will mainly find archetypical
5861thinking and a strong need to compartmentalize everything from business
5862processes to resource management. And this type of management that often
5863displays micromanagement techniques only works for short periods (couple of
5864years) and then employees either leave the company or become basically retarded
5865drones on autopilot.&lt;/p&gt;
5866&lt;h2 id=&#34;micromanaging&#34;&gt;Micromanaging&lt;/h2&gt;
5867&lt;p&gt;This basically implies that everybody on the team is an idiot who needs to have
5868a to-do list that they cannot write themselves. How about spoon-feeding the team
5869at launch because besides the team leader, everybody must be a retarded idiot at
5870best?&lt;/p&gt;
5871&lt;p&gt;I prefer milestones as they give developers much more freedom and creativity in
5872developing and not waste their time checking some bizarre to-do list that was
5873not even thought through. Projects constantly change throughout the development
5874cycle, and all you are left at the end is a list of unchecked tasks and the
5875wrath of management why they are not completed. Best WTF moment!&lt;/p&gt;
5876&lt;h2 id=&#34;human-contact--no-need-for-it&#34;&gt;Human contact — no need for it!&lt;/h2&gt;
5877&lt;p&gt;We are vigorously trying to eliminate physical contact by replacing short
5878meetings with software, with no regards that we are not machines. Many times a
5879simple 5-min meeting at morning can solve most of the problems. In rapid
5880development, short bursts of man to man communication is possibly the best way
5881to go.&lt;/p&gt;
5882&lt;p&gt;We now have all this software available, and all what we get out of it is a
5883giant clusterfuck. An obstacle and not a solution. So, why we still use them?&lt;/p&gt;
5884&lt;h2 id=&#34;mvp-is-killing-innovation&#34;&gt;MVP is killing innovation&lt;/h2&gt;
5885&lt;p&gt;Many will disagree with me on this one, but I stand strong by this statement.
5886What I noticed in my experience that all this buzz words around us only mislead
5887and capture us in a circle of solving issues that already have a solution, but
5888we are unable to see it without using some fancy word for it.&lt;/p&gt;
5889&lt;p&gt;The toughest thing to do for a developer is to minimize requirements. Well, this
5890is though only for bad developers. Yes, I said it. There are many types of
5891developers out there. And those unable to minimize feature scope are the ones
5892you don’t need on your team. Their only goal is to solve problems that exist
5893only in their heads. And then you have to argue with them, and waste energy on
5894them, instead of developing your awesome product. They are a cancer and I
5895suggest you cut them off.&lt;/p&gt;
5896&lt;p&gt;MVP as an idea is great, but sadly people don’t understand underlying
5897philosophy, and they spent too much time focusing and fixating on something that
5898every sane person with normal IQ will understand without some made up
5899acronym. And the result is a lot of talking and barely no execution.&lt;/p&gt;
5900&lt;p&gt;Well, MVP is not directly killing innovation, but stupid people do when they try
5901to understand it.&lt;/p&gt;
5902&lt;h2 id=&#34;pressure-wasteland&#34;&gt;Pressure wasteland&lt;/h2&gt;
5903&lt;p&gt;You must never allow to be pressured into confirming a deadline if you are not
5904confident. We often feel a need that we are in service of others, which is true
5905to some extent. But it is also true that others are in service to us to some
5906extent. And we forget this all the time. We are all pressured all the time to
5907make decisions just to calm other people down. And when they leave your office
5908you experience WTF moment :) How the hell did they manage to fuck me up again?&lt;/p&gt;
5909&lt;p&gt;People need to realize that the more pressure you put on somebody, the less they
5910will be able to do. So 5-min update email requests will only resolve in mental
5911breakdown and inability to work that day. Constant poking is probably the only
5912thing I lose my mind instantly. For all you that are doing this: “Stop bothering
5913us with your insecurities and let us do our job. We will do it quicker and
5914better without you breathing down our necks.”&lt;/p&gt;
5915&lt;p&gt;If this happens to me, I end up with no energy at the end. Don’t you get it?
5916You will get much more from and out of me if you ask me like a human person and
5917not your personal butler. On a long run, you are destroying your relationships
5918and nobody would want to work with you. Your schizophrenic approach will damage
5919only you in a long run. Nobody is anybody’s property.&lt;/p&gt;
5920&lt;h2 id=&#34;conclusion&#34;&gt;Conclusion&lt;/h2&gt;
5921&lt;p&gt;I am guilty of many things described in this post. And I find it hard sometimes
5922to acknowledge this. And I lie to myself and try vigorously to find some
5923explanation why I do these things. There is always space for growth. And maybe
5924you will also find some of yourself in this post and realize what needs to
5925change for you to evolve.&lt;/p&gt;
5926</content:encoded>
5927 </item>
5928
5929
5930
5931 <item>
5932 <title>Wireless sensor networks</title>
5933 <link>https://mitjafelicijan.com/wireless-sensor-networks.html</link>
5934 <pubDate>Thu, 24 Oct 2013 12:00:00 &#43;0200</pubDate>
5935 <guid>https://mitjafelicijan.com/wireless-sensor-networks.html</guid>
5936 <description>Zigbee networks have this wonderful capability to self-heal, which means theycan reorder connections between them if one of them is inoperable.</description>
5937 <content:encoded>&lt;p&gt;Zigbee networks have this wonderful capability to self-heal, which means they
5938can reorder connections between them if one of them is inoperable. This works
5939our of the box when you deploy them. But you have to have in mind that achieving
5940this is not as easy as you would think. None of it is plug&amp;amp;play. So to make
5941your life a bit easier, here are some pointers which, I hope, will help you.&lt;/p&gt;
5942&lt;ul&gt;
5943&lt;li&gt;Be careful when you are ordering your equipment abroad. There are many rules
5944and regulations you need to comply before you get your Xbee radios. What they
5945do is they wait until you prove that you won’t use the technology for some
5946kind of evil take over control of the world project :). For this, they have
5947EAR (Export Administration Regulations) which basically means “This product
5948may require a license to export from the United States.”.&lt;/li&gt;
5949&lt;li&gt;I don’t know if this applies for every country, but when we purchased our Xbee
5950radios from Mouser, this was mandatory! What we needed to do was to print out
5951a form and write information about our company and send them a copy via
5952email. With this document, we proved that we are a legitimate company.&lt;/li&gt;
5953&lt;li&gt;When you complete your purchase and send all the documentation, you are not
5954clear yet. Then customs will take it from there :). There will be some
5955additional costs. Before purchasing, make sure you have as much information
5956about costs as possible. Because it can get costly in the end.&lt;/li&gt;
5957&lt;li&gt;I suggest you use companies from your country. You can seriously cut your
5958costs. Here in Slovenia, the best option so far as I know is Farnell. And
5959based on my personal experience, they rock! All I need to say!&lt;/li&gt;
5960&lt;li&gt;Make plans when ordering larger quantities. Do not, I say, do not make your
5961orders in December! :) Believe me! You will have problems with stock they can
5962provide for you. So, we were forced to buy some things from Mouser, which was
5963extremely painful because of all the regulations you need to obey when
5964importing goods from the USA.&lt;/li&gt;
5965&lt;li&gt;Make sure that firmware version on your Xbee radios is exactly the same! Do
5966not get creative!!! I propose using templates. You can get template by
5967exporting settings/profile in X-CTU application. Make sure you have enabled
5968“Upgrade firmware” so you can be sure each radio has the same firmware.&lt;/li&gt;
5969&lt;li&gt;And again: make plans! Plan everything! In months advanced! You will thank me
5970later :)&lt;/li&gt;
5971&lt;li&gt;Test, test, test. Wireless networks can be tricky.&lt;/li&gt;
5972&lt;/ul&gt;
5973&lt;p&gt;If you are serious, I suggest you buy this book, Building Wireless Sensor
5974Networks. You will get a glimpse of how networks work in lumens terms. It is a
5975good starting point for everybody who wants to build wireless networks.&lt;/p&gt;
5976&lt;p&gt;&lt;strong&gt;Additional resources:&lt;/strong&gt;&lt;/p&gt;
5977&lt;ul&gt;
5978&lt;li&gt;&lt;a href=&#34;http://www.digi.com/aboutus/export/generalexportinfo&#34;&gt;http://www.digi.com/aboutus/export/generalexportinfo&lt;/a&gt;&lt;/li&gt;
5979&lt;li&gt;&lt;a href=&#34;http://doresearch.stanford.edu/research-scholarship/export-controls/export-controlled-or-embargoed-countries-entities-and-persons&#34;&gt;http://doresearch.stanford.edu/research-scholarship/export-controls/export-controlled-or-embargoed-countries-entities-and-persons&lt;/a&gt;&lt;/li&gt;
5980&lt;li&gt;&lt;a href=&#34;http://www.bis.doc.gov/licensing/exportingbasics.htm&#34;&gt;http://www.bis.doc.gov/licensing/exportingbasics.htm&lt;/a&gt;&lt;/li&gt;
5981&lt;/ul&gt;
5982</content:encoded>
5983 </item>
5984
5985
5986
5987 <item>
5988 <title>LED technology might not be as eco-friendly as you think</title>
5989 <link>https://mitjafelicijan.com/led-technology-not-so-eco.html</link>
5990 <pubDate>Fri, 09 Mar 2012 12:00:00 &#43;0200</pubDate>
5991 <guid>https://mitjafelicijan.com/led-technology-not-so-eco.html</guid>
5992 <description>There is a lot of talk about LED technology.</description>
5993 <content:encoded>&lt;p&gt;There is a lot of talk about LED technology. It is beginning to infiltrate
5994industry at a fast rate, and it’s a challenge for designers and also engineers.
5995I wondered when a weakness will be revealed. Then I stomped on an article
5996talking about harm in using LED technology. It looks like this magical
5997technology is not so magical and eco-friendly.&lt;/p&gt;
5998&lt;p&gt;A new study from the University of California indicates that LED lights contain
5999toxic metals, and should be produced, used and disposed of carefully. Besides
6000the lead and nickel, the bulbs and their associated parts were also found to
6001contain arsenic, copper, and other metals that have been linked to different
6002cancers, neurological damage, kidney disease, hypertension, skin rashes and
6003other illnesses in humans, and to ecological damage in waterways.&lt;/p&gt;
6004&lt;p&gt;Since then, I haven’t yet found any regulation for disposal of LED lights or any
6005other regulation or standard. This might be a problem in the future. And it is a
6006massive drawback. This might have quite an impact on consumer market.&lt;/p&gt;
6007&lt;p&gt;Nevertheless, there is a potential, and I am sure the market will adapt. I also
6008hope I will be reading documents regarding solution for this concern soon.&lt;/p&gt;
6009&lt;p&gt;&lt;strong&gt;Additional resources:&lt;/strong&gt;&lt;/p&gt;
6010&lt;ul&gt;
6011&lt;li&gt;&lt;a href=&#34;http://ezinearticles.com/?Recycling-and-Disposal-of-Light-Bulbs&amp;amp;id=1091304&#34;&gt;Recycling and Disposal of Light Bulbs&lt;/a&gt;&lt;/li&gt;
6012&lt;li&gt;&lt;a href=&#34;http://www.ehow.com/how_7483442_dispose-lowenergy-light-bulb.html&#34;&gt;How to Dispose of a Low-Energy Light Bulb&lt;/a&gt;&lt;/li&gt;
6013&lt;/ul&gt;
6014</content:encoded>
6015 </item>
6016
6017
6018
6019 <item>
6020 <title>Most likely to succeed in the year of 2011</title>
6021 <link>https://mitjafelicijan.com/most-likely-to-succeed-in-year-of-2011.html</link>
6022 <pubDate>Thu, 13 Jan 2011 12:00:00 &#43;0200</pubDate>
6023 <guid>https://mitjafelicijan.com/most-likely-to-succeed-in-year-of-2011.html</guid>
6024 <description>The year of 2010 was definitely the year of Geo-location.</description>
6025 <content:encoded>&lt;p&gt;The year of 2010 was definitely the year of Geo-location. The market responded
6026beautifully and lots of very cool services were launched. We all have to thank
6027the mobile market for such extensive adoption. With new generations of mobile
6028phones that are not only buffed with high-tech hardware but are also affordable.
6029We can now manage tasks that were not so long time ago, almost Star Trek’ish.
6030And all this had and has great influence on the destination to which we are
6031going now.&lt;/p&gt;
6032&lt;p&gt;Reading all this articles about new innovation about new thriving technologies
6033makes me wonder what’s the next step. The future is the mesh, like Lisa Gansky
6034said in her book The Mesh.&lt;/p&gt;
6035&lt;p&gt;Many still have conservative views on distributed systems. The problems with
6036security of information. Fear of not controlling every aspect of information
6037flow. I am very opened to distributed systems and heterogeneous applications,
6038and I think this is the correct and best way to proceed.&lt;/p&gt;
6039&lt;p&gt;This year will definitely be about communication platforms. Mobile to mobile.
6040Machine to mobile and vice versa. All the tech is available and ready to put
6041into action. Wireless is today’s new mantra. And the concept of semantic web is
6042now ready for industry.&lt;/p&gt;
6043&lt;p&gt;Applications and developers now can gain access to new layers of systems and can
6044prepare and build solutions to meet the high quality needs of market. The speed
6045is everything now.&lt;/p&gt;
6046&lt;p&gt;My vote goes to “Machine to Machine” and “Embedded Systems”!&lt;/p&gt;
6047&lt;ul&gt;
6048&lt;li&gt;&lt;a href=&#34;http://en.wikipedia.org/wiki/Machine-to-Machine&#34;&gt;Machine-to-Machine&lt;/a&gt;&lt;/li&gt;
6049&lt;li&gt;&lt;a href=&#34;http://www.bitxml.org/&#34;&gt;The ultimate M2M communication protocol&lt;/a&gt;&lt;/li&gt;
6050&lt;li&gt;&lt;a href=&#34;http://www.coosproject.org/maven-site/1.0.0/project-info.html&#34;&gt;COOS Project (connectivity initiative)&lt;/a&gt;&lt;/li&gt;
6051&lt;li&gt;&lt;a href=&#34;http://m2m.com/index.jspa&#34;&gt;Community for machine-to-machine&lt;/a&gt;&lt;/li&gt;
6052&lt;li&gt;&lt;a href=&#34;http://en.wikipedia.org/wiki/Embedded_system&#34;&gt;Embedded system&lt;/a&gt;&lt;/li&gt;
6053&lt;/ul&gt;
6054</content:encoded>
6055 </item>
6056
6057
6058 </channel>
6059</rss>
generated by cgit v1.2.3 (git 2.46.0) at 2026-04-30 21:32:57 +0000