From 23a56bd50b04211da3cab45f72c3390711b2416b Mon Sep 17 00:00:00 2001 From: Mitja Felicijan Date: Wed, 12 Jul 2023 18:35:08 +0200 Subject: Moved notes and posts into subfolders --- ...01-13-most-likely-to-succeed-in-year-of-2011.md | 42 ++ .../posts/2012-03-09-led-technology-not-so-eco.md | 33 ++ .../posts/2013-10-24-wireless-sensor-networks.md | 54 ++ .../2015-11-10-software-development-pitfalls.md | 181 ++++++ .../2017-03-07-golang-profiling-simplified.md | 126 +++++ ...04-17-what-i-ve-learned-developing-ad-server.md | 199 +++++++ ...ng-python-web-applications-with-visual-tools.md | 206 +++++++ content/posts/2017-08-11-simple-iot-application.md | 607 +++++++++++++++++++++ ...digitalocean-spaces-object-storage-with-fuse.md | 331 +++++++++++ ...01-03-encoding-binary-data-into-dna-sequence.md | 411 ++++++++++++++ .../2019-10-14-simplifying-and-reducing-clutter.md | 59 ++ ...g-sentiment-analysis-for-clickbait-detection.md | 108 ++++ .../2020-03-22-simple-sse-based-pubsub-server.md | 454 +++++++++++++++ ...0-03-27-create-placeholder-images-with-sharp.md | 102 ++++ ...nge-case-of-elasticsearch-allocation-failure.md | 108 ++++ ...30-my-love-and-hate-relationship-with-nodejs.md | 111 ++++ content/posts/2020-05-05-remote-work.md | 72 +++ .../2020-08-15-systemd-disable-wake-onmouse.md | 73 +++ content/posts/2020-09-06-esp-and-micropython.md | 226 ++++++++ content/posts/2020-09-08-bind-warning-on-login.md | 54 ++ content/posts/2020-09-09-digitalocean-sync.md | 112 ++++ .../posts/2021-01-24-replacing-dropbox-with-s3.md | 114 ++++ content/posts/2021-01-25-goaccess.md | 203 +++++++ content/posts/2021-06-26-simple-world-clock.md | 108 ++++ ...from-internet-consumer-to-full-hominum-again.md | 103 ++++ content/posts/2021-08-01-linux-cheatsheet.md | 287 ++++++++++ ...n-based-riced-up-distribution-for-developers.md | 276 ++++++++++ ...021-12-25-running-golang-application-as-pid1.md | 348 ++++++++++++ .../2021-12-30-wap-mobile-web-before-the-web.md | 202 +++++++ .../posts/2022-06-30-trying-out-helix-editor.md | 53 ++ ...22-07-05-what-would-dna-sound-if-synthesized.md | 364 ++++++++++++ .../2022-08-13-algae-spotted-on-river-sava.md | 31 ++ ...10-06-state-of-web-technologies-in-year-2022.md | 304 +++++++++++ ...hat-sound-that-machine-makes-when-struggling.md | 66 +++ ...ing-to-build-a-new-kind-of-terminal-emulator.md | 253 +++++++++ ...023-05-16-rekindling-my-love-for-programming.md | 74 +++ .../2023-05-22-crafting-stories-in-zed-editor.md | 88 +++ .../2023-05-23-i-was-wrong-about-git-workflows.md | 71 +++ ...nting-task-runner-that-i-actually-used-daily.md | 159 ++++++ ...l-of-my-projects-together-under-one-umbrella.md | 281 ++++++++++ ...knows-what-the-world-will-look-like-tomorrow.md | 99 ++++ ...-fix-screen-tearing-on-debian-12-xorg-and-i3.md | 22 + ...nline-radio-streaming-with-mpv-from-terminal.md | 14 + 43 files changed, 7189 insertions(+) create mode 100644 content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md create mode 100644 content/posts/2012-03-09-led-technology-not-so-eco.md create mode 100644 content/posts/2013-10-24-wireless-sensor-networks.md create mode 100644 content/posts/2015-11-10-software-development-pitfalls.md create mode 100644 content/posts/2017-03-07-golang-profiling-simplified.md create mode 100644 content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md create mode 100644 content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md create mode 100644 content/posts/2017-08-11-simple-iot-application.md create mode 100644 content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md create mode 100644 content/posts/2019-01-03-encoding-binary-data-into-dna-sequence.md create mode 100644 content/posts/2019-10-14-simplifying-and-reducing-clutter.md create mode 100644 content/posts/2019-10-19-using-sentiment-analysis-for-clickbait-detection.md create mode 100644 content/posts/2020-03-22-simple-sse-based-pubsub-server.md create mode 100644 content/posts/2020-03-27-create-placeholder-images-with-sharp.md create mode 100644 content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md create mode 100644 content/posts/2020-03-30-my-love-and-hate-relationship-with-nodejs.md create mode 100644 content/posts/2020-05-05-remote-work.md create mode 100644 content/posts/2020-08-15-systemd-disable-wake-onmouse.md create mode 100644 content/posts/2020-09-06-esp-and-micropython.md create mode 100644 content/posts/2020-09-08-bind-warning-on-login.md create mode 100644 content/posts/2020-09-09-digitalocean-sync.md create mode 100644 content/posts/2021-01-24-replacing-dropbox-with-s3.md create mode 100644 content/posts/2021-01-25-goaccess.md create mode 100644 content/posts/2021-06-26-simple-world-clock.md create mode 100644 content/posts/2021-07-30-from-internet-consumer-to-full-hominum-again.md create mode 100644 content/posts/2021-08-01-linux-cheatsheet.md create mode 100644 content/posts/2021-12-03-debian-based-riced-up-distribution-for-developers.md create mode 100644 content/posts/2021-12-25-running-golang-application-as-pid1.md create mode 100644 content/posts/2021-12-30-wap-mobile-web-before-the-web.md create mode 100644 content/posts/2022-06-30-trying-out-helix-editor.md create mode 100644 content/posts/2022-07-05-what-would-dna-sound-if-synthesized.md create mode 100644 content/posts/2022-08-13-algae-spotted-on-river-sava.md create mode 100644 content/posts/2022-10-06-state-of-web-technologies-in-year-2022.md create mode 100644 content/posts/2022-10-16-that-sound-that-machine-makes-when-struggling.md create mode 100644 content/posts/2023-01-26-trying-to-build-a-new-kind-of-terminal-emulator.md create mode 100644 content/posts/2023-05-16-rekindling-my-love-for-programming.md create mode 100644 content/posts/2023-05-22-crafting-stories-in-zed-editor.md create mode 100644 content/posts/2023-05-23-i-was-wrong-about-git-workflows.md create mode 100644 content/posts/2023-05-31-re-inventing-task-runner-that-i-actually-used-daily.md create mode 100644 content/posts/2023-07-01-bringing-all-of-my-projects-together-under-one-umbrella.md create mode 100644 content/posts/2023-07-08-who-knows-what-the-world-will-look-like-tomorrow.md create mode 100644 content/posts/2023-07-10-fix-screen-tearing-on-debian-12-xorg-and-i3.md create mode 100644 content/posts/2023-07-10-online-radio-streaming-with-mpv-from-terminal.md (limited to 'content/posts') diff --git a/content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md b/content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md new file mode 100644 index 0000000..325bd52 --- /dev/null +++ b/content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md @@ -0,0 +1,42 @@ +--- +title: Most likely to succeed in the year of 2011 +url: most-likely-to-succeed-in-year-of-2011.html +date: 2011-01-13T12:00:00+02:00 +type: post +draft: false +--- + +The year of 2010 was definitely the year of Geo-location. The market responded +beautifully and lots of very cool services were launched. We all have to thank +the mobile market for such extensive adoption. With new generations of mobile +phones that are not only buffed with high-tech hardware but are also affordable. +We can now manage tasks that were not so long time ago, almost Star Trek’ish. +And all this had and has great influence on the destination to which we are +going now. + +Reading all this articles about new innovation about new thriving technologies +makes me wonder what’s the next step. The future is the mesh, like Lisa Gansky +said in her book The Mesh. + +Many still have conservative views on distributed systems. The problems with +security of information. Fear of not controlling every aspect of information +flow. I am very opened to distributed systems and heterogeneous applications, +and I think this is the correct and best way to proceed. + +This year will definitely be about communication platforms. Mobile to mobile. +Machine to mobile and vice versa. All the tech is available and ready to put +into action. Wireless is today’s new mantra. And the concept of semantic web is +now ready for industry. + +Applications and developers now can gain access to new layers of systems and can +prepare and build solutions to meet the high quality needs of market. The speed +is everything now. + +My vote goes to “Machine to Machine” and “Embedded Systems”! + +- [Machine-to-Machine](http://en.wikipedia.org/wiki/Machine-to-Machine) +- [The ultimate M2M communication protocol](http://www.bitxml.org/) +- [COOS Project (connectivity initiative)](http://www.coosproject.org/maven-site/1.0.0/project-info.html) +- [Community for machine-to-machine](http://m2m.com/index.jspa) +- [Embedded system](http://en.wikipedia.org/wiki/Embedded_system) + diff --git a/content/posts/2012-03-09-led-technology-not-so-eco.md b/content/posts/2012-03-09-led-technology-not-so-eco.md new file mode 100644 index 0000000..2841d0a --- /dev/null +++ b/content/posts/2012-03-09-led-technology-not-so-eco.md @@ -0,0 +1,33 @@ +--- +title: LED technology might not be as eco-friendly as you think +url: led-technology-not-so-eco.html +date: 2012-03-09T12:00:00+02:00 +type: post +draft: false +--- + +There is a lot of talk about LED technology. It is beginning to infiltrate +industry at a fast rate, and it’s a challenge for designers and also engineers. +I wondered when a weakness will be revealed. Then I stomped on an article +talking about harm in using LED technology. It looks like this magical +technology is not so magical and eco-friendly. + +A new study from the University of California indicates that LED lights contain +toxic metals, and should be produced, used and disposed of carefully. Besides +the lead and nickel, the bulbs and their associated parts were also found to +contain arsenic, copper, and other metals that have been linked to different +cancers, neurological damage, kidney disease, hypertension, skin rashes and +other illnesses in humans, and to ecological damage in waterways. + +Since then, I haven’t yet found any regulation for disposal of LED lights or any +other regulation or standard. This might be a problem in the future. And it is a +massive drawback. This might have quite an impact on consumer market. + +Nevertheless, there is a potential, and I am sure the market will adapt. I also +hope I will be reading documents regarding solution for this concern soon. + +**Additional resources:** + +- [Recycling and Disposal of Light Bulbs](http://ezinearticles.com/?Recycling-and-Disposal-of-Light-Bulbs&id=1091304) +- [How to Dispose of a Low-Energy Light Bulb](http://www.ehow.com/how_7483442_dispose-lowenergy-light-bulb.html) + diff --git a/content/posts/2013-10-24-wireless-sensor-networks.md b/content/posts/2013-10-24-wireless-sensor-networks.md new file mode 100644 index 0000000..bc6b333 --- /dev/null +++ b/content/posts/2013-10-24-wireless-sensor-networks.md @@ -0,0 +1,54 @@ +--- +title: Wireless sensor networks +url: wireless-sensor-networks.html +date: 2013-10-24T12:00:00+02:00 +type: post +draft: false +--- + +Zigbee networks have this wonderful capability to self-heal, which means they +can reorder connections between them if one of them is inoperable. This works +our of the box when you deploy them. But you have to have in mind that achieving +this is not as easy as you would think. None of it is plug&play. So to make +your life a bit easier, here are some pointers which, I hope, will help you. + +- Be careful when you are ordering your equipment abroad. There are many rules + and regulations you need to comply before you get your Xbee radios. What they + do is they wait until you prove that you won’t use the technology for some + kind of evil take over control of the world project :). For this, they have + EAR (Export Administration Regulations) which basically means “This product + may require a license to export from the United States.”. +- I don’t know if this applies for every country, but when we purchased our Xbee + radios from Mouser, this was mandatory! What we needed to do was to print out + a form and write information about our company and send them a copy via + email. With this document, we proved that we are a legitimate company. +- When you complete your purchase and send all the documentation, you are not + clear yet. Then customs will take it from there :). There will be some + additional costs. Before purchasing, make sure you have as much information + about costs as possible. Because it can get costly in the end. +- I suggest you use companies from your country. You can seriously cut your + costs. Here in Slovenia, the best option so far as I know is Farnell. And + based on my personal experience, they rock! All I need to say! +- Make plans when ordering larger quantities. Do not, I say, do not make your + orders in December! :) Believe me! You will have problems with stock they can + provide for you. So, we were forced to buy some things from Mouser, which was + extremely painful because of all the regulations you need to obey when + importing goods from the USA. +- Make sure that firmware version on your Xbee radios is exactly the same! Do + not get creative!!! I propose using templates. You can get template by + exporting settings/profile in X-CTU application. Make sure you have enabled + “Upgrade firmware” so you can be sure each radio has the same firmware. +- And again: make plans! Plan everything! In months advanced! You will thank me + later :) +- Test, test, test. Wireless networks can be tricky. + +If you are serious, I suggest you buy this book, Building Wireless Sensor +Networks. You will get a glimpse of how networks work in lumens terms. It is a +good starting point for everybody who wants to build wireless networks. + +**Additional resources:** + +- http://www.digi.com/aboutus/export/generalexportinfo +- http://doresearch.stanford.edu/research-scholarship/export-controls/export-controlled-or-embargoed-countries-entities-and-persons +- http://www.bis.doc.gov/licensing/exportingbasics.htm + diff --git a/content/posts/2015-11-10-software-development-pitfalls.md b/content/posts/2015-11-10-software-development-pitfalls.md new file mode 100644 index 0000000..6a5d9bd --- /dev/null +++ b/content/posts/2015-11-10-software-development-pitfalls.md @@ -0,0 +1,181 @@ +--- +title: Software development and my favorite pitfalls +url: software-development-pitfalls.html +date: 2015-11-10T12:00:00+02:00 +type: post +draft: false +--- + +Over the years I had the privilege to work on some very excited projects both in +software development field and also in electronics field and every experience +taught me some invaluable lessons about how NOT TO approach development. And +through this post I will try to point out some absurd, outdated techniques I +find the most annoying and damaging during a development cycle. There will be +swearing because this topic really gets on my nerves and I never coherently +tried to explain them in writing. So if I get heated up, please bear with me. + +As new methods of project management are emerging, underlying processes still +stay old and outdated. This is mainly because we as people are unable to +completely shift away from these approaches. + +I was always struggling with communication, and many times that cost me a +relationship or two because I was not on the ball all the time. Through every +experience, I became more convinced that I am the problem and never ever doubted +that the problem may be that communication never evolved a single step from +emails. And if you think for a second, not many things have changed around this +topic. We just have different representations of email (message boards, chats, +project management tools). And I believe this is the real issue we are facing +now. + +There are many articles written about hyper connectivity and the effects that +are a direct result of it. But mainstream does nothing towards it. We are just +putting out fires, and we do nothing to prevent it. I am certain this will be a +major source of grief in coming years. And what we all can do to avoid this is +to change our mindset and experiment on our communication skills, development +approaches. We need to maximize possible output that a person can give. And to +achieve this we need to listen to them, encourage them. I know that not +everybody is a naturally born leader, but with enough practice and encouragement +they also can become active participants in leadership. + +There are many talks now about methodologies such as Scrum, Kanban, Cleanroom +and they all fucking piss me of :). These are all boxes that imprison people and +take away their freedom of thought. This is a straightforward mindfuck / +amputation of creativity. + +Let me list a couple of things that I find really destructive and bad for a +project and in a long run company. + +## Ping emails + +Ping emails are emails you have to write as soon as you receive an email. Its +sole purpose is to inform the sender that you received their email, and you are +working on it. Its result is only to calm down the sender that their task is +being dealt with. It’s intent basically is, I did my job by sending you this +email, so I am on clear grounds. I categorize this email as fuck you email. +This is one of the most irritating types of emails I need to write. This is the +ultimate control freak show you can experience, and it gives the sender a false +feeling of control. Newsflash: We do not live in 1982 where there was a +possibility that email never reached the destination. I really hate this from +the bottom of my heart. + +They should be like: “Yes, I am fucking alive, and I am at your service my +leash!”. I guess if I would reply like this, I wouldn’t have to write any more +of this kind of messages. + +## Everybody is a project manager + +Well, this is a tough one. I noticed that as soon as you let people to give +their suggestions, you are basically screwed. There is a truth in the saying: +“Give low expectations and deliver little more than you promised.”. + +People tend to take a role of a manager as soon as they are presented with an +opportunity. And by getting angry at them, you only provoke yourself. They are +not at fault. You just need to tell them they are only giving suggestions and +not tasks at the beginning and everything will be alright. But if you give them +a feeling that they are in control, you will have immense problems explaining +why their features are not in current release. + +Project mission must be always leading project requirements and any deviation +from it will result in major project butchering. And by this, I mean that the +project will get its own path, and you will be left with half done software that +helps nobody. Clear mission goals and clean execution will allow you to develop +software will clear intent. + +## We are never wrong + +I find this type of arrogance the worst. We must always conduct ourselves that +we are infallible and cannot make mistakes. As soon as a procedure or process is +established, there is no room for changes or improvements. This is the most +idiotic thing someone can say of think. I think that processes need to involve +and change over time. This is imperative and need to have in your organization +if you want to improve and develop company. We all need to grow balls and change +everything in order to adapt to current situations. Being a prisoner of +predefined processes kills creativity. + +I am constantly trying new software for project managing and communication. I +believe every team has its own dynamic, and it needs to be discovered +organically and naturally through many experiments. By putting the team in a +box, you are amputating their creativity and therefore minimizing their +potential. But if you talk to an executive, you will mainly find archetypical +thinking and a strong need to compartmentalize everything from business +processes to resource management. And this type of management that often +displays micromanagement techniques only works for short periods (couple of +years) and then employees either leave the company or become basically retarded +drones on autopilot. + +## Micromanaging + +This basically implies that everybody on the team is an idiot who needs to have +a to-do list that they cannot write themselves. How about spoon-feeding the team +at launch because besides the team leader, everybody must be a retarded idiot at +best? + +I prefer milestones as they give developers much more freedom and creativity in +developing and not waste their time checking some bizarre to-do list that was +not even thought through. Projects constantly change throughout the development +cycle, and all you are left at the end is a list of unchecked tasks and the +wrath of management why they are not completed. Best WTF moment! + +## Human contact — no need for it! + +We are vigorously trying to eliminate physical contact by replacing short +meetings with software, with no regards that we are not machines. Many times a +simple 5-min meeting at morning can solve most of the problems. In rapid +development, short bursts of man to man communication is possibly the best way +to go. + +We now have all this software available, and all what we get out of it is a +giant clusterfuck. An obstacle and not a solution. So, why we still use them? + +## MVP is killing innovation + +Many will disagree with me on this one, but I stand strong by this statement. +What I noticed in my experience that all this buzz words around us only mislead +and capture us in a circle of solving issues that already have a solution, but +we are unable to see it without using some fancy word for it. + +The toughest thing to do for a developer is to minimize requirements. Well, this +is though only for bad developers. Yes, I said it. There are many types of +developers out there. And those unable to minimize feature scope are the ones +you don’t need on your team. Their only goal is to solve problems that exist +only in their heads. And then you have to argue with them, and waste energy on +them, instead of developing your awesome product. They are a cancer and I +suggest you cut them off. + +MVP as an idea is great, but sadly people don’t understand underlying +philosophy, and they spent too much time focusing and fixating on something that +every sane person with normal IQ will understand without some made up +acronym. And the result is a lot of talking and barely no execution. + +Well, MVP is not directly killing innovation, but stupid people do when they try +to understand it. + +## Pressure wasteland + +You must never allow to be pressured into confirming a deadline if you are not +confident. We often feel a need that we are in service of others, which is true +to some extent. But it is also true that others are in service to us to some +extent. And we forget this all the time. We are all pressured all the time to +make decisions just to calm other people down. And when they leave your office +you experience WTF moment :) How the hell did they manage to fuck me up again? + +People need to realize that the more pressure you put on somebody, the less they +will be able to do. So 5-min update email requests will only resolve in mental +breakdown and inability to work that day. Constant poking is probably the only +thing I lose my mind instantly. For all you that are doing this: “Stop bothering +us with your insecurities and let us do our job. We will do it quicker and +better without you breathing down our necks.” + +If this happens to me, I end up with no energy at the end. Don’t you get it? +You will get much more from and out of me if you ask me like a human person and +not your personal butler. On a long run, you are destroying your relationships +and nobody would want to work with you. Your schizophrenic approach will damage +only you in a long run. Nobody is anybody’s property. + +## Conclusion + +I am guilty of many things described in this post. And I find it hard sometimes +to acknowledge this. And I lie to myself and try vigorously to find some +explanation why I do these things. There is always space for growth. And maybe +you will also find some of yourself in this post and realize what needs to +change for you to evolve. diff --git a/content/posts/2017-03-07-golang-profiling-simplified.md b/content/posts/2017-03-07-golang-profiling-simplified.md new file mode 100644 index 0000000..ee3a210 --- /dev/null +++ b/content/posts/2017-03-07-golang-profiling-simplified.md @@ -0,0 +1,126 @@ +--- +title: Golang profiling simplified +url: golang-profiling-simplified.html +date: 2017-03-07T12:00:00+02:00 +type: post +draft: false +--- + +Many posts have been written regarding profiling in Golang and I haven’t found +proper tutorial regarding this. Almost all of them are missing some part of +important information and it gets pretty frustrating when you have a deadline +and are not finding simple distilled solution. + +Nevertheless, after searching and experimenting I have found a solution that +works for me and probably should also for you. + +## Where are my pprof files? + +By default pprof files are generated in /tmp/ folder. You can override folder +where this files are generated programmatically in your golang code as we will +see below in example. + +## Why is my CPU profile empty? + +I have found out that sometimes CPU profile is empty because program was not +executing long enough. Programs, that execute too quickly don’t produce pprof +file in my cases. Well, file is generated but only contains 4KB of information. + +## Profiling + +As you can see from examples we are executing dummy_benchmark functions to +ensure some sort of execution. Memory profiling can be done without such a +“complex” function. But CPU profiling needs it. + +Both memory and CPU profiling examples are almost the same. Only parameters in +main function when calling profile.Start are different. When we set +profile.ProfilePath(“.”) we tell profiler to store pprof files in the same +folder as our program. + +### Memory profiling + +```go +package main + +import ( + "fmt" + "time" + "github.com/pkg/profile" +) + +func dummy_benchmark() { + + fmt.Println("first set ...") + for i := 0; i < 918231333; i++ { + i *= 2 + i /= 2 + } + + <-time.After(time.Second*3) + + fmt.Println("sencond set ...") + for i := 0; i < 9182312232; i++ { + i *= 2 + i /= 2 + } +} + +func main() { + defer profile.Start(profile.MemProfile, profile.ProfilePath("."), profile.NoShutdownHook).Stop() + dummy_benchmark() +} +``` + +### CPU profiling + +```go +package main + +import ( + "fmt" + "time" + "github.com/pkg/profile" +) + +func dummy_benchmark() { + + fmt.Println("first set ...") + for i := 0; i < 918231333; i++ { + i *= 2 + i /= 2 + } + + <-time.After(time.Second*3) + + fmt.Println("sencond set ...") + for i := 0; i < 9182312232; i++ { + i *= 2 + i /= 2 + } +} + +func main() { + defer profile.Start(profile.CPUProfile, profile.ProfilePath("."), profile.NoShutdownHook).Stop() + dummy_benchmark() +} +``` + +### Generating profiling reports + +```bash +# memory profiling +go build mem.go +./mem +go tool pprof -pdf ./mem mem.pprof > mem.pdf + +# cpu profiling +go build cpu.go +./cpu +go tool pprof -pdf ./cpu cpu.pprof > cpu.pdf +``` + +This will generate PDF document with visualized profile. + +- [Memory PDF profile example](/assets/go-profiling/golang-profiling-mem.pdf) +- [CPU PDF profile example](/assets/go-profiling/golang-profiling-cpu.pdf) + diff --git a/content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md b/content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md new file mode 100644 index 0000000..3a6410f --- /dev/null +++ b/content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md @@ -0,0 +1,199 @@ +--- +title: What I've learned developing ad server +url: what-i-ve-learned-developing-ad-server.html +date: 2017-04-17T12:00:00+02:00 +type: post +draft: false +--- + +For the past year and half I have been developing native advertising server that +contextually matches ads and displays them in different template forms on +variety of websites. This project grew from serving thousands of ads per day to +millions. + +The system is made from couple of core components: + +- API for serving ads, +- Utils - cronjobs and queue management tools, +- Dashboard UI. + +Initial release was using [MongoDB](https://www.mongodb.com/) for full-text +search but was later replaced by [Elasticsearch](https://www.elastic.co/) for +better CPU utilization and better search performance. This provided us with many +amazing functionalities of [Elasticsearch](https://www.elastic.co/). You should +check it out if you do any search related operations. + +Because the premise of the server is to provide native ad experience, they are +rendered on the client side via simple templating engine. This ensures that ads +can be displayed number of different ways based on the visual style of the +page. And this makes JavaScript client library quite complex. + +So now that you know basic information about the product lets get into the +lessons we learned. + +## Aggregate everything + +After beta version was released everything (impressions, clicks, etc) was +written in nanosecond resolution in the database. At that time we were using +[PostgreSQL](https://www.postgresql.org/) and database quickly grew way above +200GB in disk space. And that was problematic. Statistics took disturbingly long +time to aggregate. Also using indexes on stats table in database was no help +after we reached 500 million datapoints. + +> There is a marketing product information and there is real life experience. +And the tend to be quite the opposite. + +This was the reason that now everything is aggregated on daily basis and this +data is then fed to Elastic in form of daily summary. With this we achieved we +can now track many more dimensions such as zone, channel and platform +information. And with this information we can now adapt occurrences of ads on +specific places more precisely. + +We have also adapted [Redis](https://redis.io/) as a full-time citizen in our +stack. Because Redis also stores information on a local disk we have some sort +of backup if server would accidentally suffer some failure. + +All the real-time statistics for ad serving and redirecting is presented as +counters in Redis instance and daily extracted and pushed to Elastic. + +## Measure everything + +The thing about software is that we really don't know how well it is performing +under load until such load is presented. When testing locally everything is fine +but when on production things tend to fall apart. + +As a solution for this we are measuring everything we can. Function execution +time (by encapsulating functions with timers), server performance (cpu, memory, +disk, etc), Nginx and [uWSGI](https://uwsgi-docs.readthedocs.io/) performance. +We sacrifice a bit of performance for the sake of this information. And we store +all this information for later analysis. + +**Example of function execution time** + +```json +{ + "get_final_filtered_ads": { + "counter": 1931250, + "avg": 0.0066143431, + "elapsed": 12773.9500310003 + }, + "store_keywords_statistics": { + "counter": 1931011, + "avg": 0.0004605267, + "elapsed": 889.2821669996 + }, + "match_by_context": { + "counter": 1931011, + "avg": 0.0055960716, + "elapsed": 10806.0758889999 + }, + "match_by_high_performance": { + "counter": 262, + "avg": 0.0152770229, + "elapsed": 4.00258 + }, + "store_impression_stats": { + "counter": 1931250, + "avg": 0.0006189991, + "elapsed": 1195.4419869999 + } +} +``` + +We have also started profiling with [cProfile](https://pymotw.com/2/profile/) +and then visualizing with [KCachegrind](http://kcachegrind.sourceforge.net/). +This provides much more detailed look into code execution. + +## Cache control is your friend + +Because we use Javascript library for rendering ads we rely on this script +extensively and when in need we need to be able to change behavior of the script +quickly. + +In our case we can not simply replace javascript url in html code. It usually +takes a day or two for the guys who maintain sites to change code or add +?ver=xxx attribute. And this makes rapid deployment and testing very difficult +and time consuming. There is a limitation of how much you can test locally. + +We are now in the process of integrating [Google Tag +Manager](https://www.google.com/analytics/tag-manager/) but couple of websites +are developed on ASP.net platform that have some problems with tag manager. With +a solution below we are certain that we are serving latest version of the +script. + +And it only takes one mistake and users have the script cached and in case of +caching it for 1 year you probably know where the problem is. + +```nginx +# nginx ➜ /etc/nginx/sites-available/default +location /static/ { + alias /path-to-static-content/; + autoindex off; + charset utf-8; + gzip on; + gzip_types text/plain application/javascript application/x-javascript text/javascript text/xml text/css; + location ~* \.(ico|gif|jpeg|jpg|png|woff|ttf|otf|svg|woff2|eot)$ { + expires 1y; + add_header Pragma public; + add_header Cache-Control "public"; + } + location ~* \.(css|js|txt)$ { + expires 3600s; + add_header Pragma public; + add_header Cache-Control "public, must-revalidate"; + } +} +``` + +Also be careful when redirecting to url in your python code. We noticed that if +we didn't precisely setup cache control and expire headers in response we didn't +get the request on the server and therefore couldn't measure clicks. So when +redirecting do as follows and there will be no problems. + +```python +# python ➜ bottlepy web micro-framework +response = bottle.HTTPResponse(status=302) +response.set_header("Cache-Control", "no-store, no-cache, must-revalidate") +response.set_header("Expires", "Thu, 01 Jan 1970 00:00:00 GMT") +response.set_header("Location", url) +return response +``` + +> Cache control in browsers is quite aggressive and you need to be precise to +avoid future problems. We learned that lesson the hard way. + +## Learn NGINX + +When deciding on a web server we went with Nginx as a reverse proxy for our +applications. We adapted micro-service oriented architecture early in the +project to ensure when we scale we can easily add additional servers to our +cluster. And Nginx was crucial to perform load balancing and static content +delivery. + +At first our config file was quite simple and later grew larger. After patching +and adding new settings I sat down and learned more about the guts of Nginx. +This proved to be very useful and we were able to squeeze much more out of our +setup. So I advise you to take your time and read through the +[documentation](https://nginx.org/en/docs/). This saved us a lot of headache. +Googling for solutions only goes so far. + +## Use Redis/Memcached + +As explained above we are using caching basically for everything. It is the +corner stone of our services. At first we were very careful about the quantity +of things we stored in [Redis](https://redis.io/). But we later found out that +the memory footprint is very low even when storing large amount of data in it. + +So we gradually increased our usage to caching whole HTML outputs of dashboard. +This improved our performance in order of magnitude. And by using native TTL +support this goes hand in hand with our needs. + +The reason why we choose [Redis](https://redis.io/) over +[Memcached](https://memcached.org/) was the nature of scalability of Redis out +of the box. But all this can be achieved with Memcached. + +## Conclusion + +There are a lot more details that could have been written and every single topic +in here deserves it's own post but you probably got the idea about the problems +we faced. diff --git a/content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md b/content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md new file mode 100644 index 0000000..d1cea7c --- /dev/null +++ b/content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md @@ -0,0 +1,206 @@ +--- +title: Profiling Python web applications with visual tools +url: profiling-python-web-applications-with-visual-tools.html +date: 2017-04-21T12:00:00+02:00 +type: post +draft: false +--- + +I have been profiling my software with KCachegrind for a long time now and I was +missing this option when I am developing API's or other web services. I always +knew that this is possible but never really took the time and dive into it. + +Before we begin there are some requirements. We will need to: + +- implement [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) into our web app, +- convert output to [callgrind](http://valgrind.org/docs/manual/cl-manual.html) format with [pyprof2calltree](https://pypi.python.org/pypi/pyprof2calltree/), +- visualize data with [KCachegrind](http://kcachegrind.sourceforge.net/html/Home.html) or [Profiling Viewer](http://www.profilingviewer.com/). + + +If you are using MacOS you should check out [Profiling +Viewer](http://www.profilingviewer.com/) or +[MacCallGrind](http://www.maccallgrind.com/). + +![KCachegrind](/assets/python-profiling/kcachegrind.png) + +We will be dividing this post into two main categories: + +- writing simple web-service, +- visualize profile of this web-service. + +## Simple web-service + +Let's use virtualenv so we won't pollute our base system. If you don't have +virtualenv installed on your system you can install it with pip command. + +```bash +# let's install virtualenv globally +$ sudo pip install virtualenv + +# let's also install pyprof2calltree globally +$ sudo pip install pyprof2calltree + +# now we create project +$ mkdir demo-project +$ cd demo-project/ + +# now let's create folder where we will store profiles +$ mkdir prof + +# now we create empty virtualenv in venv/ folder +$ virtualenv --no-site-packages venv + +# we now need to activate virtualenv +$ source venv/bin/activate + +# you can check if virtualenv was correctly initialized by +# checking where your python interpreter is located +# if command bellow points to your created directory and not some +# system dir like /usr/bin/python then everything is fine +$ which python + +# we can check now if all is good ➜ if ok couple of +# lines will be displayed +$ pip freeze +# appdirs==1.4.3 +# packaging==16.8 +# pyparsing==2.2.0 +# six==1.10.0 + +# now we are ready to install bottlepy ➜ web micro-framework +$ pip install bottle + +# you can deactivate virtualenv but you will then go +# under system domain ➜ for now don't deactivate +$ deactivate +``` + +We are now ready to write simple web service. Let's create file app.py and paste +code bellow in this newly created file. + +```python +# -*- coding: utf-8 -*- + +import bottle +import random +import cProfile + +app = bottle.Bottle() + +# this function is a decorator and encapsulates function +# and performs profiling and then saves it to subfolder +# prof/function-name.prof +# in our example only awesome_random_number function will +# be profiled because it has do_cprofile defined +def do_cprofile(func): + def profiled_func(*args, **kwargs): + profile = cProfile.Profile() + try: + profile.enable() + result = func(*args, **kwargs) + profile.disable() + return result + finally: + profile.dump_stats("prof/" + str(func.__name__) + ".prof") + return profiled_func + + +# we use profiling over specific function with including +# @do_cprofile above function declaration +@app.route("/") +@do_cprofile +def awesome_random_number(): + awesome_random_number = random.randint(0, 100) + return "awesome random number is " + str(awesome_random_number) + +@app.route("/test") +def test(): + return "dummy test" + +if __name__ == '__main__': + bottle.run( + app = app, + host = "0.0.0.0", + port = 4000 + ) + +# run with 'python app.py' +# open browser 'http://0.0.0.0:4000' +``` + +When browser hits awesome\_random\_number() function profile is created in prof/ +subfolder. + +## Visualize profile + +Now let's create callgrind format from this cProfile output. + +```bash +$ cd prof/ +$ pyprof2calltree -i awesome_random_number.prof +# this creates 'awesome_random_number.prof.log' file in the same folder +``` + +This file can be opened with visualizing tools listed above. In this case we +will be using Profilling Viewer under MacOS. You can open image in new tab. As +you can see from this example there is hierarchy of execution order of your +code. + +![Profilling Viewer](/assets/python-profiling/profiling-viewer.png) + +> Make sure you convert output of the cProfile output every time you want to +refresh and take a look at your possible optimizations because cProfile updates +.prof file every time browser hits the function. + +This is just a simple example but when you are developing real-life applications +this can be very illuminating, especially to see which parts of your code are +bottlenecks and need to be optimized. + +## Update 2017-04-22 + +Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this awesome +web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/) +that directly takes output from +[cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) +module. + +
Comment from discussion Profiling Python web applications with visual tools.
+ +```bash +# let's install it globally as well +$ sudo pip install snakeviz + +# now let's visualize +$ cd prof/ +$ snakeviz awesome_random_number.prof +# this automatically opens browser window and +# shows visualized profile +``` + +![SnakeViz](/assets/python-profiling/snakeviz.png) + +Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better +way for installing pip software by targeting user level instead of using sudo. + +
Comment from discussion Profiling Python web applications with visual tools.
+ +```bash +# now we need to add this path to our $PATH variable +# we do this my adding this line at the end of your +# ~/.bashrc file +PATH=$PATH:$HOME/.local/bin/ + +# in order to use this new configuration you can close +# and reopen terminal or reload .bashrc file +$ source ~/.bashrc + +# now let's test if new directory is present in $PATH +$ echo $PATH + +# now we can install on user level by adding --user +# without use of sudo +$ pip install snakeviz --user +``` + +Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can +use [pipsi](https://github.com/mitsuhiko/pipsi). diff --git a/content/posts/2017-08-11-simple-iot-application.md b/content/posts/2017-08-11-simple-iot-application.md new file mode 100644 index 0000000..00a7802 --- /dev/null +++ b/content/posts/2017-08-11-simple-iot-application.md @@ -0,0 +1,607 @@ +--- +title: Simple IOT application supported by real-time monitoring and data history +url: simple-iot-application.html +date: 2017-08-11T12:00:00+02:00 +type: post +draft: false +--- + +## Initial thoughts + +I have been developing these kind of application for the better part of my last +5 years and people keep asking me how to approach developing such application +and I will give a try explaining it here. + +IOT applications are really no different than any other kind of applications. +We have data that needs to be collected and visualized in some form of tables or +charts. The main difference here is that most of the times these data is +collected by some kind of device foreign to developer that mainly operates in +web domain. But fear not, it's not that different than writing some JavaScript. + +There are many devices able to transmit data via wireless or wired network by +default but for the sake of example we will be using commonly known Arduino with +wireless module already on the board → [Arduino +MKR1000](https://store.arduino.cc/arduino-mkr1000). + +In order to make this little project as accessible to others as possible I will +try to make it as inexpensive as possible. And by this I mean that I will avoid +using hosted virtual servers and will be using my own laptop as a server. But +you must buy Arduino MKR1000 to follow steps below. But if you would want to +deploy this software I would suggest using +[DigitalOcean](https://www.digitalocean.com) → smallest VPS is only per month +making this one of the most affordable option out there. Please notice that this +software will not run on stock web hosting that only supports LAMP (Linux, +Apache, MySQL, and PHP). + +But before we begin please take notice that this is strictly experimental code +and not well optimized and there are much better ways in handling some aspects +of the application but that requires much deeper knowledge of technology that is +not needed for an example like this. + +**Development steps** + +1. Simple Python API that will receive and store incoming data. +2. Prototype C++ code that will read "sensor data" and transmit it to API. +3. Data visualization with charts → extends Python web application. + +Step 1. and 3. will share the same web application. One route will be dedicated +to API and another to serving HTML with chart. + +Schema below represents what we will try to achieve and how different parts +correlates to each other. + +![Overview](/assets/iot-application/simple-iot-application-overview.svg) + +## Simple Python API + +I have always been a fan of simplicity so we will be using [Bottle: Python Web +Framework](https://bottlepy.org/docs/dev/). It is a single file web framework +that seriously simplifies working with routes, templating and has built-in web +server that satisfies our need in this case. + +First we need to install bottle package. This can be done by downloading +```bottle.py``` and placing it in the root of your application or by using pip +software ```pip install bottle --user```. + +If you are using Linux or MacOS then Python is already installed. If you will +try to test this on Windows please install [Python for +Windows](https://www.python.org/downloads/windows/). There may be some problems +with path when you will try to launch ```python webapp.py``` so please take care +of this before you continue. + +### Basic web application + +Most basic bottle application is quite simple. Paste code below in +```webapp.py``` file and save. + +```python +# -*- coding: utf-8 -*- + +import bottle + +# initializing bottle app +app = bottle.Bottle() + +# triggered when / is accessed from browser +# only accepts GET → no POST allowed +@app.route("/", method=["GET"]) +def route_default(): + return "howdy from python" + +# starting server on http://0.0.0.0:5000 +if __name__ == "__main__": + bottle.run( + app = app, + host = "0.0.0.0", + port = 5000, + debug = True, + reloader = True, + catchall = True, + ) +``` + +To run this simple application you should open command prompt or terminal on +your machine and go to the folder containing your file and type ```python +webapp.py```. If everything goes ok then open your web browser and point it to +```http://0.0.0.0:5000```. + +If you would like change the port of your application (like port 80) and not use +root to run your app this will present a problem. The TCP/IP port numbers below +1024 are privileged ports → this is a security feature. So in order of +simplicity and security use a port number above 1024 like I have used port 5000. + +If this fails at any time please fix it before you continue, because nothing +below will work otherwise. + +We use 0.0.0.0 as default host so that this app is available over your local +network. If you find your local ip ```ifconfig``` and try accessing this site +with your phone (if on same network/router as your machine) this should work as +well (example of such ip ```http://192.168.1.15:5000```). This is a must have +because Arduino will be accessing this application to send it's data. + +### Web application security + +There is a lot to be said about security and is a topic of many books. Of course +all this can not be written here but to just establish some basic security → you +should always use SSL with your application. Some fantastic free certificates +are available by [Let's Encrypt - Free SSL/TLS +Certificates](https://letsencrypt.org). With SSL certificate installed you +should then make use of HTTP headers and send your "API key" via a header. If +your key is send via header then this key is encrypted by SSL and send encrypted +over the network. Never send your api keys by GET parameter like +```http://example.com/?api_key=somekeyvalue```. The problem that this kind of +sending presents is that this key is visible in logs and by network sniffers. + +There is a fantastic article describing some aspects about security: [11 Web +Application Security Best +Practices](https://www.keycdn.com/blog/web-application-security-best-practices/). Please +check it out. + +### Simple API for writing data-points + +We will now be using boilerplate code from example above and extend it to be +SQLite3 because it plays well with Python and can store quite large amount of +able to write data received by API to local storage. For example use I will use +data. I have been using it to collect gigabytes of data in a single database +without any corruption or problems → your experience may vary. + +To avoid learning SQLite I will be using [Dataset: databases for lazy +people](https://dataset.readthedocs.io/en/latest/index.html). This package +abstracts SQL and simplifies writing and reading data from database. You should +install this package with pip software ```pip install dataset --user```. + +Because API will use POST method I will be testing if code works correctly by +using [Restlet Client for Google +Chrome](https://chrome.google.com/webstore/detail/restlet-client-rest-api-t/aejoelaoggembcahagimdiliamlcdmfm). +This software also allows you to set headers → for basic security with API_KEY. + +To quickly generate passwords or API keys I usually use this nifty website +[RandomKeygen](https://randomkeygen.com/). + +Copy and paste code below over your previous code in file ```webapp.py```. + +```python +# -*- coding: utf-8 -*- + +import time +import bottle +import random +import dataset + +# initializing bottle app +app = bottle.Bottle() + +# connects to sqlite database +# check_same_thread=False allows using it in multi-threaded mode +app.config["dsn"] = dataset.connect("sqlite:///data.db?check_same_thread=False") + +# api key that will be used in Arduino code +app.config["api_key"] = "JtF2aUE5SGHfVJBCG5SH" + +# triggered when /api is accessed from browser +# only accepts POST → no GET allowed +@app.route("/api", method=["POST"]) +def route_default(): + status = 400 + ts = int(time.time()) # current timestamp + value = bottle.request.body.read() # data from device + api_key = bottle.request.get_header("Api_Key") # api key from header + + # outputs to console received data for debug reason + print ">>> {} :: {}".format(value, api_key) + + # if api_key is correct and value is present + # then writes attribute to point table + if api_key == app.config["api_key"] and value: + app.config["dsn"]["point"].insert(dict(ts=ts, value=value)) + status = 200 + + # we only need to return status + return bottle.HTTPResponse(status=status, body="") + +# starting server on http://0.0.0.0:5000 +if __name__ == "__main__": + bottle.run( + app = app, + host = "0.0.0.0", + port = 5000, + debug = True, + reloader = True, + catchall = True, + ) +``` + +To run this simply go to folder containing python file and run ```python +webapp.py``` from terminal. If everything goes ok you should have simple API +available via POST method on /api route. + +After testing the service with Restlet Client you should be able to view your +data in a database file ```data.db```. + +![REST settings example](/assets/iot-application/iot-rest-example.png) + +You can also check the contents of new database file by using desktop client +for SQLite → [DB Browser for SQLite](http://sqlitebrowser.org/). + +![SQLite database example](/assets/iot-application/iot-sqlite-db.png) + +Table structure is as simple as it can be. We have ts (timestamp) and value +(value from Arduino). As you can see timestamp is generated on API side. If you +would happen to have atomic clock on Arduino it would be then better to generate +and send timestamp with the value. This would be particularity useful if we +would be collecting sensor data at a higher frequency and then sending this data +in bulk to API. + +If you will deploy this app with uWSGI and multi-threaded, use DSN (Data Source +Name) url with ```?check_same_thread=False```. + +Ok, now that we have some sort of a working API with some basic security so +unwanted people can not post data to your database can we proceed further and +try to program Arduino to send data to API. + +## Sending data to API with Arduino MKR1000 + +First of all you should have MKR1000 module and microUSB cable to proceed. If +you have ever done any work with Arduino you should know that you also need +[Arduino IDE](https://www.arduino.cc/en/Main/Software). On provided link you +should be able to download and install IDE. Once that task is completed and you +have successfully run blink example you should proceed to the next step. + +In order to use wireless capabilities of MKR1000 you need to first install +[WiFi101 library](https://www.arduino.cc/en/Reference/WiFi101) in Arduino IDE. +Please check before you install, you may already have it installed. + +Code below is a working example that sends data to API. Before you try to test +your code make sure you have run Python web application. Then change settings +for wifi, api endpoint and api_key. If by some reason code bellow doesn't work +for you please leave a comment and I'll try to help. + +Once you have opened IDE and copied this code try to compile and upload it. +Then open "Serial monitor" to see if any output is presented by Arduino. + +```c +#include + +// wifi settings +char ssid[] = "ssid-name"; +char pass[] = "ssid-password"; + +// api server enpoint +char server[] = "192.168.6.22"; +int port = 5000; + +// api key that must be the same as the one in Python code +String api_key = "JtF2aUE5SGHfVJBCG5SH"; + +// frequency data is sent in ms - every 5 seconds +int timeout = 1000 * 5; + +int status = WL_IDLE_STATUS; + +void setup() { + + // initialize serial and wait for port to open: + Serial.begin(9600); + delay(1000); + + // check for the presence of the shield + if (WiFi.status() == WL_NO_SHIELD) { + Serial.println("WiFi shield not present"); + while (true); + } + + // attempt to connect to wifi network + while (status != WL_CONNECTED) { + Serial.print("Attempting to connect to SSID: "); + Serial.println(ssid); + status = WiFi.begin(ssid, pass); + // wait 10 seconds for connection + delay(10000); + } + + // output wifi status to serial monitor + Serial.print("SSID: "); + Serial.println(WiFi.SSID()); + + IPAddress ip = WiFi.localIP(); + Serial.print("IP Address: "); + Serial.println(ip); + + long rssi = WiFi.RSSI(); + Serial.print("signal strength (RSSI):"); + Serial.print(rssi); + Serial.println(" dBm"); +} + +void loop() { + WiFiClient client; + + if (client.connect(server, port)) { + + // I use random number generator for this example + // but you can use analog or digital inputs from arduino + String content = String(random(1000)); + + client.println("POST /api HTTP/1.1"); + client.println("Connection: close"); + client.println("Api-Key: " + api_key); + client.println("Content-Length: " + String(content.length())); + client.println(); + client.println(content); + + delay(100); + client.stop(); + Serial.println("Data sent successfully ..."); + + } else { + Serial.println("Problem sending data ..."); + } + + // waits for x seconds and continue looping + delay(timeout); +} +``` + +As seen from example you can notice that Arduino is generating random integer +between [ 0 .. 1000 ]. You can easily replace this with a temperature sensor or +any other kind of sensor. + +Now that we have API under the hood and Arduino is sending demo data we can now +focus on data visualization. + +## Data visualization + +Before we continue we should examine our project folder structure. Currently we +only have two files in our project: + +_simple-iot-app/_ + +* _webapp.py_ +* _data.db_ + +We will now add HTML template that will contain CSS and JavaScript code inline +for the simplicity reason. And for the bottle framework to be able to scan root +application folder for templates we will add ```bottle.TEMPLATE_PATH.insert(0, +"./")``` in ```webapp.py```. By default bottle framework uses ```views/``` +subfolder to store templates. This is not the ideal situation and if you will +use bottle to develop web applications you should use native behavior and store +templates in it's predefined folder. But for the sake of example we will +over-ride this. Be careful to fully replace your code with new code that is +provided below. Avoid partially replacing code in file :) Also new code for +reading data-points is provided in Python example below. + +First we add new route to our web application. It should be trigger when browser +hits root of application ```http://0.0.0.0:5000/```. This route will do nothing +more than render ```frontend.html``` template. This is done by ```return +bottle.template("frontend.html")```. Check code below to further examine how +exactly this is done. + +Now we will expand ```/api``` route and use different methods to write or read +data-points. For writing data-point we will use POST method and for reading +points we will use GET method. GET method will return JSON object with latest +readings and historical data. + +There is a fantastic JavaScript library for plotting time-series charts called +[MetricsGraphics.js](https://www.metricsgraphicsjs.org) that is based on +[D3.js](https://d3js.org/) library for visualizing data. + +Data schema required by MetricsGraphics.js → to achieve this we need to +transform data from database into this format: + +```json +[ + { + "date": "2017-08-11 01:07:20", + "value": 933 + }, + { + "date": "2017-08-11 01:07:30", + "value": 743 + } +] +``` + +Web application is now complete and we only need ```frontend.html``` that we +will develop now. If you would try to start web app now and go to root app this +will return error because we don't have frontend.html yet. + +```python +# -*- coding: utf-8 -*- + +import time +import bottle +import json +import datetime +import random +import dataset + +# initializing bottle app +app = bottle.Bottle() + +# adds root directory as template folder +bottle.TEMPLATE_PATH.insert(0, "./") + +# connects to sqlite database +# check_same_thread=False allows using it in multi-threaded mode +app.config["db"] = dataset.connect("sqlite:///data.db?check_same_thread=False") + +# api key that will be used in Arduino code +app.config["api_key"] = "JtF2aUE5SGHfVJBCG5SH" + +# triggered when / is accessed from browser +# only accepts GET → no POST allowed +@app.route("/", method=["GET"]) +def route_default(): + return bottle.template("frontend.html") + +# triggered when /api is accessed from browser +# accepts POST and GET +@app.route("/api", method=["GET", "POST"]) +def route_default(): + + # if method is POST then we write datapoint + if bottle.request.method == "POST": + status = 400 + ts = int(time.time()) # current timestamp + value = bottle.request.body.read() # data from device + api_key = bottle.request.get_header("Api-Key") # api key from header + + # outputs to console recieved data for debug reason + print ">>> {} :: {}".format(value, api_key) + + # if api_key is correct and value is present + # then writes attribute to point table + if api_key == app.config["api_key"] and value: + app.config["db"]["point"].insert(dict(ts=ts, value=value)) + status = 200 + + # we only need to return status + return bottle.HTTPResponse(status=status, body="") + + # if method is GET then we read datapoint + else: + response = [] + datapoints = app.config["db"]["point"].all() + + for point in datapoints: + response.append({ + "date": datetime.datetime.fromtimestamp(int(point["ts"])).strftime("%Y-%m-%d %H:%M:%S"), + "value": point["value"] + }) + + bottle.response.content_type = "application/json" + return json.dumps(response) + +# starting server on http://0.0.0.0:5000 +if __name__ == "__main__": + bottle.run( + app = app, + host = "0.0.0.0", + port = 5000, + debug = True, + reloader = True, + catchall = True, + ) +``` + +And now finally we can implement ```frontend.html```. Create file with this name +and copy code below. When you are done you can start web application. Steps for +this part are listed below the code. + +```html + + + + + + Simple IOT application + + + + +

Simple IOT application

+ +
+
+
+ + + + + + + + + + + + + +``` + +Now the folder structure should look like: + +_simple-iot-app/_ + +* _webapp.py_ +* _data.db_ +* _frontend.html_ + +Ok, lets now start application and start feeding it data. + +1. ```python webapp.py``` +2. connect Arduino MKR1000 to power source +3. open browser and go to ```http://0.0.0.0:5000``` + +If everything goes well you should be seeing new data-points rendered on chart +every 5 seconds. + +If you navigate to ```http://0.0.0.0:5000``` you should see rendered chart as +shown on picture below. + +![Application output](/assets/iot-application/iot-app-output.png) + +Complete application with all the code is available for +[download](/assets/iot-application/simple-iot-application.zip). + +## Conclusion + +I hope this clarifies some aspects of IOT application development. Of course +this is a minimal example and is far from what can be done in real life with +some further dive into other technologies. + +If you would like to continue exploring IOT world here are some interesting +resources for you to examine: + +* [Reading Sensors with an Arduino](https://www.allaboutcircuits.com/projects/reading-sensors-with-an-arduino/) +* [MQTT 101 – How to Get Started with the lightweight IoT Protocol](http://www.hivemq.com/blog/how-to-get-started-with-mqtt) +* [Stream Updates with Server-Sent Events](https://www.html5rocks.com/en/tutorials/eventsource/basics/) +* [Internet of Things (IoT) Tutorials](http://www.tutorialspoint.com/internet_of_things/) + +Any comment or additional ideas are welcomed in comments below. diff --git a/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md b/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md new file mode 100644 index 0000000..d2fa558 --- /dev/null +++ b/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md @@ -0,0 +1,331 @@ +--- +title: Using DigitalOcean Spaces Object Storage with FUSE +url: using-digitalocean-spaces-object-storage-with-fuse.html +date: 2018-01-16T12:00:00+02:00 +type: post +draft: false +--- + +Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced new +product called +[Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) which +is Object Storage very similar to Amazon's S3. This really peaked my interest, +because this was something I was missing and even the thought of going over the +internet for such functionality was in no interest to me. Also in fashion with +their previous pricing this also is very cheap and pricing page is a no-brainer +compared to AWS or GCE. [Prices are clearly and precisely defined and +outlined](https://www.digitalocean.com/pricing/). You must love them for that +:) + +## Initial requirements + +* Is it possible to use them as a mounted drive with FUSE? (tl;dr YES) +* Will the performance degrade over time and over different sizes of objects? + (tl;dr NO&YES) +* Can storage be mounted on multiple machines at the same time and be writable? + (tl;dr YES) + +> Let me be clear. This scripts I use are made just for benchmarking and are not +> intended to be used in real-life situations. Besides that, I am looking into +> using this approaches but adding caching service in front of it and then +> dumping everything as an object to storage. This could potentially be some +> interesting post of itself. But in case you would need real-time data without +> eventual consistency please take this scripts as they are: not usable in such +> situations. + +## Is it possible to use them as a mounted drive with FUSE? + +Well, actually they can be used in such manor. Because they are similar to [AWS +S3](https://aws.amazon.com/s3/) many tools are available and you can find many +articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse). + +To make this work you will need DigitalOcean account. If you don't have one you +will not be able to test this code. But if you have an account then you go and +[create new +Droplet](https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb®ion=ams3&distro=debian&distroImage=debian-9-x64&options=private_networking,install_agent). +If you click on this link you will already have preselected Debian 9 with +smallest VM option. + +* Please be sure to add you SSH key, because we will login to this machine + remotely. +* If you change your region please remember which one you choose because we will + need this information when we try to mount space to our machine. + +Instuctions on how to use SSH keys and how to setup them are available in +article [How To Use SSH Keys with DigitalOcean +Droplets](https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets). + +![DigitalOcean Droplets](/assets/do-fuse/fuse-droplets.png) + +After we created Droplet it's time to create new Space. This is done by clicking +on a button [Create](https://cloud.digitalocean.com/spaces/new) (right top +corner) and selecting Spaces. Choose pronounceable ```Unique name``` because we +will use it in examples below. You can either choose Private or Public, it +doesn't matter in our case. And you can always change that in the future. + +When you have created new Space we should [generate Access +key](https://cloud.digitalocean.com/settings/api/tokens). This link will guide +to the page when you can generate this key. After you create new one, please +save provided Key and Secret because Secret will not be shown again. + +![DigitalOcean Spaces](/assets/do-fuse/fuse-spaces.png) + +Now that we have new Space and Access key we should SSH into our machine. + +```bash +# replace IP with the ip of your newly created droplet +ssh root@IP + +# this will install utilities for mounting storage objects as FUSE +apt install s3fs + +# we now need to provide credentials (access key we created earlier) +# replace KEY and SECRET with your own credentials but leave the colon between them +# we also need to set proper permissions +echo "KEY:SECRET" > .passwd-s3fs +chmod 600 .passwd-s3fs + +# now we mount space to our machine +# replace UNIQUE-NAME with the name you choose earlier +# if you choose different region for your space be careful about -ourl option (ams3) +s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp + +# now we try to create a file +# once you mount it may take a couple of seconds to retrieve data +echo "Hello cruel world" > /mnt/hello.txt +``` + +After all this you can return to your browser and go to [DigitalOcean +Spaces](https://cloud.digitalocean.com/spaces) and click on your created +space. If file hello.txt is present you have successfully mounted space to your +machine and wrote data to it. + +I choose the same region for my Droplet and my Space but you don't have to. You +can have different regions. What this actually does to performance I don't know. + +Additional information on FUSE: + +* [Github project page for s3fs](https://github.com/s3fs-fuse/s3fs-fuse) +* [FUSE - Filesystem in Userspace](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) + +## Will the performance degrade over time and over different sizes of objects? + +For this task I didn't want to just read and write text files or uploading +images. I actually wanted to figure out if using something like SQlite is viable +in this case. + +### Measurement experiment 1: File copy + +```bash +# first we create some dummy files at different sizes +dd if=/dev/zero of=10KB.dat bs=1024 count=10 #10KB +dd if=/dev/zero of=100KB.dat bs=1024 count=100 #100KB +dd if=/dev/zero of=1MB.dat bs=1024 count=1024 #1MB +dd if=/dev/zero of=10MB.dat bs=1024 count=10240 #10MB + +# now we set time command to only return real +TIMEFORMAT=%R + +# now lets test it +(time cp 10KB.dat /mnt/) |& tee -a 10KB.results.txt + +# and now we automate +# this will perform the same operation 100 times +# this will output results into separated files based on objecty size +n=0; while (( n++ < 100 )); do (time cp 10KB.dat /mnt/10KB.$n.dat) |& tee -a 10KB.results.txt; done +n=0; while (( n++ < 100 )); do (time cp 100KB.dat /mnt/100KB.$n.dat) |& tee -a 100KB.results.txt; done +n=0; while (( n++ < 100 )); do (time cp 1MB.dat /mnt/1MB.$n.dat) |& tee -a 1MB.results.txt; done +n=0; while (( n++ < 100 )); do (time cp 10MB.dat /mnt/10MB.$n.dat) |& tee -a 10MB.results.txt; done +``` + +Files of size 100MB were not successfully transferred and ended up displaying +error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted). + +As I suspected, object size is not really that important. Sadly I don't have the +time to test performance over periods of time. But if some of you would do it +please send me your data. I would be interested in seeing results. + +**Here are plotted results** + +You can download [raw result here](/assets/do-fuse/copy-benchmarks.tsv). +Measurements are in seconds. + + +
+ + +As far as these tests show, performance is quite stable and can be predicted +which is fantastic. But this is a small test and spans only over couple of +hours. So you should not completely trust them. + +### Measurement experiment 2: SQLite performanse + +I was unable to use database file directly from mounted drive so this is a no-go +as I suspected. So I executed code below on a local disk just to get some +benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY, +FETCHALL, COMMIT for 1000 times to generate statistics. As you can see +performance of SQLite is quite amazing. You could then potentially just copy +file to mounted drive and be done with it. + +```python +import time +import sqlite3 +import sys + +if len(sys.argv) < 3: + print("usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT") + exit() + +def data_iter(x): + for i in range(x): + yield "m" + str(i), "f" + str(i*i) + +header_line = "%s\t%s\t%s\t%s\t%s\n" % ("DROPTABLE", "CREATETABLE", "INSERTMANY", "FETCHALL", "COMMIT") +with open("sqlite-benchmarks.tsv", "w") as fp: + fp.write(header_line) + +start_time = time.time() +conn = sqlite3.connect(sys.argv[1]) +c = conn.cursor() +end_time = time.time() +result_time = CONNECT = end_time - start_time +print("CONNECT: %g seconds" % (result_time)) + +start_time = time.time() +c.execute("PRAGMA journal_mode=WAL") +c.execute("PRAGMA temp_store=MEMORY") +c.execute("PRAGMA synchronous=OFF") +result_time = PRAGMA = end_time - start_time +print("PRAGMA: %g seconds" % (result_time)) + +for i in range(int(sys.argv[3])): + print("#%i" % (i)) + + start_time = time.time() + c.execute("drop table if exists test") + end_time = time.time() + result_time = DROPTABLE = end_time - start_time + print("DROPTABLE: %g seconds" % (result_time)) + + start_time = time.time() + c.execute("create table if not exists test(a,b)") + end_time = time.time() + result_time = CREATETABLE = end_time - start_time + print("CREATETABLE: %g seconds" % (result_time)) + + start_time = time.time() + c.executemany("INSERT INTO test VALUES (?, ?)", data_iter(int(sys.argv[2]))) + end_time = time.time() + result_time = INSERTMANY = end_time - start_time + print("INSERTMANY: %g seconds" % (result_time)) + + start_time = time.time() + c.execute("select count(*) from test") + res = c.fetchall() + end_time = time.time() + result_time = FETCHALL = end_time - start_time + print("FETCHALL: %g seconds" % (result_time)) + + start_time = time.time() + conn.commit() + end_time = time.time() + result_time = COMMIT = end_time - start_time + print("COMMIT: %g seconds" % (result_time)) + + print + log_line = "%f\t%f\t%f\t%f\t%f\n" % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT) + with open("sqlite-benchmarks.tsv", "a") as fp: + fp.write(log_line) + +start_time = time.time() +conn.close() +end_time = time.time() +result_time = CLOSE = end_time - start_time +print("CLOSE: %g seconds" % (result_time)) +``` + +You can download [raw result here](/assets/do-fuse/sqlite-benchmarks.tsv). And +again, these results are done on a local block storage and do not represent +capabilities of object storage. With my current approach and state of the test +code these can not be done. I would need to make Python code much more robust +and check locking etc. + +
+ + +## Can storage be mounted on multiple machines at the same time and be writable? + +Well, this one didn't take long to test. And the answer is **YES**. I mounted +space on both machines and measured same performance on both machines. But +because file is downloaded before write and then uploaded on complete there +could potentially be problems is another process is trying to access the same +file. + +## Observations and conslusion + +Using Spaces in this way makes it easier to access and manage files. But besides +that you would need to write additional code to make this one play nice with you +applications. + +Nevertheless, this was extremely simple to setup and use and this is just +another excellent product in DigitalOcean product line. I found this exercise +very valuable and am thinking about implementing some sort of mechanism for +SQLite, so data can be stored on Spaces and accessed by many VM's. For a project +where data doesn't need to be accessible in real-time and can have couple of +minutes old data this would be very interesting. If any of you find this +proposal interesting please write in a comment box below or shoot me an email +and I will keep you posted. diff --git a/content/posts/2019-01-03-encoding-binary-data-into-dna-sequence.md b/content/posts/2019-01-03-encoding-binary-data-into-dna-sequence.md new file mode 100644 index 0000000..b285756 --- /dev/null +++ b/content/posts/2019-01-03-encoding-binary-data-into-dna-sequence.md @@ -0,0 +1,411 @@ +--- +title: Encoding binary data into DNA sequence +url: encoding-binary-data-into-dna-sequence.html +date: 2019-01-03T12:00:00+02:00 +type: post +draft: false +--- + +## Initial thoughts + +Imagine a world where you could go outside and take a leaf from a tree and put +it through your personal DNA sequencer and get data like music, videos or +computer programs from it. Well, this is all possible now. It was not done on a +large scale because it is quite expensive to create DNA strands but it's +possible. + +Encoding data into DNA sequence is relatively simple process once you understand +the relationship between binary data and nucleotides and scientists have been +making large leaps in this field in order to provide viable long-term storage +solution for our data that would potentially survive our specie if case of +global disaster. We could imprint all the world's knowledge into plants and +ensure the survival of our knowledge. + +More optimistic usage for this technology would be easier storage of ever +growing data we produce every day. Once machines for sequencing DNA become fast +enough and cheaper this could mean the next evolution of storing data and +abandoning classical hard and solid state drives in data warehouses. + +As we currently stand this is still not viable but it is quite an amazing and +cool technology. + +My interests in this field are purely in encoding processes and experimental +testing mainly because I don't have the access to this expensive machines. My +initial goal was to create a toolkit that can be used by everybody to encode +their data into a proper DNA sequence. + +## Glossary + +**deoxyribose** A five-carbon sugar molecule with a hydrogen atom rather than a +hydroxyl group in the 2′ position; the sugar component of DNA nucleotides. + +**double helix** The molecular shape of DNA in which two strands of nucleotides +wind around each other in a spiral shape. + +**nitrogenous base** A nitrogen-containing molecule that acts as a base; often +referring to one of the purine or pyrimidine components of nucleic acids. + +**phosphate group** A molecular group consisting of a central phosphorus atom +bound to four oxygen atoms. + +**RGB** The RGB color model is an additive color model in which red, green and +blue light are added together in various ways to reproduce a broad array of +colors. + +**GCC** The GNU Compiler Collection is a compiler system produced by the GNU +Project supporting various programming languages. + +## Data encoding + +**TL;DR:** Encoding involves the use of a code to change original data into a +form that can be used by an external process. + +Encoding is the process of converting data into a format required for a number +of information processing needs, including: + +- Program compiling and execution +- Data transmission, storage and compression/decompression +- Application data processing, such as file conversion + +Encoding can have two meanings: + +- In computer technology, encoding is the process of applying a specific code, + such as letters, symbols and numbers, to data for conversion into an + equivalent cipher. +- In electronics, encoding refers to analog to digital conversion. + +## Quick history of DNA + +- **1869** - Friedrich Miescher identifies "nuclein". +- **1900s** - The Eugenics Movement. +- **1900** – Mendel's theories are rediscovered by researchers. +- **1944** - Oswald Avery identifies DNA as the 'transforming principle'. +- **1952** - Rosalind Franklin photographs crystallized DNA fibres. +- **1953** - James Watson and Francis Crick discover the double helix structure of DNA. +- **1965** - Marshall Nirenberg is the first person to sequence the bases in each codon. +- **1983** - Huntington's disease is the first mapped genetic disease. +- **1990** - The Human Genome Project begins. +- **1995** - Haemophilus Influenzae is the first bacterium genome sequenced. +- **1996** - Dolly the sheep is cloned. +- **1999** - First human chromosome is decoded. +- **2000** – Genetic code of the fruit fly is decoded. +- **2002** – Mouse is the first mammal to have its genome decoded. +- **2003** – The Human Genome Project is completed. +- **2013** – DNA Worldwide and Eurofins Forensic discover identical twins have differences in their genetic makeup. + +## What is DNA? + +Deoxyribonucleic acid, a self-replicating material which is **present in nearly +all living organisms** as the main constituent of chromosomes. It is the +**carrier of genetic information**. + +> The nitrogen in our DNA, the calcium in our teeth, the iron in our blood, +> the carbon in our apple pies were made in the interiors of collapsing stars. +> We are made of starstuff. +> **-- Carl Sagan, Cosmos** + +The nucleotide in DNA consists of a sugar (deoxyribose), one of four bases +(cytosine (C), thymine (T), adenine (A), guanine (G)), and a phosphate. +Cytosine and thymine are pyrimidine bases, while adenine and guanine are purine +bases. The sugar and the base together are called a nucleoside. + +![DNA](/assets/dna-sequence/dna-basics.jpg) + +*DNA (a) forms a double stranded helix, and (b) adenine pairs with thymine and +cytosine pairs with guanine. (credit a: modification of work by Jerome Walker, +Dennis Myts)* + +## Encode binary data into DNA sequence + +As an input file you can use any file you want: + +- ASCII files, +- Compiled programs, +- Multimedia files (MP3, MP4, MVK, etc), +- Images, +- Database files, +- etc. + +Note: If you would copy all the bytes from RAM to file or pipe data to file you +could encode also this data as long as you provide file pointer to the encoder. + +### Basic Encoding + +As already mentioned, the Basic Encoding is based on a simple mapping. Since DNA +is composed of 4 nucleotides (Adenine, Cytosine, Guanine, Thymine; usually +referred using the first letter). Using this technique we can encode + +$$ log_2(4) = log_2(2^2) = 2 bits $$ + +using a single nucleotide. In this way, we are able to use the 4 bases that +compose the DNA strand to encode each byte of data. + +| Two bits | Nucleotides | +| -------- | ---------------- | +| 00 | **A** (Adenine) | +| 10 | **G** (Guanine) | +| 01 | **C** (Cytosine) | +| 11 | **T** (Thymine) | + +With this in mind we can simply encode any data by using two-bit to Nucleotides +conversion. + +```python +{ Algorithm 1: Naive byte array to DNA encode } +procedure EncodeToDNASequence(f) string +begin + enc string + while not eof(f) do + c byte := buffer[0] { Read 1 byte from buffer } + bin integer := sprintf('08b', c) { Convert to string binary } + for e in range[0, 2, 4, 6] do + if e[0] == 48 and e[1] == 48 then { 0x00 - A (Adenine) } + enc += 'A' + else if e[0] == 48 and e[1] == 49 then { 0x01 - G (Guanine) } + enc += 'G' + else if e[0] == 49 and e[1] == 48 then { 0x10 - C (Cytosine) } + enc += 'C' + else if e[0] == 49 and e[1] == 49 then { 0x11 - T (Thymine) } + enc += 'T' + return enc { Return DNA sequence } +end +``` + +Another encoding would be **Goldman encoding**. Using this encoding helps with +Nonsense mutation (amino acids replaced by a stop codon) that occurs and is the +most problematic during translation because it leads to truncated amino acid +sequences, which in turn results in truncated proteins. + +[Where to store big data? In DNA: Nick Goldman at TEDxPrague](https://www.youtube.com/watch?v=a4PiGWNsIEU) + +### FASTA file format + +In bioinformatics, FASTA format is a text-based format for representing either +nucleotide sequences or peptide sequences, in which nucleotides or amino acids +are represented using single-letter codes. The format also allows for sequence +names and comments to precede the sequences. The format originates from the +FASTA software package, but has now become a standard in the field of +bioinformatics. + +The first line in a FASTA file started either with a ">" (greater-than) symbol +or, less frequently, a ";" (semicolon) was taken as a comment. Subsequent lines +starting with a semicolon would be ignored by software. Since the only comment +used was the first, it quickly became used to hold a summary description of the +sequence, often starting with a unique library accession number, and with time +it has become commonplace to always use ">" for the first line and to not use +";" comments (which would otherwise be ignored). + +``` +;LCBO - Prolactin precursor - Bovine +; a sample sequence in FASTA format +MDSKGSSQKGSRLLLLLVVSNLLLCQGVVSTPVCPNGPGNCQVSLRDLFDRAVMVSHYIHDLSS +EMFNEFDKRYAQGKGFITMALNSCHTSSLPTPEDKEQAQQTHHEVLMSLILGLLRSWNDPLYHL +VTEVRGMKGAPDAILSRAIEIEEENKRLLEGMEMIFGQVIPGAKETEPYPVWSGLPSLQTKDED +ARYSAFYNLLHCLRRDSSKIDTYLKLLNCRIIYNNNC* + +>MCHU - Calmodulin - Human, rabbit, bovine, rat, and chicken +ADQLTEEQIAEFKEAFSLFDKDGDGTITTKELGTVMRSLGQNPTEAELQDMINEVDADGNGTID +FPEFLTMMARKMKDTDSEEEIREAFRVFDKDGNGYISAAELRHVMTNLGEKLTDEEVDEMIREA +DIDGDGQVNYEEFVQMMTAK* + +>gi|5524211|gb|AAD44166.1| cytochrome b [Elephas maximus maximus] +LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV +EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG +LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL +GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX +IENY +``` + +FASTA format was extended by [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) +format from the [Sanger Centre](https://www.sanger.ac.uk/) in Cambridge. + +### PNG encoded DNA sequence + +| Nucleotides | RGB | Color name | +| ------------ | ----------- | ---------- | +| A ➞ Adenine | (0,0,255) | Blue | +| G ➞ Guanine | (0,100,0) | Green | +| C ➞ Cytosine | (255,0,0) | Red | +| T ➞ Thymine | (255,255,0) | Yellow | + +With this in mind we can create a simple algorithm to create PNG representation +of a DNA sequence. + +```python +{ Algorithm 2: Naive DNA to PNG encode from FASTA file } +procedure EncodeDNASequenceToPNG(f) +begin + i image + while not eof(f) do + c char := buffer[0] { Read 1 char from buffer } + case c of + 'A': color := RGB(0, 0, 255) { Blue } + 'G': color := RGB(0, 100, 0) { Green } + 'C': color := RGB(255, 0, 0) { Red } + 'T': color := RGB(255, 255, 0) { Yellow } + drawRect(i, [x, y], color) + save(i) { Save PNG image } +end +``` + +## Encoding text file in practice + +In this example we will take a simple text file as our input stream for +encoding. This file will have a quote from Niels Bohr and saved as txt file. + +> How wonderful that we have met with a paradox. Now we have some hope of +> making progress. +> ― Niels Bohr + +First we encode text file into FASTA file. + +```bash +./dnae-encode -i quote.txt -o quote.fa +2019/01/10 00:38:29 Gathering input file stats +2019/01/10 00:38:29 Starting encoding ... + 106 B / 106 B [==================================] 100.00% 0s +2019/01/10 00:38:29 Saving to FASTA file ... +2019/01/10 00:38:29 Output FASTA file length is 438 B +2019/01/10 00:38:29 Process took 987.263µs +2019/01/10 00:38:29 Done ... +``` + +Output of `quote.fa` file contains the encoded DNA sequence in ASCII format. + +``` +>SEQ1 +GACAGCTTGTGTACAAGTGTGCTTGCTCGCGAGCGGGTACGCGCGTGGGCTAACAAGTGA +GCCAGCAGGTGAACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGCTGGCGGGTGA +ACAAGTGTGCCGGTGAGCCAACAAGCAGACAAGTAAGCAGGTACGCAGGCGAGCTTGTCA +ACTCACAAGATCGCTTGTGTACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGTAT +GCTTGCTGGCGGACAAGCCAGCTTGTAAGCGGACAAGCTTGCGCACAAGCTGGCAGGCCT +GCCGGCTCGCGTACAAATTCACAAGTAAGTACGCTTGCGTGTACGCGGGTATGTATACTC +AACCTCACCAAACGGGACAAGATCGCCGGCGGGCTAGTATACAAGAACGCTTGCCAGTAC +AACC +``` + +Then we encode FASTA file from previous operation to encode this data into PNG. + +```bash +./dnae-png -i quote.fa -o quote.png +2019/01/10 00:40:09 Gathering input file stats ... +2019/01/10 00:40:09 Deconstructing FASTA file ... +2019/01/10 00:40:09 Compositing image file ... + 424 / 424 [==================================] 100.00% 0s +2019/01/10 00:40:09 Saving output file ... +2019/01/10 00:40:09 Output image file length is 1.1 kB +2019/01/10 00:40:09 Process took 19.036117ms +2019/01/10 00:40:09 Done ... +``` + +After encoding into PNG format this file looks like this. + +![Encoded Quote in PNG format](/assets/dna-sequence/quote.png) + +The larger the input stream is the larger the PNG file would be. + +Compiled basic Hello World C program with +[GCC](https://www.gnu.org/software/gcc/) would [look +like](/assets/dna-sequence/sample.png). + +```c +// gcc -O3 -o sample sample.c +#include + +main() { + printf("Hello, world!\n"); + return 0; +} +``` + +## Toolkit for encoding data + +I have created a toolkit with two main programs: + +- dnae-encode (encodes file into FASTA file) +- dnae-png (encodes FASTA file into PNG) + +Toolkit with full source code is available on +[github.com/mitjafelicijan/dna-encoding](https://github.com/mitjafelicijan/dna-encoding). + +### dnae-encode + +```bash +> ./dnae-encode --help +usage: dnae-encode --input=INPUT [] + +A command-line application that encodes file into DNA sequence. + +Flags: + --help Show context-sensitive help (also try --help-long and --help-man). + -i, --input=INPUT Input file (ASCII or binary) which will be encoded into DNA sequence. + -o, --output="out.fa" Output file which stores DNA sequence in FASTA format. + -s, --sequence=SEQ1 The description line (defline) or header/identifier line, gives a name and/or a unique identifier for the sequence. + -c, --columns=60 Row characters length (no more than 120 characters). Devices preallocate fixed line sizes in software. + --version Show application version. +``` + +### dnae-png + +```bash +> ./dnae-png --help +usage: dnae-png --input=INPUT [] + +A command-line application that encodes FASTA file into PNG image. + +Flags: + --help Show context-sensitive help (also try --help-long and --help-man). + -i, --input=INPUT Input FASTA file which will be encoded into PNG image. + -o, --output="out.png" Output file in PNG format that represents DNA sequence in graphical way. + -s, --size=10 Size of pairings of DNA bases on image in pixels (lower resolution lower file size). + --version Show application version. +``` + +## Benchmarks + +First we generate some binary sample data with dd. + +```bash +dd if=<(openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) of=1KB.bin bs=1KB count=1 iflag=fullblock +``` + +Our freshly generated 1KB file looks something like this (its full of garbage +data as intended). + +![Sample binary file 1KB](/assets/dna-sequence/sample-binary-file.png) + +We create following binary files: + +- 1KB.bin +- 10KB.bin +- 100KB.bin +- 1MB.bin +- 10MB.bin +- 100MB.bin + +After this we create FASTA files for all the binary files by encoding them +into DNA sequence. + +```bash +./dnae-encode -i 100MB.bin -o 100MB.fa +``` + +Then we GZIP all the FASTA files to see how much the can be compressed. + +```bash +gzip -9 < 10MB.fa > 10MB.fa.gz +``` + +[Download ODS file with benchmarks](/dna-sequence/benchmarks.ods). + +![Sample binary file 1KB](/assets/dna-sequence/chart-1.png) + +![Sample binary file 1KB](/assets/dna-sequence/chart-2.png) + +## References + +- https://www.techopedia.com/definition/948/encoding +- https://www.dna-worldwide.com/resource/160/history-dna-timeline +- https://opentextbc.ca/biology/chapter/9-1-the-structure-of-dna/ +- https://arxiv.org/abs/1801.04774 +- https://en.wikipedia.org/wiki/FASTA_format diff --git a/content/posts/2019-10-14-simplifying-and-reducing-clutter.md b/content/posts/2019-10-14-simplifying-and-reducing-clutter.md new file mode 100644 index 0000000..25f9ca0 --- /dev/null +++ b/content/posts/2019-10-14-simplifying-and-reducing-clutter.md @@ -0,0 +1,59 @@ +--- +title: Simplifying and reducing clutter in my life and work +url: simplifying-and-reducing-clutter.html +date: 2019-10-14T12:00:00+02:00 +type: post +draft: false +--- + +I recently moved my main working machine back from Hachintosh to Linux. Well the +experiment was interesting and I have done some great work on macOS but it was +time to move back. + +I actually really missed Linux. The simplicity of `apt-get` or just the amount +of software that exists for Linux should be a no-brainer. I spent most of my +time on macOS finding solutions to make things work. Using +[Brew](https://brew.sh/) was just a horrible experience and far from package +managers of Linux. At least they managed to get that `sudo` debacle sorted. + +Not all was bad. macOS in general was a perfectly good environment. Things like +Docker and tooling like this worked without any hiccups. My normal tools like +coding IDE worked flawlessly and the whole look and feel is just superb. I have +been using MacBook Air for couple of years so I was used to the system but never +as a daily driver. + +One of the things I did after I installed Linux back on my machine was cleaning +up my Dropbox folder. I have everything on Dropbox. Even projects folder. I +write code for living so my whole life revolves around couple of megs of code +(with assets). So it's not like I have huge files on my machine. I don't have +movies or music or pictures on my PC. All of that stuff is in cloud. I use +Google music and I have Netflix account which is more than enough for me. + +I also went and deleted some of the repositories on my Github account. I have +deleted more code than deployed. People find this strange but for me deleting +something feels so cathartic and also forces me to write better code next time +around when I am faced with similar problem. That was a huge relief if I am +being totally honest. + +Next step was to do something with my webpage. I have been using some scripts I +wrote a while ago to generate static pages from markdown source posts. I kept on +adding and adding stuff on top of it and it became a source of a +frustration. And this is just a simple blog and I was using gulp and npm. +Anyways after couple of hours of searching and testing static generators I found +an interesting one +[https://github.com/piranha/gostatic](https://github.com/piranha/gostatic) and I +just decided to use this one. It was the only one that had a simple templating +engine, not that I really need one. But others had this convoluted way of trying +to solve everything and at the end just required quite bigger learning curve I +was ready to go with. So I deleted couple of old posts, simplified HTML, trashed +most of the CSS and went with +[https://motherfuckingwebsite.com/](https://motherfuckingwebsite.com/) +aesthetics. Yeah, the previous site was more visually stimulating but all I +really care is the content at this point. And Times New Roman font is kind of +awesome. + +I stopped working on most of the projects in the past couple of months because +the overhead was just too insane. There comes a point when you stretch yourself +too much and then you stop progressing and with that comes dissatisfaction. + +So that's about it. Moving forward minimal style. diff --git a/content/posts/2019-10-19-using-sentiment-analysis-for-clickbait-detection.md b/content/posts/2019-10-19-using-sentiment-analysis-for-clickbait-detection.md new file mode 100644 index 0000000..8322e70 --- /dev/null +++ b/content/posts/2019-10-19-using-sentiment-analysis-for-clickbait-detection.md @@ -0,0 +1,108 @@ +--- +title: Using sentiment analysis for clickbait detection in RSS feeds +url: using-sentiment-analysis-for-clickbait-detection-in-rss-feeds.html +date: 2019-10-19T12:00:00+02:00 +type: post +draft: false +--- + +## Initial thoughts + +One of the things that interested me for a while now is if major well +established news sites use click bait titles to drive additional traffic to +their sites and generate additional impressions. + +Goal is to see how article titles and actual content of article differ from each +other and see if titles are clickbaited. + +## Preparing and cleaning data + +For this example I opted to just use RSS feed from a new website and decided to +go with [The Guardian](https://www.theguardian.com) World news. While this gets +us limited data (~40) articles and also description (actual content) is trimmed +this really doesn't reflect the actual article contents. + +To get better content I could use web scraping and use RSS as link list and +fetch contents directly from website, but for this simple example this will +suffice. + +There are couple of requirements we need to install before we continue: + +- `pip3 install feedparser` (parses RSS feed from url) +- `pip3 install vaderSentiment` (does sentiment polarity analysis) +- `pip3 install matplotlib` (plots chart of results) + +So first we need to fetch RSS data and sanitize HTML content from description. + +```python +import re +import feedparser + +feed_url = "https://www.theguardian.com/world/rss" +feed = feedparser.parse(feed_url) + +# sanitize html +for item in feed.entries: + item.description = re.sub('<[^<]+?>', '', item.description) +``` + +## Perform sentiment analysis + +Since we now have cleaned up data in our `feed.entries` object we can start with +performing sentiment analysis. + +There are many sentiment analysis libraries available that range from rule-based +sentiment analysis up to machine learning supported analysis. To keep things +simple I decided to use rule-based analysis library +[vaderSentiment](https://github.com/cjhutto/vaderSentiment) from +[C.J. Hutto](https://github.com/cjhutto). Really nice library and quite easy to +use. + +```python +from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer +analyser = SentimentIntensityAnalyzer() + +sentiment_results = [] +for item in feed.entries: + sentiment_title = analyser.polarity_scores(item.title) + sentiment_description = analyser.polarity_scores(item.description) + sentiment_results.append([sentiment_title['compound'], sentiment_description['compound']]) +``` + +Now that we have this data in a shape that is compatible with matplotlib we can +plot results to see the difference between title and description sentiment of an +article. + +```python +import matplotlib.pyplot as plt + +plt.rcParams['figure.figsize'] = (15, 3) +plt.plot(sentiment_results, drawstyle='steps') +plt.title('Sentiment analysis relationship between title and description (Guardian World News)') +plt.legend(['title', 'description']) +plt.show() +``` + +## Results and assets + +1. Because of the small sample size further conclusions are impossible to make. +2. Rule-based approach may not be the best way of doing this. By using deep + learning we would be able to get better insights. +3. **Next step would be to** periodically fetch RSS items and store them over a + longer period of time and then perform analysis again and use either machine + learning or deep learning on top of it. + +![Relationship between title and description](/assets/sentiment-analysis/guardian-sa-title-desc-relationship.png) + +Figure above displays difference between title and description sentiment for +specific RSS feed item. 1 means positive and -1 means negative sentiment. + +[» Download Jupyter Notebook](/assets/sentiment-analysis/sentiment-analysis.ipynb) + +## Going further + +- [Twitter Sentiment Analysis by Bryan Schwierzke](https://github.com/bswiss/news_mood) +- [AFINN-based sentiment analysis for Node.js by Andrew Sliwinski](https://github.com/thisandagain/sentiment) +- [Sentiment Analysis with LSTMs in Tensorflow by Adit Deshpande](https://github.com/adeshpande3/LSTM-Sentiment-Analysis) +- [Sentiment analysis on tweets using Naive Bayes, SVM, CNN, LSTM, etc. by Abdul Fatir](https://github.com/abdulfatir/twitter-sentiment-analysis) + diff --git a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md new file mode 100644 index 0000000..8e46138 --- /dev/null +++ b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md @@ -0,0 +1,454 @@ +--- +title: Simple Server-Sent Events based PubSub Server +url: simple-server-sent-events-based-pubsub-server.html +date: 2020-03-22T12:00:00+02:00 +type: post +draft: false +--- + +## Before we continue ... + +Publisher Subscriber model is nothing new and there are many amazing solutions +out there, so writing a new one would be a waste of time if other solutions +wouldn't have quite complex install procedures and weren't so hard to maintain. +But to be fair, comparing this simple server with something like +[Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is +laughable at the least. Those solutions are enterprise grade and have many +mechanisms there to ensure messages aren't lost and much more. Regardless of +these drawbacks, this method has been tested on a large website and worked until +now without any problems. So now, that we got that cleared up, let's continue. + +***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a +form of asynchronous service-to-service communication used in serverless and +microservices architectures. In a pub/sub model, any message published to a +topic is immediately received by all the subscribers to the topic.* + +## General goals + +- provide a simple server that relays messages to all the connected clients, +- messages can be posted on specific topics, +- messages get sent via [Server-Sent + Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) + to all the subscribers. + +## How exactly does the pub/sub model work? + +The easiest way to explain this is with diagram bellow. Basic function is +simple. We have subscribers that receive messages, and we have publishers that +create and post messages. Similar model is also well know pattern that works on +a premise of consumers and producers, and they take similar roles. + +![How PubSub works](/assets/simple-pubsub-server/pubsub-overview.png) + +**These are some naive characteristics we want to achieve:** + +- producer is publishing messages to subscribe topic, +- consumer is receiving messages from subscribed topic, +- servers is also known as Broker, +- broker does not store messages or tracks success, +- broker uses + [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) method + for delivering messages, +- if consumer wants to receive messages from a topic, producer and consumer + topics must match, +- consumer can subscribe to multiple topics, +- producer can publish to multiple topics, +- each message has a messageId. + +**Known drawbacks:** + +- messages will not be stored in a persistent queue or unreceived messages like + [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old + messages could be lost on server restart, +- [Server-Sent + Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) + opens a long-running connection between the client and the server so make sure + if your setup is load balanced that the load balancer in this case can have + long opened connection, +- no system moderation due to the dynamic nature of creating queues. + +## Server-Sent Events + +Read more about it on [official specification +page](https://html.spec.whatwg.org/multipage/server-sent-events.html). + +### Current browser support + +![Browser support](/assets/simple-pubsub-server/caniuse.png) + +Check +[https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) +for latest information about browser support. + +### Known issues + +- Firefox 52 and below do not support EventSource in web/shared workers +- In Firefox prior to version 36 server-sent events do not reconnect + automatically in case of a connection interrupt (bug) +- Reportedly, CORS in EventSource is currently supported in Firefox 10+, Opera + 12+, Chrome 26+, Safari 7.0+. +- Antivirus software may block the event streaming data chunks. + +Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) + +### Message format + +The simplest message that can be sent is only with data attribute: + +```bash +data: this is a simple message + +``` + +You can send message IDs to be used if the connection is dropped: + +```bash +id: 33 +data: this is line one +data: this is line two + +``` + +And you can specify your own event types (the above messages will all trigger +the message event): + +```bash +id: 36 +event: price +data: 103.34 + +``` + +### Server requirements + +The important thing is how you send headers and which headers are sent by the +server that triggers browser to threat response as a EventStream. + +Headers responsible for this are: + +```bash +Content-Type: text/event-stream +Cache-Control: no-cache +Connection: keep-alive +``` + +### Debugging with Google Chrome + +Google Chrome provides build-in debugging and exploration tool for [Server-Sent +Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) +which is quite nice and available from Developer Tools under Network tab. + +> You can debug only client side events that get received and not the server +> ones. For debugging server events add `console.log` to `server.js` code and +> print out events. + +![Google Chrome Developer Tools EventStream](/assets/simple-pubsub-server/chrome-debugging.png) + +## Server implementation + +For the sake of this example we will use [Node.js](https://nodejs.org/en/) with +[Express](https://expressjs.com) as our router since this is the easiest way to +get started and we will use already written SSE library for node +[sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent the +wheel. + +```bash +npm init --yes + +npm install express +npm install body-parser +npm install sse-pubsub +``` + +Basic implementation of a server (`server.js`): + +```js +const express = require('express'); +const bodyParser = require('body-parser'); +const SSETopic = require('sse-pubsub'); + +const app = express(); +const port = process.env.PORT || 4000; + +// topics container +const sseTopics = {}; + +app.use(bodyParser.json()); + +// open for all cors +app.all('*', (req, res, next) => { + res.header('Access-Control-Allow-Origin', '*'); + res.header('Access-Control-Allow-Headers', 'X-Requested-With, Content-Type'); + next(); +}); + +// preflight request error fix +app.options('*', async (req, res) => { + res.header('Access-Control-Allow-Origin', '*'); + res.header('Access-Control-Allow-Headers', 'X-Requested-With, Content-Type'); + res.send('OK'); +}); + +// serve the event streams +app.get('/stream/:topic', async (req, res, next) => { + const topic = req.params.topic; + + if (!(topic in sseTopics)) { + sseTopics[topic] = new SSETopic({ + pingInterval: 0, + maxStreamDuration: 15000, + }); + } + + // subscribing client to topic + sseTopics[topic].subscribe(req, res); +}); + +// accepts new messages into topic +app.post('/publish', async (req, res) => { + let body = req.body; + let status = 200; + + console.log('Incoming message:', req.body); + + if ( + body.hasOwnProperty('topic') && + body.hasOwnProperty('event') && + body.hasOwnProperty('message') + ) { + const topic = req.body.topic; + const event = req.body.event; + const message = req.body.message; + + if (topic in sseTopics) { + // sends message to all the subscribers + sseTopics[topic].publish(message, event); + } + } else { + status = 400; + } + + res.status(status).send({ + status, + }); +}); + +// returns JSON object of all opened topics +app.get('/status', async (req, res) => { + res.send(sseTopics); +}); + +// health-check endpoint +app.get('/', async (req, res) => { + res.send('OK'); +}); + +// return a 404 if no routes match +app.use((req, res, next) => { + res.set('Cache-Control', 'private, no-store'); + res.status(404).end('Not found'); +}); + +// starts the server +app.listen(port, () => { + console.log(`PubSub server running on http://localhost:${port}`); +}); +``` + +### Our custom message format + +Each message posted on a server must be in a specific format that out server +accepts. Having structure like this allows us to have multiple separated type of +events on each topic. + +With this we can separate streams and only receive events that belong to the +topic. + +One example would be, that we have index page and we want to receive messages +about new upvotes or new subscribers but we don't want to follow events for +other pages. This reduces clutter and overall network. And structure is much +nicer and maintanable. + +```json +{ + "topic": "sample-topic", + "event": "sample-event", + "message": { "name": "John" } +} +``` + +## Publisher and subscriber clients + +### Publisher and subscriber in action + + + +You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) and +follow along. + +### Publisher + +As talked about above publisher is the one that send messages to the +broker/server. Message inside the payload can be whatever you want (string, +object, array). I would however personally avoid send large chunks of data like +blobs and such. + +```html + + + + + + + Publisher + + + + +

Publisher

+ +
+

+ + +

+

+ + +

+

+ + +

+

+ + +

+

+ +

+
+ + + + + + +``` + +### Subscriber + +Subscriber is responsible for receiving new messages that come from server via +publisher. The code bellow is very rudimentary but works and follows the +implementation guidelines for EventSource. + +You can use either Developer Tools Console to see incoming messages or you can +defer to Debugging with Google Chrome section above to see all EventStream +messages. + +> Don't be alarmed if the subscriber gets disconnected from the server every so +> often. The code we have here resets connection every 15s but it automatically +> get reconnected and fetches all messages up to last received message id. This +> setting can be adjusted in `server.js` file; search for the +> `maxStreamDuration` variable. + +```html + + + + + + + Subscriber + + + + + +

Subscriber

+ +
+

+ + +

+

+ + +

+

+ + +

+

+ +

+
+ + + + + + +``` + +## Reading further + +- [Using server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) +- [Using SSE Instead Of WebSockets For Unidirectional Data Flow Over HTTP/2](https://www.smashingmagazine.com/2018/02/sse-websockets-data-flow-http2/) +- [What is Server-Sent Events?](https://apifriends.com/api-streaming/server-sent-events/) +- [An HTTP/2 extension for bidirectional messaging communication](https://tools.ietf.org/id/draft-xie-bidirectional-messaging-01.html) +- [Introduction to HTTP/2](https://developers.google.com/web/fundamentals/performance/http2) +- [The WebSocket API (WebSockets)](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) + diff --git a/content/posts/2020-03-27-create-placeholder-images-with-sharp.md b/content/posts/2020-03-27-create-placeholder-images-with-sharp.md new file mode 100644 index 0000000..1c2b042 --- /dev/null +++ b/content/posts/2020-03-27-create-placeholder-images-with-sharp.md @@ -0,0 +1,102 @@ +--- +title: Create placeholder images with sharp Node.js image processing library +url: create-placeholder-images-with-sharp.html +date: 2020-03-27T12:00:00+02:00 +type: post +draft: false +--- + +I have been searching for a solution to pre-generate some placeholder images for +image server I needed to develop that resizes images on S3. I though this would +be a 15min job and quickly found out how very mistaken I was. + +Even though Node.js is not really the best way to do this kind of things (surely +something written in C or Rust or even Golang would be the correct way to do +this but we didn't need the speed in our case) I found an excellent library +[sharp - High performance Node.js image +processing](https://github.com/lovell/sharp). + +Getting things running was a breeze. + +## Fetch image from S3 and save resized + +```js +const sharp = require('sharp'); +const aws = require('aws-sdk'); + +const x,y = 100; +const s3 = new aws.S3({}); + +aws.config.update({ + secretAccessKey: 'secretAccessKey', + accessKeyId: 'accessKeyId', + region: 'region' +}); + +const originalImage = await s3.getObject({ + Bucket: 'some-bucket-name', + Key: 'image.jpg', +}).promise(); + +const resizedImage = await sharp(originalImage.Body) + .resize(x, y) + .jpeg({ progressive: true }) + .toBuffer(); + +s3.putObject({ + Bucket: 'some-bucket-name', + Key: `optimized/${x}x${y}/image.jpg`, + Body: resizedImage, + ContentType: 'image/jpeg', + ACL: 'public-read' +}).promise(); +``` + +All this code was wrapped inside a web service with some additional security +checks and defensive coding to detect if key is missing on S3. + +And at that point I needed to return placeholder images as a response in case +key is missing or x,y are not allowed by the server etc. I could have created +PNG in Gimp and just serve them but I wanted to respect aspect ratio and I +didn't want to return some mangled images. + +> Main problem with finding a clean solution I could copy and paste and change a +> bit was a task. API is changing constantly and there weren't clear examples or +> I was unable to find them. + +## Generating placeholder images using SVG + +What I ended up was using SVG to generate text and created image with sharp and +used composition to combine both layers. Response returned by this function is a +buffer you can use to either upload to S3 or save to local file. + +```js +const generatePlaceholderImageWithText = async (width, height, message) => { + const overlay = ` + ${message} + `; + + return await sharp({ + create: { + width: width, + height: height, + channels: 4, + background: { r: 230, g: 230, b: 230, alpha: 1 } + } + }) + .composite([{ + input: Buffer.from(overlay), + gravity: 'center', + }]) + .jpeg() + .toBuffer(); +} +``` + +That is about it. Nothing more to it. You can change the color of the image by +changing `background` and if you want to change text styling you can adapt SVG +to your needs. + +> Also be careful about the length of the text. This function positions text at +> the center and adds `20px` padding on all sides. If text is longer than the +> image it will get cut. diff --git a/content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md b/content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md new file mode 100644 index 0000000..efe88fa --- /dev/null +++ b/content/posts/2020-03-29-the-strange-case-of-elasticsearch-allocation-failure.md @@ -0,0 +1,108 @@ +--- +title: The strange case of Elasticsearch allocation failure +url: the-strange-case-of-elasticsearch-allocation-failure.html +date: 2020-03-29T12:00:00+02:00 +type: post +draft: false +--- + +I've been using Elasticsearch in production for 5 years now and never had a +single problem with it. Hell, never even known there could be a problem. Just +worked. All this time. The first node that I deployed is still being used in +production, never updated, upgraded, touched in anyway. + +All this bliss came to an abrupt end this Friday when I got notification that +Elasticsearch cluster went warm. Well, warm is not that bad right? Wrong! +Quickly after that I got another email which sent chills down my spine. Cluster +is now red. RED! Now, shit really hit the fan! + +I tried googling what could be the problem and after executing allocation +function noticed that some shards were unassigned and 5 attempts were already +made (which is BTW to my luck the maximum) and that meant I am basically fucked. +They also applied that one should wait for cluster to re-balance itself. So, I +waited. One hour, two hours, several hours. Nothing, still RED. + +The strangest thing about it all was, that queries were still being fulfilled. +Data was coming out. On the outside it looked like nothing was wrong but +everybody that would look at the cluster would know immediately that something +was very very wrong and we were living on borrowed time here. + +> **Please, DO NOT do what I did.** Seriously! Please ask someone on official +forums or if you know an expert please consult him. There could be million of +reasons and these solution fit my problem. Maybe in your case it would +disastrous. I had all the data backed up and even if I would fail spectacularly +I would be able to restore the data. It would be a huge pain and I would loose +couple of days but I had a plan B. + +Executing allocation and told me what the problem was but no clear solution yet. + +```yaml +GET /_cat/allocation?format=json +``` + +I got a message that `ALLOCATION_FAILED` with additional info `failed to create +shard, failure ioexception[failed to obtain in-memory shard lock]`. Well +splendid! I must also say that our cluster is capable more than enough to handle +the traffic. Also JVM memory pressure never was an issue. So what happened +really then? + +I tried also re-routing failed ones with no success due to AWS restrictions on +having managed Elasticsearch cluster (they lock some of the functions). + +```yaml +POST /_cluster/reroute?retry_failed=true +``` + +I got a message that significantly reduced my options. + +```json +{ + "Message": "Your request: '/_cluster/reroute' is not allowed." +} +``` + +After that I went on a hunt again. I won't bother you with all the details +because hours/days went by until I was finally able to re-index the problematic +index and hoped for the best. Until that moment even re-indexing was giving me +errors. + +```yaml +POST _reindex +{ + "source": { + "index": "myindex" + }, + "dest": { + "index": "myindex-new" + } +} +``` + +I needed to do this multiple times to get all the documents re-indexed. Then I +dropped the original one with the following command. + +```yaml +DELETE /myindex +``` + +And re-indexed again new one in the original one (well by name only). + +```yaml +POST _reindex +{ + "source": { + "index": "myindex-new" + }, + "dest": { + "index": "myindex" + } +} +``` + +On the surface it looks like all is working but I have a long road in front of +me to get all the things working again. Cluster now shows that it is in Green +mode but I am also getting a notification that the cluster has processing status +which could mean million of things. + +Godspeed! + diff --git a/content/posts/2020-03-30-my-love-and-hate-relationship-with-nodejs.md b/content/posts/2020-03-30-my-love-and-hate-relationship-with-nodejs.md new file mode 100644 index 0000000..dec6f8d --- /dev/null +++ b/content/posts/2020-03-30-my-love-and-hate-relationship-with-nodejs.md @@ -0,0 +1,111 @@ +--- +title: My love and hate relationship with Node.js +url: my-love-and-hate-relationship-with-nodejs.html +date: 2020-03-30T12:00:00+02:00 +type: post +draft: false +--- + +Previous project I was working on was being coded in +[Golang](https://golang.org/). Also was my first project using it. And damn, +that was an awesome experience. The whole thing is just superb. From how errors +are handled. The C-like way you handle compiling. The way the language is +structured making it incredibly versatile and easy to learn. + +It may cause some pain for somebody that is not used of using interfaces to map +JSON and doing the recompilation all the time. But we have tools like +[entr](http://eradman.com/entrproject/) and +[make](https://www.gnu.org/software/make/) to fix that. + +But we are not here to talk about my undying love for **Golang**. Only in some +way we probably should. It is an excellent example of how modern language should +be designed. And because I have used it extensively in the last couple of years +this probably taints my views of other languages. And is doing me a great +disservice. Nevertheless, here we are. + +About two years ago I started flirting with [Node.js](https://nodejs.org/en/) +for a project I started working on. What I wanted was to have things written in +a language that is widely used, and we could get additional developers for. As +much as **Golang** is amazing it's really hard to get developers for it. Even +now. And after playing around with it for a week I felt in love with the speed +of iteration and massive package ecosystem. Do you want SSO? You got it! Do you +want some esoteric library for something? There is a strong chance somebody +wrote it. It is so extensive that you find yourself evaluating packages based on +**GitHub stars** and number of contributors. You get swallowed by the vanity +metrics and that potentially will become the downfall of Node.js. + +Because of the sheer amount of choice I often got anxiety when choosing +libraries. Will I choose the correct one? Is this library something that will be +supported for a foreseeable future or not? I am used of using libraries that are +being in development for 10 years plus (Python, C) and that gave me some sort of +comfort. And it is probably unfair to Node.js and community to expect same +dedication. + +Moving forward ... Work started and things were great. **Speed of iteration was +insane**. For some feature that I would need a day in Golang only took me hour +or two. I became lazy! Using packages all over the place. Falling into the same +trap as others. Packages on top of packages. And [npm](https://www.npmjs.com/) +didn't help at all. The way that the package manager works is just +horrendous. And not allowing to have node_modules outside the project is also +the stupidest idea ever. + +So at that point I started feeling the technical debt that comes with Node.js +and the whole ecosystem. What nobody tells you is that **structuring large +Node.js apps** is more problematic than one would think. And going microservice +for every single thing is also a bad idea. The amount of networking you +introduce with that approach always ends up being a pain in the ass. And I don't +even want to go into system administration here. The overhead is +insane. Package-lock.json made many days feel like living hell for me. And I +would eat the cost of all this if it meant for better development +experience. Well, it didn't. + +The **lack of Typescript** support in the interpreter is still mind boggling to +me. Why haven't they added native support yet for this is beyond me?! That would +have solved so many problems. Lack of type safety became a problem somewhere in +the middle of the project where the codebase was sufficiently large enough to +present problems. We started adding arguments to functions and there was **no +way to implicitly define argument types**. And because at that point there were +a lot of functions, it became impossible to know what each one accepts, +development became more and more trial and error based. + +I tried **implementing Typescript**, but that would present a large refactor +that we were not willing to do at that point. The benefits were not enough. I +also tried [Flow - static type checker](https://flow.org/) but implementation +was also horrible. What Typescript and Flow forces you is to have src folder and +then **transpile** your code into dist folder and run it with node. WTH is that +all about. Why can't this be done in memory or some virtual file system? Why? I +see no reason why this couldn't be done like this. But it is what it is. I +abandoned all hope for static type checking. + +One of the problems that resulted from not having interfaces or types was +inability to model out our data from **Elasticsearch**. I could have done a +**pedestrian implementation** of it, but there must be a better way of doing +this without resorting to some hack basically. Or maybe I haven't found a +solution, which is also a possibility. I have looked, though. No juice! + +**Error handling?** Is that a joke? + +Thank god for **await/async**. Without it, I would have probably just abandoned +the whole thing and went with something else like Python. That's all I am going +to say about this :) + +I started asking myself a question if Node.js is actually ready to be used in a +**large scale applications**? And this was a totally wrong question. What I +should have been asking myself was, how to use Node.js in large scale +application. And you don't get this in **marketing material** for Express or Koa +etc. They never tell you this. Making Node.js scale on infrastructure or in +codebase is really **more of an art than a science**. And just like with the +whole JavaScript ecosystem: + +- impossible to master, +- half of your time you work on your tooling, +- just accept transpilers that convert one code into another (holly smokes), +- error handling is a joke, +- standards? What standards? + +But on the other hand. As I did, you will also learn to love it. Learn to use it +quickly and do impossible things in crazy limited time. + +I hate to admit it. But I love Node.js. Dammit, I love it :) + +2023 Update: I hate Node.js! diff --git a/content/posts/2020-05-05-remote-work.md b/content/posts/2020-05-05-remote-work.md new file mode 100644 index 0000000..905d169 --- /dev/null +++ b/content/posts/2020-05-05-remote-work.md @@ -0,0 +1,72 @@ +--- +title: Remote work and how it affects the daily lives of people +url: remote-work.html +date: 2020-05-05T12:00:00+02:00 +type: post +draft: false +--- + +I have been working remotely for the past 5 years. I love it. Love the freedom +and make your schedule thingy. + +## You work more not less + +I've heard from people things like: "Oh, you are so lucky, working from home, +having all the free time you want". It was obvious they had no clue what means +working remotely. They had this romantic idea of remote work. You can watch TV +whenever you like, you can go outside for a picnic if you want and stuff like +that. + +This may be true if you work a day or two in a week from home. But if you go +completely remote all these changes completely. I take some time to acclimate +but then you start feeling the consequences of going fully remote. And it's not +all rainbows and unicorns. Rather the opposite. + +## Feeling lost + +At first, I remembered I felt lost. I was not used to this kind of environment. +It felt disoriented and a part of you that is used to procrastinate turns on. +You start thinking of a workday as a whole day. And soon this idea of "I can do +this later" starts creeping in. Well, I have the whole day ahead of me. I can do +this a bit later. + +## Hyper-performance + +As a direct result, you become more focused on your work since you don't have +all the interruptions common in the workplace. And you can quickly get used to +this hyper-performance. But this mode requires also a lot of peace and quiet. + +And here we come to the ugly parts of all this. **People rarely have the +self-control** to not waste other people's time. It is paralyzing when people +start calling you, sending you chat messages, etc. The thing is, that when I +achieve this hyper-performance mode I am completely embroiled in the problem I +am solving and this kind of interruptions mess with your head. I need an hour at +least to get back in the zone. Sometimes not achieving the same focus the whole +day. + +I know that life is not how you want it to be and takes its route but from what +I've learned this kind of interruptions can be avoided in 90% of the case easily +just by closing any chat programs and putting your phone in a drawer. + +## Suggestion to all the new remote workers + +- Stop wasting other people's time. You don't bother people at their desks in + the office either. +- Do not replace daily chats in the hallways with instant messaging software. + It will only interrupt people. Nothing good will come of it. +- Set your working hours and try to not allow it to bleed outside these + boundaries and maintain your routine. +- Be prepared that hours will be longer regardless of your good intentions and + your well thought of routine. +- Try to be hyper-focused and do only one thing at the time. Multitasking is the + enemy of progress. +- Avoid long meetings and if possible eliminate them. Rather take time to write + them out and allow others to respond in their own time. Meetings are usually a + large waste of time and most of the people attending them are there just + because the manager said so. +- The software will not solve your problems. And throwing money at problems + neither. +- If you are in a managerial position don't supervise any single minute of + workers. They are probably giving you more hours anyways. Track progress + weekly not daily. You hired them and give them the benefit of the doubt that + they will deliver what you agreed upon. diff --git a/content/posts/2020-08-15-systemd-disable-wake-onmouse.md b/content/posts/2020-08-15-systemd-disable-wake-onmouse.md new file mode 100644 index 0000000..8f411d6 --- /dev/null +++ b/content/posts/2020-08-15-systemd-disable-wake-onmouse.md @@ -0,0 +1,73 @@ +--- +title: Disable mouse wake from suspend with systemd service +url: disable-mouse-wake-from-suspend-with-systemd-service.html +date: 2020-08-15T12:00:00+02:00 +type: post +draft: false +--- + +I recently bought [ThinkPad +X220](https://www.laptopmag.com/reviews/laptops/lenovo-thinkpad-x220) just as a +joke on eBay to test Linux distributions and play around with things and not +destroy my main machine. Little to my knowledge I felt in love with it. Man, +they really made awesome machines back then. + +After changing disk that came with it to SSD and installing Ubuntu to test if  +everything works I noticed that even after a single touch of my external mouse +the system would wake up from sleep even though the lid was shut down. + +I wouldn't even noticed it if laptop didn't have [LED +sleep indicator](https://support.lenovo.com/lk/en/solutions/~/media/Images/ContentImages/p/pd025386_x1_status_03.ashx?w=426&h=262). +I already had a bad experience with Linux and it's power management. I had a +[Dell Inspiron 7537](https://www.pcmag.com/reviews/dell-inspiron-15-7537) laptop +with a touchscreen and while traveling it decided to wake up and started cooking +in my backpack to the point that the digitizer responsible for touch actually +glue off and the whole screen got wrecked. So, I am a bit touchy about this. + +I went on solution hunting and to my surprise there is no easy way to disable +specific devices to perform wake up. Why is this not under the power management  +tab in setting is really strange. + +After googling for a solution I found [this nice article describing the +solution](https://codetrips.com/2020/03/18/ubuntu-disable-mouse-wake-from-suspend/) +that worked for me. The only problem with this solution was that he added his +solution to `.bashrc` and this triggers `sudo` that asks for a password each +time new terminal is opened, which get annoying quickly since I open a lot of +terminals all the time. + +I followed his instructions and got to solution `sudo sh -c "echo 'disabled' > +/sys/bus/usb/devices/2-1.1/power/wakeup"`. + +I created a system service file `sudo nano +/etc/systemd/system/disable-mouse-wakeup.service` and removed `sudo` and +replaced `sh` with `/usr/bin/sh` and pasted all that in `ExecStart`. + +```ini +[Unit] +Description=Disables wakeup on mouse event +After=network.target +StartLimitIntervalSec=0 + +[Service] +Type=simple +Restart=always +RestartSec=1 +User=root +ExecStart=/usr/bin/sh -c "echo 'disabled' > /sys/bus/usb/devices/2-1.1/power/wakeup" + +[Install] +WantedBy=multi-user.target +``` + +After that I enabled, started and checked status of service. + +```sh +sudo systemctl enable disable-mouse-wakeup.service +sudo systemctl start disable-mouse-wakeup.service +sudo systemctl status disable-mouse-wakeup.service +``` + +This will permanently disable that device from wakeing up you computer on boot. +If you have many devices you would like to surpress from waking up your machine +I would create a shell script and call that instead of direclty doing it in +service file. diff --git a/content/posts/2020-09-06-esp-and-micropython.md b/content/posts/2020-09-06-esp-and-micropython.md new file mode 100644 index 0000000..6a2d5fe --- /dev/null +++ b/content/posts/2020-09-06-esp-and-micropython.md @@ -0,0 +1,226 @@ +--- +title: Getting started with MicroPython and ESP8266 +url: esp8266-and-micropython-guide.html +date: 2020-09-06T12:00:00+02:00 +type: post +draft: false +--- + +## Introduction + +A while ago I bought some +[ESP8266](https://www.espressif.com/en/products/socs/esp8266) and +[ESP32](https://www.espressif.com/en/products/socs/esp32) dev boards to play +around with and I finally found a project to try it out. + +For my project, I used [ESP32](https://www.espressif.com/en/products/socs/esp32) +but I could easily choose +[ESP8266](https://www.espressif.com/en/products/socs/esp8266). This guide +contains which tools I use and how I prepared my workspace to code for +[ESP8266](https://www.espressif.com/en/products/socs/esp8266). + +![ESP8266 and ESP32 boards](/assets/esp8366-micropython/boards.jpg) + +This guide covers: + +- flashing SOC +- install proper tooling +- deploying a simple script + +> Make sure that you are using **a good USB cable**. I had some problems with +mine and once I replaced it everything started to work. + +## Flashing the SOC + +Plug your ESP8266 to USB port and check if the device was recognized with +executing `dmesg | grep ch341-uart`. + +Then check if the device is available under `/dev/` by running `ls +/dev/ttyUSB*`. + +> **Linux users**: if a device is not available be sure you are in `dialout` +> group. You can check this by executing `groups $USER`. You can add a user to +> `dialout` group with `sudo adduser $USER dialout`. + +After these conditions are meet go to the navigate to +[https://micropython.org/download/esp8266/](https://micropython.org/download/esp8266/) +and download `esp8266-20200902-v1.13.bin`. + +```sh +mkdir esp8266-test +cd esp8266-test + +wget https://micropython.org/resources/firmware/esp8266-20200902-v1.13.bin +``` + +After obtaining firmware we will need some tooling to flash the firmware to the +board. + +```sh +sudo pip3 install esptool +``` + +You can read more about `esptool` at +[https://github.com/espressif/esptool/](https://github.com/espressif/esptool/). + +Before flashing the firmware we need to erase the flash on device. Substitute +`USB0` with the device listed in output of `ls /dev/ttyUSB*`. + +```sh +esptool.py --port /dev/ttyUSB0 erase_flash +``` + +If flash was successfully erased it is now time to flash the new firmware to it. + +```sh +esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash --flash_size=detect 0 esp8266-20200902-v1.13.bin +``` + +If everything went ok you can try accessing MicroPython REPL with ` screen +/dev/ttyUSB0 115200` or `picocom /dev/ttyUSB0 -b115200`. + +> Sometimes you will need to press `ENTER` in `screen` or `picocom` to access +> REPL. + +When you are in REPL you can test if all is working properly following steps. + +```py +> import machine +> machine.freq() +``` + +This should output a number representing a frequency of the CPU (mine was +`80000000`). + +When you are in `screen` or `picocom` these can help you a bit. + +| Key | Command | +| -------- | -------------------- | +| CTRL+d | preforms soft reboot | +| CTRL+a x | exits picocom | +| CTRL+a \ | exits screen | + + +## Install better tooling + +Now, to make our lives a little bit easier there are couple of additional tools +that will make this whole experience a little more bearable. + +There are twq cool ways of uploading local files to SOC flash. + +- ampy → [https://github.com/scientifichackers/ampy](https://github.com/scientifichackers/ampy) +- rshell → [https://github.com/dhylands/rshell](https://github.com/dhylands/rshell) + +### ampy + +```bash +# installing ampy +sudo pip3 install adafruit-ampy +``` + +Listed below are some common commands I used. + +```bash + +# uploads file to flash +ampy --delay 2 --port /dev/ttyUSB0 put boot.py + +# lists file on flash +ampy --delay 2 --port /dev/ttyUSB0 ls + +# outputs contents of file on flash +ampy --delay 2 --port /dev/ttyUSB0 cat boot.py +``` + +> I added `delay` of 2 seconds because I had problems with executing commands. + +### rshell + +Even though `ampy` is a cool tool I opted with `rshell` in the end since it's +much more polished and feature rich. + +```bash +# installing ampy +sudo pip3 install rshell +``` + +Now that `rshell` is installed we can connect to the board. + +```bash +rshell --buffer-size=30 -p /dev/ttyUSB0 -a +``` + +This will open a shell inside bash and from here you can execute multiple +commands. You can check what is supported with `help` once you are inside of a +shell. + +```bash +m@turing ~/Junk/esp8266-test +$ rshell --buffer-size=30 -p /dev/ttyUSB0 -a + +Using buffer-size of 30 +Connecting to /dev/ttyUSB0 (buffer-size 30)... +Trying to connect to REPL connected +Testing if ubinascii.unhexlify exists ... Y +Retrieving root directories ... /boot.py/ +Setting time ... Sep 06, 2020 23:54:28 +Evaluating board_name ... pyboard +Retrieving time epoch ... Jan 01, 2000 +Welcome to rshell. Use Control-D (or the exit command) to exit rshell. +/home/m/Junk/esp8266-test> help + +Documented commands (type help ): +======================================== +args cat connect date edit filesize help mkdir rm shell +boards cd cp echo exit filetype ls repl rsync + +Use Control-D (or the exit command) to exit rshell. +``` + +> Inside a shell `ls` will display list of files on your machine. To get list +> of files on flash folder `/pyboard` is remapped inside the shell. To list files +> on flash you must perform `ls /pyboard`. + +#### Moving files to flash + +To avoid copying files all the time I used `rsync` function from the inside of +`rshell`. + +```bash +rsync . /pyboard +``` + +#### Executing scripts + +It is a pain to continuously reboot the device to trigger `/pyboard/boot.py` and +there is a better way of testing local scripts on remote device. + +Lets assume we have `src/freq.py` file that displays CPU frequency of a remote +device. + +```py +# src/freq.py + +import machine +print(machine.freq()) +``` + +Now lets upload this and execute it. + +```bash +# syncs files to remove device +rsync ./src /pyboard + +# goes into REPL +repl + +# we import file by importing it without .py extension and this will run the script +> import freq + +# CTRL+x will exit REPL +``` + +## Additional resources + +- https://randomnerdtutorials.com/getting-started-micropython-esp32-esp8266/ +- http://docs.micropython.org/en/latest/esp8266/quickref.html diff --git a/content/posts/2020-09-08-bind-warning-on-login.md b/content/posts/2020-09-08-bind-warning-on-login.md new file mode 100644 index 0000000..f213cd9 --- /dev/null +++ b/content/posts/2020-09-08-bind-warning-on-login.md @@ -0,0 +1,54 @@ +--- +title: Fix bind warning in .profile on login in Ubuntu +url: bind-warning-on-login-in-ubuntu.html +date: 2020-09-08T12:00:00+02:00 +type: post +draft: false +--- + +Recently I moved back to [bash](https://www.gnu.org/software/bash/) as my +default shell. I was previously using [fish](https://fishshell.com/) and got +used to the cool features it has. But, regardless of that, I wanted to move to a +more standard shell because I was hopping back and forth with exporting +variables and stuff like that which got pretty annoying. + +So I embarked on a mission to make [bash](https://www.gnu.org/software/bash/) +more like [fish](https://fishshell.com/) and in the process found that I really +missed autosuggest with TAB on changing directories. + +I found a nice alternative that emulates [zsh](http://zsh.sourceforge.net/) like +autosuggestion and autocomplete so I added the following to my `.bashrc` file. + +```bash +bind "TAB:menu-complete" +bind "set show-all-if-ambiguous on" +bind "set completion-ignore-case on" +bind "set menu-complete-display-prefix on" +bind '"\e[Z":menu-complete-backward' +``` + +I haven't noticed anything wrong with this and all was working fine until I +restarted my machine and then I got this error. + +![Profile bind error](/assets/profile-bind-error/error.jpg) + +When I pressed OK, I got into the [Gnome +shell](https://wiki.gnome.org/Projects/GnomeShell) and all was working fine, but +the error was still bugging me. I started looking for the reason why this is +happening and found a solution to this error on [Remote SSH Commands - bash bind +warning: line editing not enabled](https://superuser.com/a/892682). + +So I added a simple `if [ -t 1 ]` around `bind` statements to avoid running +commands that presume the session is interactive when it isn't. + +```bash +if [ -t 1 ]; then + bind "TAB:menu-complete" + bind "set show-all-if-ambiguous on" + bind "set completion-ignore-case on" + bind "set menu-complete-display-prefix on" + bind '"\e[Z":menu-complete-backward' +fi +``` + +After logging out and back in the problem was gone. diff --git a/content/posts/2020-09-09-digitalocean-sync.md b/content/posts/2020-09-09-digitalocean-sync.md new file mode 100644 index 0000000..e16b827 --- /dev/null +++ b/content/posts/2020-09-09-digitalocean-sync.md @@ -0,0 +1,112 @@ +--- +title: Using Digitalocean Spaces to sync between computers +url: digitalocean-spaces-to-sync-between-computers.html +date: 2020-09-09T12:00:00+02:00 +type: post +draft: false +--- + +I've been using [Dropbox](https://www.dropbox.com/) for probably **10+ years** +now and I-ve became so used to it that it runs in the background that I don't +even imagine a world without it. But it's not without problems. + +At first I had problems with `.venv` environments for Python and the only +solution for excluding synchronization for this folder was to manually exclude a +specific folder which is not really scalable. FYI, my whole project folder is +synced on [Dropbox](https://www.dropbox.com/). This of course introduced a lot +of syncing of files and folders that are not needed or even break things on +other machines. In the case of **Python**, I couldn't use that on my second +machine. I needed to delete `.venv` folder and pip it again which synced files +again to the main machine. This was very frustrating. **Nodejs** handles this +much nicer and I can just run the scripts without deleting `node_modules` again +and reinstalling. However, `node_modules` is a beast of its own. It creates so +many files that OS has a problem counting them when you check the folder +contents for size. + +I wanted something similar to Dropbox. I could without the instant syncing but +it would need to be fast and had the option for me to exclude folders like +`node_modules, .venv, .git` and folders like that. + +I went on a hunt for an alternative to [Dropbox](https://www.dropbox.com/) +and found: + +- [Tresorit](https://tresorit.com/) +- [Sync.com](https://sync.com) +- [Box](https://www.box.com/) + +You know, the usual list of suspects. I didn't include [Google +drive](https://drive.google.com) or [One drive](https://onedrive.live.com/) +since they are even more draconian than Dropbox. + +> All this does not stem from me being paranoid but recently these companies +> have became more and more aggressive and they keep violating our privacy when +> they share our data with 3rd party services. It is getting out of control. + +So, my main problem was still there. No way of excluding a specific folder from +syncing. And before we go into "*But you have git, isn't that enough?*", I must +say, that many of the files (PDFs, spreadsheets, etc) I have in a `git` repo +don't get pushed upstream to Git and I still want to have them synced across my +computers. + +I initially wanted to use [rsync](https://linux.die.net/man/1/rsync) but I would +need to then have a remote VPS or transfer between my computers directly. I +wanted a solution where all my files could be accessible to me without my +machine. + +> **WARNING: This solution will cost you money!** DigitalOcean Spaces are $5 per +month and there are some bandwidth limitations and if you go beyond that you get +billed additionally. + +Then I remembered that I could use something like +[S3](https://en.wikipedia.org/wiki/Amazon_S3) since it has versioning and is +fully managed. I didn't want to go down the AWS rabbit hole with this so I +choose [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/). + +Then I needed a command-line tool to sync between source and target. I found +this nice tool [s3cmd](https://s3tools.org/s3cmd) and it is in the Ubuntu +repositories. + +```bash +sudo apt install s3cmd +``` + +After installation will I create a new Space bucket on DigitalOcean. Remember +the zone you will choose because you will need it when you will configure +`s3cmd`. + +Then I visited [Digitalocean Applications & +API](https://cloud.digitalocean.com/account/api/tokens) and generated **Spaces +access keys**. Save both key and secret somewhere safe because when you will +leave the page secret will not be available anymore to you and you will need to +re-generate it. + +```bash +# enter your key and secret and correct endpoint +# my endpoint is ams3.digitaloceanspaces.com because +# I created my bucket in Amsterdam regiin +s3cmd --configure +``` + +After that I played around with options for `s3cmd` and got to the following +command. + +```bash +# I executed this command from my projects folder +cd projects +s3cmd sync --delete-removed --exclude 'node_modules/*' --exclude '.git/*' --exclude '.venv/*' ./ s3://my-bucket-name/projects/ +``` + +When syncing int he other direction you will need to change the order of the +`SOURCE` and `TARGET` to `s3://my-bucket-name/projects/` and `./`. + +> Be sure that all the paths have trailing slash so that sync knows that this +> are directories. + +I am planning to implement some sort of a `.ignore` file that will enable me to +have a project-specific exclude options. + +I am currently running this every hour as a cronjob which is perfectly fine for +now when I am testing how this whole thing works and how it all will turn out. + +I have also created a small Gnome extension which is still very unstable, but +when/if this whole experiment pays of I will share on Github. diff --git a/content/posts/2021-01-24-replacing-dropbox-with-s3.md b/content/posts/2021-01-24-replacing-dropbox-with-s3.md new file mode 100644 index 0000000..a44a1aa --- /dev/null +++ b/content/posts/2021-01-24-replacing-dropbox-with-s3.md @@ -0,0 +1,114 @@ +--- +title: Replacing Dropbox in favor of DigitalOcean spaces +url: replacing-dropbox-in-favor-of-digitalocean-spaces.html +date: 2021-01-24T12:00:00+02:00 +type: post +draft: false +--- + +A few months ago I experimented with DigitalOcean spaces as my backup solution +that could [replace Dropbox +eventually](/digitalocean-spaces-to-sync-between-computers.html). That solution +worked quite nicely, and I was amazed how smashing together a couple of existing +solutions would work this fine. + +I have been running that solution in the background for a couple of months now +and kind of forgot about it. But recent developments around deplatforming and +having us people hostages of technology and big companies speed up my goals to +become less dependent on +[Google](https://edition.cnn.com/2020/12/17/tech/google-antitrust-lawsuit/index.html), +[Dropbox](https://www.pcworld.com/article/2048680/dropbox-takes-a-peek-at-files.html) +etc and take back some control. + +I am not a conspiracy theory nut, but to be honest, what these companies are +doing lately is out of control. It is a matter of principle at this point. I +have almost completely degoogled my life all the way from ditching Gmail, +YouTube and most of the services surrounding Google. And I must tell you, I feel +so good. I haven't felt this way for a long time. + +**Anyways. Let's get to the meat of things.** + +Before you continue you should read my post about [syncing to +Dropbox](/digitalocean-spaces-to-sync-between-computers.html). + +> Also to note, I am using Linux on my machine with Gnome desktop environment. +This should work on MacOS too. To use this on Windows I suggest using +[Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10) +or [Cygwin](https://www.cygwin.com/). + +## Folder structure + +I liked structure from Dropbox. One folder where everything is located and +synced. So, that's why adopted this also for my sync setup. + +```go +~/Vault + ↳ backup + ↳ bin + ↳ documents + ↳ projects +``` + +All of my code is located in `~/Vault/projects` folder. And most of the projects +are Git repositories. I do not use this sync method for backup per see but in +case I reinstall my machine I can easily recreate all the important folder +structure with one quick command. No external drives needed that can fail etc. + +## Sync script + +My sync script is located in `~/Vault/bin/vault-backup.sh` + +```bash +#!/bin/bash + +# dconf load /com/gexperts/Tilix/ < tilix.dconf +# 0 2 * * * sh ~/Vault/bin/vault-backup.sh + +cd ~/Vault/backup/dotfiles + +MACHINE=$(whoami)@$(hostname) +mkdir -p $MACHINE +cd $MACHINE + +cp ~/.config/VSCodium/User/settings.json settings.json +cp ~/.s3cfg s3cfg +cp ~/.bash_extended bash_extended +cp ~/.ssh ssh -rf + +codium --list-extensions > vscode-extension.txt +dconf dump /com/gexperts/Tilix/ > tilix.dconf + +cd ~/Vault +s3cmd sync --delete-removed --exclude 'node_modules/*' --exclude '.git/*' --exclude '.venv/*' ./ s3://bucket-name/backup/ + +echo `date +"%D %T"` >> ~/.vault.log + +notify-send \ + -u normal \ + -i /usr/share/icons/Adwaita/96x96/status/security-medium-symbolic.symbolic.png \ + "Vault sync succeded at `date +"%D %T"`" +``` + +This script also backups some of the dotfiles I use and sends notification to +Gnome notification center. It is a straightforward solution. Nothing special +going on. + +> One obvious benefit of this is that I can omit syncing Node's `node_modules` +> or Python's `.venv` and `.git` folders. + +You can use this script in a combination with [Cron](https://en.wikipedia.org/wiki/Cron). + +``` +0 2 * * * sh ~/Vault/bin/vault-backup.sh +``` + +When you start syncing your local stuff with a remote server you can review your +items on DigitalOcean. + +![Dropbox Spaces](/assets/dropbox-sync/dropbox-spaces.png) + +I have been using this script now for quite some time, and it's working +flawlessly. I also uninstalled Dropbox and stopped using it completely. + +All I need to do is write a Bash script that does the reverse and downloads from +remote server to local folder. This could be another post. diff --git a/content/posts/2021-01-25-goaccess.md b/content/posts/2021-01-25-goaccess.md new file mode 100644 index 0000000..0eb2461 --- /dev/null +++ b/content/posts/2021-01-25-goaccess.md @@ -0,0 +1,203 @@ +--- +title: Using GoAccess with Nginx to replace Google Analytics +url: using-goaccess-with-nginx-to-replace-google-analytics.html +date: 2021-01-25T12:00:00+02:00 +type: post +draft: false +--- + +## Introduction + +I know! You cannot simply replace Google Analytics with parsing access logs and +displaying a couple of charts. But to be honest, I actually never used Google +Analytics to the fullest extent and was usually interested in seeing page hits +and which pages were visited most often. + +I recently moved my blog from Firebase to a VPS and also decided to remove +Google Analytics tracking code from the site since its quite malicious and +tracks users across other pages also and is creating a profile of a user, and +I've had it. But I also need some insight of what is happening on a server and +which content is being read the most etc. + +I have looked at many existing solutions like: + +- [Umami](https://umami.is/) +- [Freshlytics](https://github.com/sheshbabu/freshlytics) +- [Matomo](https://matomo.org/) + +But the more I looked at them the more I noticed that I am replacing one evil +with another one. Don't get me wrong. Some of these solutions are absolutely +fantastic but would require installation of databases and something like PHP or +Node. And I was not ready to put those things on my fresh server. Also having +Docker installed is out of the question. + +## Opting for log parsing + +So, I defaulted to parsing already existing logs and generating HTML reports +from this data. + +I found this amazing software [GoAccess](https://goaccess.io/) which provides +all the functionalities I need, and it's a single binary. Written in Go. + +GoAccess can be used in two different modes. + +![GoAccess Terminal](/assets/goaccess/goaccess-dash-term.png) +
Running in a terminal
+ +![GoAccess HTML](/assets/goaccess/goaccess-dash-html.png) +
Running in a browser
+ +I, however, need this to run in a browser. So, the second option is the way to +go. The Idea is to periodically run cronjob and export this report into a folder +that gets then server by Nginx behind a Basic authentication. + +## Getting Nginx ready + +I choose Ubuntu on [DigitalOcean](https://www.digitalocean.com/). First I +installed [Nginx](https://nginx.org/en/), and +[Letsencrypt](https://letsencrypt.org/getting-started/) certbot and all the +necessary dependencies. + +```sh +# log in as root user +sudo su - + +# first let's update the system +apt update && apt upgrade -y + +# let's install +apt install nginx certbot python3-certbot-nginx apache2-utils +``` + +After all this is installed we can create a new configuration for a statistics. +Stats will be available at `stats.domain.com`. + +```sh +# creates directory where html will be hosted +mkdir -p /var/www/html/stats.domain.com + +cp /etc/nginx/sites-available/default /etc/nginx/sites-available/stats.domain.com +nano /etc/nginx/sites-available/stats.domain.com +``` + +```nginx +server { + root /var/www/html/stats.domain.com; + server_name stats.domain.com; + + index index.html; + location / { + try_files $uri $uri/ =404; + } +} +``` + +Now we check if the configuration is ok. We can do this with `nginx -t`. If all +is ok, we can restart Nginx with `service nginx restart`. + +After all that you should add A record for this domain that points to IP of a +droplet. + +Before enabling SSL you should test if DNS records have propagated with `curl +stats.domain.com`. + +Now, it's time to provision TLS certificate. To achieve this, you execute +command `certbot --nginx`. Follow the wizard and when you are asked about +redirection always choose 2 (always redirect to HTTPS). + +When this is done you can visit https://stats.domain.com and you should get 404 +not found error which is correct. + +## Getting GoAccess ready + +If you are using Debian like system GoAccess should be available in repository. +Otherwise refer to the official website. + +```sh +apt install goaccess +``` + +To enable Geo location we also need one additiona thing. + +```sh +cd /var/www/html/stats.stats.com +wget https://github.com/P3TERX/GeoLite.mmdb/raw/download/GeoLite2-City.mmdb +``` + +Now we create a shell script that will be executed every 10 minutes. + +```sh +nano /var/www/html/stats.domain.com/generate-stats.sh +``` + +Contents of this file should look like this. + +```sh +#!/bin/sh + +zcat -f /var/log/nginx/access.log* > /var/log/nginx/access-all.log + +goaccess \ + --log-file=/var/log/nginx/access-all.log \ + --log-format=COMBINED \ + --exclude-ip=0.0.0.0 \ + --geoip-database=/var/www/html/stats.domain.com/GeoLite2-City.mmdb \ + --ignore-crawlers \ + --real-os \ + --output=/var/www/html/stats.domain.com/index.html + +rm /var/log/nginx/access-all.log +``` + +Because after a while nginx creates multiple files with access logs we use +[`zcat`](https://linux.die.net/man/1/zcat) to extract Gziped contents and create +a file that has all the access logs. After this file is used we delete it. + +If you want to exclude your home IP's result look at the `--exclude-ip` option +in script and instead of `0.0.0.0` add your own home IP address. You can find +your home IP by executing `curl ifconfig.me` from your local machine and NOT +from the droplet. + +Test the script by executing `sh +/var/www/html/stats.domain.com/generate-stats.sh` and then checking +`https://stats.domain.com`. If you can see stats instead of 404 than you are +set. + +It's time to add this script to cron with `cron -e`. + +```go +*/10 * * * * sh /var/www/html/stats.domain.com/generate-stats.sh +``` + +## Securing with Basic authentication + +You probably don't want stats to be publicly available, so we should create a +user and a password for Basic authentication. + +First we create a password for a user `stats` with `htpasswd -c /etc/nginx/.htpasswd stats`. + +Now we update config file with `nano +/etc/nginx/sites-available/stats.domain.com`. You probably noticed that the +file looks a bit different from before. This is because `certbot` added +additional rules for SSL. + +Your location portion the config file should now look like. You should add +`auth_basic` and `auth_basic_user_file` lines to the file. + +```nginx +location / { + try_files $uri $uri/ =404; + auth_basic "Private Property"; + auth_basic_user_file /etc/nginx/.htpasswd; +} +``` + +Test if config is still ok with `nginx -t` and if it is you can restart Nginx +with `service nginx restart`. + +If you now visit `https://stats.domain.com` you should be prompted for username +and password. If not, try reopening your browser. + +That is all. You now have analytics for your server that gets refreshed every 10 +minutes. + diff --git a/content/posts/2021-06-26-simple-world-clock.md b/content/posts/2021-06-26-simple-world-clock.md new file mode 100644 index 0000000..0c17f09 --- /dev/null +++ b/content/posts/2021-06-26-simple-world-clock.md @@ -0,0 +1,108 @@ +--- +title: Simple world clock with eInk display and Raspberry Pi Zero +url: simple-world-clock-with-eiink-display-and-raspberry-pi-zero.html +date: 2021-06-26T12:00:00+02:00 +type: post +draft: false +--- + +Our team is spread across the world, from the USA all the way to Australia, so +having some sort of world clock makes sense. + +Currently, I am using an extension for Gnome called [Timezone +extension](https://extensions.gnome.org/extension/2657/timezones-extension/), +and it serves the purpose quite well. + +But I also have a bunch of electronics that I bought through the time, and I am +not using any of them, and it's time to stop hording this stuff and use it in a +project. + +A while ago I bought a small eInk display [Inky +pHAT](https://shop.pimoroni.com/products/inky-phat?variant=12549254217811) and I +have a bunch of [Raspberry Pi's +Zero](https://www.raspberrypi.org/products/raspberry-pi-zero/) lying around that +I really need to use. + +![Inky pHAT, Raspberry Pi Zero](/assets/world-clock/hardware.jpg) + +Since the Inky [Inky +pHAT](https://shop.pimoroni.com/products/inky-phat?variant=12549254217811) is +essentially a HAT, it can easily be added on top of the [Raspberry Pi +Zero](https://www.raspberrypi.org/products/raspberry-pi-zero/). + +First, I installed the necessary software on Raspberry Pi with `pip3 install +inky`. + +And then I created a file `clock.py` in home directory `/home/pi`. + +```python +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +import sys +import os +from inky.auto import auto +from PIL import Image, ImageFont, ImageDraw +from font_fredoka_one import FredokaOne + +clocks = [ + 'America/New_York', + 'Europe/Ljubljana', + 'Australia/Brisbane', +] + +board = auto() +board.set_border(board.WHITE) +board.rotation = 90 + +img = Image.new('P', (board.WIDTH, board.HEIGHT)) +draw = ImageDraw.Draw(img) + +big_font = ImageFont.truetype(FredokaOne, 18) +small_font = ImageFont.truetype(FredokaOne, 13) + +x = board.WIDTH / 3 +y = board.HEIGHT / 3 + +idx = 1 +for clock in clocks: + ctime = os.popen('TZ="{}" date +"%a,%H:%M"'.format(clock)) + ctime = ctime.read().strip().split(',') + city = clock.split('/')[1].replace('_', ' ') + + draw.text((15, (idx*y)-y+10), city, fill=board.BLACK, font=small_font) + draw.text((110, (idx*y)-y+7), str(ctime[0]), fill=board.BLACK, font=big_font) + draw.text((155, (idx*y)-y+7), str(ctime[1]), fill=board.BLACK, font=big_font) + + idx += 1 + +board.set_image(img) +board.show() +``` + +And because eInk displays are rather slow to refresh and the clock requires +refreshing only once a minute, this can be done through cronjob. + +Before we add this job to cron we need to make `clock.py` executable with `chmod ++x clock.py`. + +Then we add a cronjob with `crontab -e`. + +``` +* * * * * /home/pi/clock.py +``` + +So, we end up with a result like this. + +![World Clock](/assets/world-clock/world-clock.jpg) + +And for the enclosure that can be 3D printed, but I haven't yet something like +this can be used. + + + +You can download my [STL file for the enclosure +here](/assets/world-clock/enclosure.stl), but make sure that dimensions make +sense and also opening for USB port should be added or just use a drill and some +hot glue to make it stick in the enclosure. + diff --git a/content/posts/2021-07-30-from-internet-consumer-to-full-hominum-again.md b/content/posts/2021-07-30-from-internet-consumer-to-full-hominum-again.md new file mode 100644 index 0000000..100645b --- /dev/null +++ b/content/posts/2021-07-30-from-internet-consumer-to-full-hominum-again.md @@ -0,0 +1,103 @@ +--- +title: My journey from being an internet über consumer to being a full hominum again +url: from-internet-consumer-to-full-hominum-again.html +date: 2021-07-30T12:00:00+02:00 +type: post +draft: false +--- + +It's been almost a year since I started purging all my online accounts and +going down this rabbit hole of being almost independent of the current internet +machine. Even though I initially thought that I will have problems adapting, +I was pleasantly surprised that the transition went so smoothly. Even better, +it brought many benefits to my life. Such as increased focus, less stress +about trivial things, etc. + +It all started with me doing small changes like unsubscribing from emails that I +have either subscribed to by accepting terms and conditions. Or even some more +malicious emails that I was getting because I was on a shared mailing list. And +the later ones I hate the most of all. How the hell do they keep sharing my +email and sending me unsolicited emails and get away with it? I have a suspicion +that these marketing people share an Excel file between them and keep +resubscribing me when they import lists into Mailchimp or similar software. + +It's fascinating to see how much crap you get subscribed to when you are not +paying attention. It got so bad that my primary Gmail address is a full of junk +and need constant monitoring and cleaning up. And because I want to have Inbox +Zero, this presents an additional problem for me. + +The stress that email presented for me didn't occur to me for a long time. I was +noticing that I was unable to go through one single hour without hysterically +refreshing email. And if somebody wrote me something, I needed to see it right +then, even though I didn't immediately reply to it. I can only describe this +with FOMO (fear of missing out). I have no other explanation than that. It was +crippling, and I was constantly context switching, which I will address further +down this post in more details. + +This was one of the reasons why I spawned up my personal email server, and I am +using it now as my primary and person email. I still have Gmail as my “junk” +email that I use for throw away stuff. I log in to Gmail once a week and check +if there are any important emails that I got, but apart from that, it's sitting +dormant and collecting dust. + +The more I was watching the world loose it's self with allowing anti freedom +things to happen to it, the more I started to realize that something has to +change. I don't have the power to change the world. And I also don't have a +grandiose opinion of myself to even think to try it. But what I can do is to not +subscribe to this consumer way of thinking. I will not be complicit in this. My +moral and ethical stances won't allow it. So, this brings us to the second part +of my journey. + +I was using all these 3rd party services because I was either lazy or OK with +the drawbacks of them. I watched these services and companies became more and +privacy policies and everybody is OK with accepting them, and they pray on that +more evil. It is evil if you sell your user's data in this manner. Nobody reads +flaw in human nature. I really hate the hypocrisy they manage to muster. These +companies prey on our laziness, and we are at fault here. Nobody else. And I +truly understand the reasons why we rather accept and move on, and not object +and have our lives a little more difficult. They have perfected this through +years of small changes that make us a little more dependent on them. You could +not convince a person to give away all his rights and data in one day. This was +gradual and slow. And it caught us all in surprise. When I really stopped and +thought about it, I felt repulsed. By really stopping and thinking about it, I +really mean stopping and thinking about it. Thoroughly and in depth. + +Each step I took depleted my character a bit more. Like I was trading myself bit +by bit without understanding what it all meant. What it meant to be a full +person, not divided by all this bought attention they want from me. They don't +just get your data, but they also take your attention away from you. They +scatter your and go with the divide and conquer tactic from there. And a person +divided is a person not fully there. Not at the moment. Not alive fully. + +I was unable to form long thoughts. Well, I thought I was. But now that I see +what being a full person is again, I can see that I was not at my 100% back +then. + +A revolt was inevitable. There was no other way of continuing my story without +it. Without taking back my attention, my thoughts, my time, and my privacy, +regardless of how too late it maybe is. + +This has nothing to do with conspiracy theories. Even less with changing the +world. All I wanted was to get my life back in order and not waste the energy +that could be spent in other, better places. + +I started reading more. I can focus now fully on things I work on. Furthermore, +I have the mental acuity that I never had before. My mind feels sharp. I don't +get angry so much. I can cherish the finer things in life now without the need +to interpret them intellectually. Not only that, but I have a feeling of +belonging again. Sense of purpose has returned with a vengeance. And I can now +help people without depleting myself. + +The last step so far was to finish closing all the remaining online accounts +that I still had. And when I was thinking what value they bring me, I wasn't +surprised that the answer was none. I wasn't logging in them and using them. I +stopped being afraid of FOMO. If somebody wants to get in contact me, they will +find a way. I am one search away. + +We are not beholden to anybody. Our lives are our own. So dare yourself to +delete Facebook, LinkedIn. To unsubscribe. Dare yourself to take your time and +attention back. Use that time and energy to go for a walk without thinking about +work. Read a book instead of reading comment on social media that you will +forget in an hour. Enrich your life instead of wasting it. It only requires a +small step. And you will feel the benefits immediately. Lose the weight of the +world that is crushing you without your consent. diff --git a/content/posts/2021-08-01-linux-cheatsheet.md b/content/posts/2021-08-01-linux-cheatsheet.md new file mode 100644 index 0000000..20e3382 --- /dev/null +++ b/content/posts/2021-08-01-linux-cheatsheet.md @@ -0,0 +1,287 @@ +--- +title: List of essential Linux commands for server management +url: linux-cheatsheet.html +date: 2021-08-01T12:00:00+02:00 +type: post +draft: false +--- + +**Generate SSH key** + +```bash +ssh-keygen -t ed25519 -C "your_email@example.com" + +# when no support for Ed25519 present +ssh-keygen -t rsa -b 4096 -C "your_email@example.com" +``` + +Note: By default SSH keys get stored to `/home//.ssh/` folder. + +**Login to host via SSH** + +```bash +# connect to host as your local username +ssh host + +# connect to host as user +ssh @ + +# connect to host using port +ssh -p @ +``` + +**Execute command on a server through SSH** + +```bash +# execute one command +ssh root@100.100.100.100 "ls /root" + +# execute many commands +ssh root@100.100.100.100 "cd /root;touch file.txt" +``` + +**Displays currently logged in users in the system** + +```bash +w +``` + +**Displays Linux system information** + +```bash +uname +``` + +**Displays kernel release information** + +```bash +uname -r +``` + +**Shows the system hostname** + +```bash +hostname +``` + +**Shows system reboot history** + +```bash +last reboot +``` + +**Displays information about the user** + +```bash +sudo apt install finger +finger +``` + +**Displays IP addresses and all the network interfaces** + +```bash +ip addr show +``` + +**Downloads a file from an online source** + +```bash +wget https://example.com/example.tgz +``` + +Note: If URL contains ?, & enclose the URL in double quotes. + +**Compress a file with gzip** + +```bash +# will not keep the original file +gzip file.txt + +# will keep the original file +gzip --keep file.txt +``` + +**Interactive disk usage analyzer** + +```bash +sudo apt install ncdu + +ncdu +ncdu +``` + +**Install Node.js using the Node Version Manager** + +```bash +curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash +source ~/.bashrc + +nvm install v13 +``` + +**Too long; didn't read** + +```bash +npm install -g tldr + +tldr tar +``` + +**Combine all Nginx access logs to one big log file** + +```bash +zcat -f /var/log/nginx/access.log* > /var/log/nginx/access-all.log +``` + +**Set up Redis server** + +```bash +sudo apt install redis-server redis-tools + +# check if server is running +sudo service redis status + +# set and get a key value +redis-cli set mykey myvalue +redis-cli get mykey + +# interactive shell +redis-cli +``` + +**Generate statistics of your webserver** + +```bash +sudo apt install goaccess + +# check if installed +goaccess -v + +# combine logs +zcat -f /var/log/nginx/access.log* > /var/log/nginx/access-all.log + +# export to single html +goaccess \ + --log-file=/var/log/nginx/access-all.log \ + --log-format=COMBINED \ + --exclude-ip=0.0.0.0 \ + --ignore-crawlers \ + --real-os \ + --output=/var/www/html/stats.html + +# cleanup afterwards +rm /var/log/nginx/access-all.log +``` + +**Search for a given pattern in files** + +```bash +grep -r ‘pattern’ files +``` + +**Find proccess ID for a specific program** + +```bash +pgrep nginx +``` + +**Print name of current/working directory** + +```bash +pwd +``` + +**Creates a blank new file** + +```bash +touch newfile.txt +``` + +**Displays first lines in a file** + +```bash +# -n presents the number of lines (10 by default) +head -n 20 somefile.txt +``` + +**Displays last lines in a file** + +```bash +# -n presents the number of lines (10 by default) +tail -n 20 somefile.txt + +# -f follows the changes in file (doesn't closes) +tail -f somefile.txt +``` + +**Count lines in a file** + +```bash +wc -l somefile.txt +``` + +**Find all instances of the file** + +```bash +sudo apt install mlocate + +locate somefile.txt +``` + +**Find file names that begin with ‘index’ in /home folder** + +```bash +find /home/ -name "index" +``` + +**Find files larger than 100MB in the home folder** + +```bash +find /home -size +100M +``` + +**Displays block devices related information** + +```bash +lsblk +``` + +**Displays free space on mounted systems** + +```bash +df -h +``` + +**Displays free and used memory in the system** + +```bash +free -h +``` + +**Displays all active listening ports** + +```bash +sudo apt install net-tools + +netstat -pnltu +``` + +**Kill a process violently** + +```bash +kill -9 +``` + +**List files opened by user** + +```bash +lsof -u +``` + +**Execute "df -h", showing periodic updates** + +```bash +# -n 1 means every second +watch -n 1 df -h +``` + diff --git a/content/posts/2021-12-03-debian-based-riced-up-distribution-for-developers.md b/content/posts/2021-12-03-debian-based-riced-up-distribution-for-developers.md new file mode 100644 index 0000000..58c9b0d --- /dev/null +++ b/content/posts/2021-12-03-debian-based-riced-up-distribution-for-developers.md @@ -0,0 +1,276 @@ +--- +title: Debian based riced up distribution for Developers and DevOps folks +url: debian-based-riced-up-distribution-for-developers-and-devops-folks.html +date: 2021-12-03T12:00:00+02:00 +type: post +draft: false +--- + +## Introduction + +I have been using [Ubuntu](https://ubuntu.com/) for quite a longtime now. I have +used [Debian](https://www.debian.org/) in the past and +[Manjaro](https://manjaro.org/). Also had [Arch](https://archlinux.org/) for +some time and even ran [Gentoo](https://www.gentoo.org/) way back. + +What I learned from all this is that I prefer running a bit older versions and +having them be stable than run bleeding edge rolling release. For that reason, I +stuck with Ubuntu for a couple of years now. I am also at a point in my life +where I just don't care what is cool or hip anymore. I just want a stable system +that doesn't get in my way. + +During all this, I noticed that these distributions were getting very bloated +and a lot of software got included that I usually uninstall on fresh +installation. Maybe this is my OCD speaking, but why do I have to give fresh +installation min 1 GB of ram out of the box just to have a blank screen in front +of me? I get it, there are many things included in the distro to make my life +easier. I understand. But at this point I have a feeling that modern Linux +distributions are becoming similar to [Node.js project with +node_modules](https://devhumor.com/content/uploads/images/August2017/node-modules.jpg). +Just a crazy number of packages serving very little or no purpose, just +supporting other software. + +I felt I needed a fresh start. To start over with something minimal and clean. +Something that would put a little more joy into using a computer again. + +For the first version, I wanted to target the following machines I have at home +that I want this thing to work on. + +```yaml +# My main stationary work machine +Resolution: 3840x1080 (Super Ultrawide Monitor 32:9) +CPU: Intel i7-8700 (12) @ 4.600GHz +GPU: AMD ATI Radeon RX 470/480/570/570X/580/580X/590 +Memory: 32020MiB +``` + +```yaml +# Thinkpad x220 for testing things and goofing around +Resolution: 1366x768 +CPU: Intel i5-2520M (4) @ 3.200GHz +GPU: Intel 2nd Generation Core Processor Family +Memory: 15891MiB +``` + +## How should I approach this? + +I knew I wanted to use [minimal Debian netinst +](https://www.debian.org/CD/netinst/) for the base to give myself a head +start. No reason to go through changing the installer and also testing all that +behemoth of a thing. So, some sort of ricing was the only logical option to get +this thing of the grounds somewhat quickly. + +> **What is ricing anyway?** +> The term “RICE” stands for Race Inspired Cosmetic Enhancement. A group of +> people (could be one, idk) decided to see if they could tweak their own +> distros like they/others did their cars. This gave rise to a community of +> Linux/Unix enthusiasts trying to make their distros look cooler and better +> than others... For more information, read this article +> [What in the world is ricing!?](https://pesos.github.io/2020/07/14/what-is-ricing.html). + +I didn't want this to just be a set of config files for theming purpose. I +wanted this to include a set of pre-installed tools and services that are being +used all the time by a modern developer. Theming is just a tiny part of it. +Fonts being applied across the distro and things like that. + +First, I choose terminal installer and left it to load additional components. +Avoid using graphical installer in this case. + +![](/assets/dfd-rice/install-00.png) + +After that I selected hostname and created a normal user and set password for +that user and root user and choose guided mode for disk partitioning. + +![](/assets/dfd-rice/install-01.png) + +I left it run to install all the things required for the base system and opted +out of scanning additional media for use by the package manager. Those will be +downloaded from the internet during installation. + +![](/assets/dfd-rice/install-02.png) + +I opted out of the popularity contest, and **now comes the important part**. +Uncheck all the boxes in Software selection and only leave 'standard system +utilities'. I also left an SSH server, so I was able to log in to the machine +from my main PC. + +![](/assets/dfd-rice/install-03.png) + +At this point, I installed GRUB bootloader on the disk where I installed the +system. + +![](/assets/dfd-rice/install-04.png) + +That concluded the installation of base Debian and after restarting the computer +I was prompted with the login screen. + +![](/assets/dfd-rice/install-05.png) + +Now that I had the base installation, it was time to choose what software do I +want to include in this so-called distribution. I wanted out of the box +developer experience, so I had plenty to choose. + +Let's not waste time and go through the list. + +## Desktop environments + +I have been using [Gnome](https://www.gnome.org/) for my whole Linux life. From +version 2 forward. It's been quite a ride. I hated version 3 when it came out +and replaced version 2. But I got used to it. And now with version 40+ they also +made couple of changes which I found both frustrating and presently surprised. + +The amount of vertical space you loose because of the beefy title bars on +windows is ridiculous. And then in case of +[Tilix](https://gnunn1.github.io/tilix-web/) you also have tabs, and you are +100px deep. Vertical space is one of the most important things for a +developer. The more real estate you have, the more code you can have in a +viewport. + +But on the other hand, I still love how Gnome feels and looks. I gotta give them +that. They really are trying to make Gnome feel unified and modern. + +Regardless of all the nice things Gnome has, I was looking at the tiling window +managers for some time, but never had the nerve to actually go with it. But now +was the ideal time to give it a go. No guts, no glory kind of a thing. + +One of the requirements for me was easy custom layouts because I use a really +strange monitor with aspect ratio of 32:9. So relying on included layouts most +of them have is a non-starter. + +What I was doing in Gnome was having windows in a layout like the diagram +below. This is my common practice. And if you look at it you can clearly see I +was replicating tiling window manager setup in Gnome. + +![](/assets/dfd-rice/layout.png) + +That made me look into a bunch of tiling window managers and then tested them +out. Candidates I was looking at were: + +- [i3](https://i3wm.org/) +- [bspwm](https://github.com/baskerville/bspwm) +- [awesome](https://awesomewm.org/index.html) +- [XMonad](https://xmonad.org/) +- [sway](https://swaywm.org/) +- [Qtile](http://www.qtile.org/) +- [dwm](https://dwm.suckless.org/) + +You can also check article [13 Best Tiling Window Managers for +Linux](https://www.tecmint.com/best-tiling-window-managers-for-linux/) I was +referencing while testing them out. + +While all of them provided what I needed, I liked i3 the most. What particular +caught my eye was the ease to use and tree based layouts which allows flexible +layouts. I know others can be set up also to have custom layouts other than +spiral, dwindle etc. I think i3 is a good entry-level window manager for +somebody like me. + +## Batteries included + +The source for the whole thing is located on Github +https://github.com/mitjafelicijan/dfd-rice. + +Currenly included: + +- `non-free` (enables non-free packages in apt) +- `sudo` (adds sudo and adds user to sudo group) +- `essentials` (gcc, htop, zip, curl, etc...) +- `wifi` (network manager nmtui) +- `desktop` (i3, dmenu, fonts, configurations) +- `pulseaudio` (pulseaudio with pavucontrol) +- `code-editors` (vim, micro, vscode) +- `ohmybash` (make bash pretty) +- `file-managers` (mc) +- `git-ui` (terminal git gui) +- `meld` (diff tool) +- `profiling` (kcachegrind, valgrind, strace, ltrace) +- `browsers` (brave, firefox, chromium) +- programming languages: + - `python` + - `golang` + - `nodejs` + - `rust` + - `nim` + - `php` + - `ruby` +- `docker` (with docker-compose) +- `ansible` + +Install script also allows you to install only specific packages (example for: +essentials ohmybash docker rust). + +```sh +su - root \ + bash -c "$(wget -q https://raw.github.com/mitjafelicijan/dfd-rice/master/tools/install.sh -O -)" -- \ + essentials ohmybash docker rust +``` + +Currently, most of these recipes use what Debian and this is totally fine with +me since I never use bleeding edge features of a package. But if something major +would come to light, I will replace it with a possible compilation script or +something similar. + +This is some of the output from the installation script. + +![](/assets/dfd-rice/script.png) + +Let's take a look at some examples in the installation script. + +### Docker recipe + +```sh +# docker +print_header "Installing Docker" +curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --yes --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg +echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null +apt update +apt -y install docker-ce docker-ce-cli containerd.io docker-compose + +systemctl start docker +systemctl enable docker +systemctl status docker --no-pager + +/sbin/usermod -aG docker $USERNAME +``` + +### Making bash pretty + +I really like [Oh My Zsh](https://ohmyz.sh/), but I don't like zsh shell. When +I used it, I constantly needed to be aware of it and running bash scripts was a +pain. So, I was really delighted when I found out that a version for bash +existed called [Oh My Bash](https://ohmybash.nntoan.com/). Let's take a look at +the recipe for installing it. + +```sh +# ohmybash +print_header "Enabling OhMyBash" +sudo -u $USERNAME sh -c "$(curl -fsSL https://raw.github.com/ohmybash/oh-my-bash/master/tools/install.sh)" & +T1=${!} +wait ${T1} +``` + +Because OhMyBash does `exec bash` at the end, this traps our script inside +another shell and our script cannot continue. For that reason, I executed this +in background. But that presents a new problem. Because this is executed in +background, we lose track of progress naturally. And that strange trick with +`T1=${!}` and `wait ${T1}` waits for the background process to finish before +continuing to another task in bash script. + +Check [Multi-Threaded Processing in Bash Scripts](https://www.cloudsavvyit.com/12277/how-to-use-multi-threaded-processing-in-bash-scripts/) +for more details. + +## Conclusion + +Take a look at +https://github.com/mitjafelicijan/dfd-rice/blob/develop/tools/install.sh script +to get familiar with it. This is just a first iteration and I will continue to +update it because I need this in my life. + +The current version boots in 4s to the login prompt, and after you log in, the +desktop environment loads in 2s. So, its fast, very fast. And on clean boot, I +measured ~230 MB of RAM usage. + +And this is how it looks with two terminals side by side. I really like the +simplicity and clean interface. I will polish the colors and stuff like that, +but I really do like the results. + +![](/assets/dfd-rice/desktop.png) diff --git a/content/posts/2021-12-25-running-golang-application-as-pid1.md b/content/posts/2021-12-25-running-golang-application-as-pid1.md new file mode 100644 index 0000000..10543f2 --- /dev/null +++ b/content/posts/2021-12-25-running-golang-application-as-pid1.md @@ -0,0 +1,348 @@ +--- +title: Running Golang application as PID 1 with Linux kernel +url: running-golang-application-as-pid1.html +date: 2021-12-25T12:00:00+02:00 +type: post +draft: false +--- + +## Unikernels, kernels, and alike + +I have been reading a lot about +[unikernernels](https://en.wikipedia.org/wiki/Unikernel) lately and found them +very intriguing. When you push away all the marketing speak and look at the +idea, it makes a lot of sense. + +> A unikernel is a specialized, single address space machine image constructed +> by using library operating systems. ([Wikipedia](https://en.wikipedia.org/wiki/Unikernel)) + +I really like the explanation from the article +[Unikernels: Rise of the Virtual Library Operating System](https://queue.acm.org/detail.cfm?id=2566628). +Really worth a read. + +If we compare a normal operating system to a unikernel side by side, they would +look something like this. + +![Virtual machines vs Containers vs Unikernels](/assets/pid1/unikernels.png) + +From this image, we can see how the complexity significantly decreases with +the use of Unikernels. This comes with a price, of course. Unikernels are hard +to get running and require a lot of work since you don't have an actual proper +kernel running in the background providing network access and drivers etc. + +So as a half step to make the stack simpler, I started looking into using +Linux kernel as a base and going from there. I came across this +[Youtube video talking about Building the Simplest Possible Linux System](https://www.youtube.com/watch?v=Sk9TatW9ino) +by [Rob Landley](https://landley.net) and apart from statically compiling the +application to be run as PID1 there was really no other obstacles. + +## What is PID 1? + +PID 1 is the first process that Linux kernel starts after the boot process. +It also has a couple of unique properties that are unique to it. + +- When the process with PID 1 dies for any reason, all other processes are + killed with KILL signal. +- When any process having children dies for any reason, its children are + re-parented to process with PID 1. +- Many signals which have default action of Term do not have one for PID 1. +- When the process with PID 1 dies for any reason, kernel panics, which + result in system crash. + +PID 1 is considered as an Init application which takes care of running other +and handling services like: + +- sshd, +- nginx, +- pulseaudio, +- etc. + +If you are on a Linux machine, you can check what your process is with PID 1 +by running the following. + +```sh +$ cat /proc/1/status +Name: systemd +Umask: 0000 +State: S (sleeping) +Tgid: 1 +Ngid: 0 +Pid: 1 +PPid: 0 +... +``` + +As we can see on my machine the process with id of 1 is [systemd](https://systemd.io/) +which is a software suite that provides an array of system components for Linux +operating systems. If you look closely you can also see that the `PPid` +(process id of the parent process) is `0` which additionally confirms that +this process doesn't have a parent. + +## So why even run application as PID 1 instead of just using a container? + +Containers are wonderful, but they come with a lot of baggage. And because they +are in their nature layered, the images require quite a lot of space and also a +lot of additional software to handle them. They are not as lightweight as they +seem, and many popular images require 500 MB plus disk space. + +The idea of running this as PID 1 would result in a significantly smaller footprint, +as we will see later in the post. + +> You could run a simple init system inside Docker container described more +> in this article [Docker and the PID 1 zombie reaping problem](https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/). + +## The master plan + +1. Compile Linux kernel with the default definitions. +2. Prepare a Hello World application in Golang that is statically compiled. +3. Run it with [QEMU](https://www.qemu.org/) and providing Golang application + as init application / PID 1. + +For the sake of simplicity we will not be cross-compiling any of it and just +use the 64bit version. + +## Compiling Linux kernel + +```sh +$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.7.tar.xz +$ tar xf linux-5.15.7.tar.xz + +$ cd linux-5.15.7 + +$ make clean + +# read more about this https://stackoverflow.com/a/41886394 +$ make defconfig + +$ time make -j `nproc` + +$ cd .. +``` + +At this point we have kernel image that is located in `arch/x86_64/boot/bzImage`. +We will use this in QEMU later. + +To make our lives a bit easier lets move the kernel image to another place. +Lets create a folder `bin/` in the root of our project with `mkdir -p bin`. + + +At this point we can copy `bzImage` to `bin/` folder with +`cp linux-5.15.7/arch/x86_64/boot/bzImage bin/bzImage`. + +The folder structure of this experiment should look like this. + +``` +pid1/ + bin/ + bzImage + linux-5.15.7/ + linux-5.15.7.tar.xz +``` + +## Preparing PID 1 application in Golang + +This step is relatively easy. The only thing we must have in mind that we will +need to compile the binary as a static one. + +Let's create `init.go` file in the root of the project. + +```go +package main + +import ( + "fmt" + "time" +) + +func main() { + for { + fmt.Println("Hello from Golang") + time.Sleep(1 * time.Second) + } +} +``` + +If you notice, we have a forever loop in the main, with a simple sleep of 1 +second to not overwhelm the CPU. This is because PID 1 should never complete +and/or exit. That would result in a kernel panic. Which is BAD! + +There are two ways of compiling Golang application. Statically and dynamically. + +To statically compile the binary, use the following command. + +```sh +$ go build -ldflags="-extldflags=-static" init.go +``` + +We can also check if the binary is statically compiled with: + +```sh +$ file init +init: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=Ypu8Zw_4NBxm1Yxg2OYO/H5x721rQ9uTPiDVh-VqP/vZN7kXfGG1zhX_qdHMgH/9vBfmK81tFrygfOXDEOo, not stripped + +$ ldd init +not a dynamic executable +``` + +At this point, we need to create [initramfs](https://www.linuxfromscratch.org/blfs/view/svn/postlfs/initramfs.html) +(abbreviated from "initial RAM file system", is the successor of initrd. It +is a cpio archive of the initial file system that gets loaded into memory +during the Linux startup process). + +```sh +$ echo init | cpio -o --format=newc > initramfs +$ mv initramfs bin/initramfs +``` + +The projects at this stage should look like this. + +``` +pid1/ + bin/ + bzImage + initramfs + linux-5.15.7/ + linux-5.15.7.tar.xz + init.go +``` + +## Running all of it with QEMU + +[QEMU](https://www.qemu.org/) is a free and open-source hypervisor. It emulates +the machine's processor through dynamic binary translation and provides a set +of different hardware and device models for the machine, enabling it to run a +variety of guest operating systems. + +```sh +$ qemu-system-x86_64 -serial stdio -kernel bin/bzImage -initrd bin/initramfs -append "console=ttyS0" -m 128 +``` + +```sh +$ qemu-system-x86_64 -serial stdio -kernel bin/bzImage -initrd bin/initramfs -append "console=ttyS0" -m 128 +[ 0.000000] Linux version 5.15.7 (m@khan) (gcc (GCC) 11.2.1 20211203 (Red Hat 11.2.1-7), GNU ld version 2.37-10.fc35) #7 SMP Mon Dec 13 10:23:25 CET 2021 +[ 0.000000] Command line: console=ttyS0 +[ 0.000000] x86/fpu: x87 FPU will use FXSAVE +[ 0.000000] signal: max sigframe size: 1440 +[ 0.000000] BIOS-provided physical RAM map: +[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable +[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved +[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved +[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007fdffff] usable +[ 0.000000] BIOS-e820: [mem 0x0000000007fe0000-0x0000000007ffffff] reserved +[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved +[ 0.000000] NX (Execute Disable) protection: active +[ 0.000000] SMBIOS 2.8 present. +[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.14.0-6.fc35 04/01/2014 +[ 0.000000] tsc: Fast TSC calibration failed +... +[ 2.016106] ALSA device list: +[ 2.016329] No soundcards found. +[ 2.053176] Freeing unused kernel image (initmem) memory: 1368K +[ 2.056095] Write protecting the kernel read-only data: 20480k +[ 2.058248] Freeing unused kernel image (text/rodata gap) memory: 2032K +[ 2.058811] Freeing unused kernel image (rodata/data gap) memory: 500K +[ 2.059164] Run /init as init process +Hello from Golang +[ 2.386879] tsc: Refined TSC clocksource calibration: 3192.032 MHz +[ 2.387114] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2e02e31fa14, max_idle_ns: 440795264947 ns +[ 2.387380] clocksource: Switched to clocksource tsc +[ 2.587895] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 +Hello from Golang +Hello from Golang +Hello from Golang +``` + +The whole [log file here](/assets/pid1/qemu.log). + +## Size comparison + +The cool thing about this approach is that the Linux kernel and the application +together only take around 12 MB, which is impressive as hell. And we need to +also know that the size of bzImage (Linux kernel) could be greatly decreased +by going into `make menuconfig` and removing a ton of features from the kernel, +making the size even smaller. I managed to get kernel size down to 2 MB and +still working properly. + +```sh +total 12M +-rw-r--r--. 1 m m 9.3M Dec 13 10:24 bzImage +-rw-r--r--. 1 m m 1.9M Dec 27 01:19 initramfs +``` + +## Creating ISO image and running it with Gnome Boxes + +First we need to create proper folder structure with `mkdir -p iso/boot/grub`. + +Then we need to download the [grub binary](https://github.com/littleosbook/littleosbook/raw/master/files/stage2_eltorito). +You can read more about this program on https://github.com/littleosbook/littleosbook. + +```sh +$ wget -O iso/boot/grub/stage2_eltorito https://github.com/littleosbook/littleosbook/raw/master/files/stage2_eltorito +``` + +```sh +$ tree iso/boot/ +iso/boot/ +├── bzImage +├── grub +│   ├── menu.lst +│   └── stage2_eltorito +└── initramfs +``` + +Let's copy files into proper folders. + + +```sh +$ cp stage2_eltorito iso/boot/grub/ +$ cp bin/bzImage iso/boot/ +$ cp bin/initramfs iso/boot/ +``` + +Lets create a GRUB config file at `nano iso/boot/grub/menu.lst` with contents. + +```ini +default=0 +timeout=5 + +title GoAsPID1 +kernel /boot/bzImage +initrd /boot/initramfs +``` + +Let's create iso file by using genisoimage: + +```sh +genisoimage -R \ + -b boot/grub/stage2_eltorito \ + -no-emul-boot \ + -boot-load-size 4 \ + -A os \ + -input-charset utf8 \ + -quiet \ + -boot-info-table \ + -o GoAsPID1.iso \ + iso +``` + +This will produce `GoAsPID1.iso` which you can use with [Virtualbox](https://www.virtualbox.org/) +or [Gnome Boxes](https://apps.gnome.org/app/org.gnome.Boxes/). + + + +## Is running applications as PID 1 even worth it? + +Well, the answer to this is not as simple as one would think. Sometimes it is +and sometimes it's not. For embedded systems and very specialized applications +it is worth for sure. But in normal uses, I don't think so. It was an interesting +exercise in compiling kernels and looking at the guts of the Linux kernel, +but sticking to containers for most of the things is a better option in my +opinion. + +An interesting experiment would be creating an image that supports networking +and could be deployed to AWS as an EC2 instance and observing how it fares. +But in that case, we would need to write some sort of supervisor that would +run on a separate EC2 that would check if other EC2 instances are running +properly. Remember that if your application fails, kernel panics and the +whole machine is inoperable in this case. + diff --git a/content/posts/2021-12-30-wap-mobile-web-before-the-web.md b/content/posts/2021-12-30-wap-mobile-web-before-the-web.md new file mode 100644 index 0000000..442943b --- /dev/null +++ b/content/posts/2021-12-30-wap-mobile-web-before-the-web.md @@ -0,0 +1,202 @@ +--- +title: Wireless Application Protocol and the mobile web before the web +url: wap-mobile-web-before-the-web.html +date: 2021-12-30T12:00:00+02:00 +type: post +draft: false +--- + +## A little stroll down the history lane + +About two weeks ago, I watched this outstanding documentary on YouTube +[Springboard: the secret history of the first real +smartphone](https://www.youtube.com/watch?v=b9_Vh9h3Ohw) about the history of +smartphones and phones in general. It brought back so many memories. I never had +an actual smartphone before the Android. The closest to smartphone was [Sony +Ericsson P1](https://www.gsmarena.com/sony_ericsson_p1-1982.php). A fantastic +phone and I broke it in Prague after a party and that was one of those rare +occasions where I was actually mad at myself. But nevertheless, after that +phone, the next one was an Android one. + +Before that, I only owned normal phones from Nokia and Siemens etc. Nothing +special, actually. These are the phones we are talking about. Before 2007. +Apple and Android phones didn't exist yet. + +These phones were rocking: + +- No selfie cameras. +- ~2 inch displays. +- ~120 MHz beast CPU's. +- 144p main cameras. +- But they had a headphone jack. + +Let's take a look at these beauties. + +![Old phones](/assets/wap/phones.gif) + +## WAP - Wireless Application Protocol + +Not that one! We are talking about Wireless Application Protocol and not Cardi +B's song 😃 + +WAP stands for Wireless Application Protocol. It is a protocol designed for +micro-browsers, and it enables the access of internet in the mobile devices. It +uses the mark-up language WML (Wireless Markup Language and not HTML), WML is +defined as XML 1.0 application. Furthermore, it enables creating web +applications for mobile devices. In 1998, WAP Forum was founded by Ericson, +Motorola, Nokia and Unwired Planet whose aim was to standardize the various +wireless technologies via protocols. +[(source)](https://www.geeksforgeeks.org/wireless-application-protocol/) + +WAP protocol was resulted by the joint efforts of the various members of WAP +Forum. In 2002, WAP forum was merged with various other forums of the industry, +resulting in the formation of Open Mobile Alliance (OMA). +[(source)](https://www.geeksforgeeks.org/wireless-application-protocol/) + +These were some wild times. Devices had tiny screens and data transmission rates +were abominable. But they were capable of rendering WML (Wireless Markup +Language). This was very similar to HTML, actually. It is a markup language, +after all. + +These pages could be served by [Apache](https://apache.org/) and could be +generated by CGI scripts on the backend. The only difference was the limited +markup language. + +## WML - Wireless Markup Language + +Just like web browsers use HTML for content structure, older mobile device +browsers use WML - if you need to support really old mobile phones using WML +browsers, you will need to know about it. WML is XML-based (an XML vocabulary +just like XHTML and MathML, but not HTML) and does not use the same metaphor as +HTML. HTML is a single document with some metadata packed away in the head, and +a body encapsulating the visible page. With WML, the metaphor does not envisage +a page, but rather a deck of cards. A WML file might have several pages or cards +contained within it. +[(source)](https://www.w3.org/wiki/Introduction_to_mobile_web) + +```html + + + + +

Welcome to the Example homepage

+ + +``` + +There is an amazing tutorial on [Tutorialpoint about +WML](https://www.tutorialspoint.com/wml/index.htm). + +## Converting Digg to WML + +This task is completely useless and not really feasible nowadays, but I had to +give it a try for old-time sake. Since the data is already there in a form of +RSS feed, I could take this feed and parse it and create a WML version of the +homepage. + +We will need: + +- Python3 + Pip +- ImageMagick +- feedparser and mako templating + +```sh +# for fedora 35 +sudo dnf install ImageMagick python3-pip + +# tempalting engine for python +pip install mako --user + +# for parsing rss feeds +pip install feedparser --user +``` + +Project folder structure should look like the following. + +``` +12:43:53 m@khan wap → tree -L 1 +. +├── generate.py +└── template.wml + +``` + +After that, I created a small template for the homepage. + +```html + + + + + + + + % for item in entries: +

${item.title}

+

${item.kicker}

+

${item.title}

+

${item.description}

+ % endfor + + + + +``` + +And the parser that parses RSS feed looks like this. + +```python +import os +import feedparser +from mako.template import Template + +os.system('mkdir -p www/images') + +template = Template(filename='template.wml') + +feed = feedparser.parse('https://digg.com/rss/top.xml') + +entries = feed.entries[:15] + +for entry in entries: + print('Processing image with id {}'.format(entry.id)) + os.system('wget -q -O www/images/{}.jpg "{}"'.format(entry.id, entry.links[1].href)) + os.system('convert www/images/{}.jpg -type Grayscale -resize 175x -depth 3 -quality 30 www/images/{}.jpg'.format(entry.id, entry.id)) + +html = template.render(entries = entries) + +with open('www/index.wml', 'w+') as fp: + fp.write(html) +``` + +This script will create a folder `www` and in the folder `www/images` for +storing resized images. + +> Be sure you don't use SSL and use just normal HTTP for serving the content. +> These old phones will have problems with TLS 1.3 etc. + +If you look at the python file, I convert all the images into tiny B&W images. +They should be WBMP (Wireless BitMaP) but I choose JPEGs for this, and it seems +to work properly. + +Because I currently don't have a phone old enough to test it on, I used an +emulator. And it was really hard to find one. I found [WAP +Proof](http://wap-proof.sharewarejunction.com/) on shareware junction, and it +did the job well enough. I will try to find and actual device to test it on. + + + +If you are using Nginx to serve the contents, add a directive to the hosts file +that will automatically server `index.wml` file. + +```nginx +server { + index index.wml index.html index.htm index.nginx-debian.html; +} +``` + +## Conclusion + +Well, this was pointless, but very fun! I hope you enjoyed it as much as I did. +I will try to find an old phone to test it on. If you have any questions, feel +free to ask in the comments. diff --git a/content/posts/2022-06-30-trying-out-helix-editor.md b/content/posts/2022-06-30-trying-out-helix-editor.md new file mode 100644 index 0000000..305d4b7 --- /dev/null +++ b/content/posts/2022-06-30-trying-out-helix-editor.md @@ -0,0 +1,53 @@ +--- +title: Trying out Helix code editor as my main editor +url: tying-out-helix-code-editor.html +date: 2022-06-30T12:00:00+02:00 +type: post +draft: false +--- + +I have been searching for a lightweight code editor for quite some time. One of +the main reasons was that I wanted something that doesn't burn through CPU and +RAM usage is not through the roof. I have been mostly using Visual Studio Code. +It's been an outstanding editor. I have no quarrel with it at all. It's just +time to spice life up with something new. + +I have been on this search for a couple of years. I have tried Vim, Neovim, +Emacs, Doom Emacs, Micro and couple more. Among most of them, I liked Micro and +Doom Emacs the most. Micro editor was a little too basic for me. And Doom Emacs +was a bit too hardcore. This does not reflect on any of the editors. It's just +my personal preference. + +> I tried Helix Editor about a year ago. But I didn't pay attention to it. +> Tried it and saw it's similar to Vi and just said no. I was premature to +> dismiss it. + +One of the things I actually miss is line wrapping for certain files. When +writing Markdown, line wrapping would be very helpful. Editing such a document +is frustrating to say the least. Some of the Markdown to HTML converters don't +take kindly of new lines between sentences. Not paragraphs, sentences. And I use +Markdown to write this blog you are reading. + +But other than this, I have been extremely satisfied by it. It's been a pleasant +surprise. There have been zero issues with the editor. + +One thing to do before you are able to use autocompletion and make use Language +Server support is to install the language server with NPM. + +```sh +npm install -g typescript typescript-language-server +``` + +I am still getting used to the keyboard shortcuts and getting better. What Helix +does really well is packing in sane defaults and even though because currently +there is no plugin support I haven't found any need for them. It has all that +you would need. It goes to extreme measures to show a user what is going on with +popups that show you what the keyboard shortcuts are. + +And it comes us packed with many +[really good themes](https://github.com/helix-editor/helix/wiki/Themes). + +![Editor](/assets/helix-editor/editor.png) + +It's still young but has this mature feeling to it. It has sane defaults and +mimics Vim (works a bit differently, but the overall idea is similar). diff --git a/content/posts/2022-07-05-what-would-dna-sound-if-synthesized.md b/content/posts/2022-07-05-what-would-dna-sound-if-synthesized.md new file mode 100644 index 0000000..968341c --- /dev/null +++ b/content/posts/2022-07-05-what-would-dna-sound-if-synthesized.md @@ -0,0 +1,364 @@ +--- +title: What would DNA sound if synthesized to an audio file +url: what-would-dna-sound-if-synthesized.html +date: 2022-07-05T12:00:00+02:00 +type: post +draft: false +--- + +## Introduction + +Lately, I have been thinking a lot about the nature of life, what are the +foundation blocks of life and things like that. It's remarkable how complex and +on the other hand simple the creation is when you look at it. The miracle of +life keeps us grounded when our imagination goes wild. If the DNA are the blocks +of life, you could consider them to be an API nature provided us to better +understand all of this chaos masquerading as order. + +I have been reading a lot about superintelligence and our somehow misguided path +to create general artificial intelligence. What would the building blocks or our +creation look like? Is the compression really the ultimate storage of +information? Will our creation also ponder this questions when creating new +worlds for themselves, or will we just disappear into the vastness of +possibilities? It is a little offensive that we are playing God whilst being +completely ignorant of our own reality. Who knows! Like many other +breakthroughs, this one will also come at a cost not known to us when it finally +happens. + +To keep things a bit lighter, I decided to convert some popular DNA sequences +into an audio files for us to listen to. I am not the first one, nor I will be +the last one to do this. But it is an interesting exercise in better +understanding the relationship between art and science. Maybe listening to DNA +instead of parsing it will find a way into better understanding, or at least +enjoying the creation and cryptic nature of life. + +## DNA encoding and primer example + +I have been exploring DNA in the past in my post from about 3 years ago in +[Encoding binary data into DNA +sequence](/encoding-binary-data-into-dna-sequence.html) where I have been +converting all sorts of data into DNA sequences. + +This will be a similar exercise but instead of converting to DNA, I will be +generating tones from Nucleotides. + +| Nucleotides | Note | Frequency | +| ---------------- | ---- | --------- | +| **A** (Adenine) | A | 440 Hz | +| **C** (Cytosine) | C | 783.99 Hz | +| **G** (Guanine) | G | 523.25 Hz | +| **T** (Thymine) | D | 587.33 Hz | + +Since we do not have T in equal-tempered scale, I choose D to represent T note. + +You can check [Frequencies for equal-tempered scale, A4 = 440 +Hz](https://pages.mtu.edu/~suits/notefreqs.html). For this tuning, we also +choose `Speed of Sound = 345 m/s = 1130 ft/s = 770 miles/hr`. + +Now that we have this out of the way, we can also brush up on the DNA sequencing +a bit. This is a famous quote I also used for the encoding tests, and it goes +like this. + +> How wonderful that we have met with a paradox. Now we have some hope of +> making progress. +> ― Niels Bohr + +```shell +>SEQ1 +GACAGCTTGTGTACAAGTGTGCTTGCTCGCGAGCGGGTACGCGCGTGGGCTAACAAGTGA +GCCAGCAGGTGAACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGCTGGCGGGTGA +ACAAGTGTGCCGGTGAGCCAACAAGCAGACAAGTAAGCAGGTACGCAGGCGAGCTTGTCA +ACTCACAAGATCGCTTGTGTACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGTAT +GCTTGCTGGCGGACAAGCCAGCTTGTAAGCGGACAAGCTTGCGCACAAGCTGGCAGGCCT +GCCGGCTCGCGTACAAATTCACAAGTAAGTACGCTTGCGTGTACGCGGGTATGTATACTC +AACCTCACCAAACGGGACAAGATCGCCGGCGGGCTAGTATACAAGAACGCTTGCCAGTAC +AACC +``` + +This is what we gonna work with to get things rolling forward, when creating +parser and waveform generator. + +## Parsing DNA data + +This step is rather simple one. All we need to do is parse input DNA sequence in +[FASTA format](https://en.wikipedia.org/wiki/FASTA_format) well known in +[Bioinformatics](https://en.wikipedia.org/wiki/Bioinformatics) to extract single +Nucleotides that will be converted into separate tones based on equal-tempered +scale explained above. + +```python +nucleotide_tone_map = { + 'A': 440, + 'C': 523.25, + 'G': 783.99, + 'T': 587.33, # converted to D +} + +def split(word): + return [char for char in word] + +def generate_from_dna_sequence(sequence): + for nucleotide in split(sequence): + print(nucleotide, nucleotide_tone_map[nucleotide]) +``` + +## Generating sine wave + +Because we are essentially creating a long stream of notes we will be appending +sine notes to a global array we will later use for creating a WAV file out of +it. + +```python +import math + +def append_sinewave(freq=440.0, duration_milliseconds=500, volume=1.0): + global audio + + num_samples = duration_milliseconds * (sample_rate / 1000.0) + + for x in range(int(num_samples)): + audio.append(volume * math.sin(2 * math.pi * freq * (x / sample_rate))) + + return +``` + +The sine wave generated here is the standard beep. If you want something more +aggressive, you could try a square or saw tooth waveform. + +## Generating a WAV file from accumulated sine waves + + +```python +import wave +import struct + +def save_wav(file_name): + wav_file = wave.open(file_name, 'w') + nchannels = 1 + sampwidth = 2 + + nframes = len(audio) + comptype = 'NONE' + compname = 'not compressed' + wav_file.setparams((nchannels, sampwidth, sample_rate, nframes, comptype, compname)) + + for sample in audio: + wav_file.writeframes(struct.pack('h', int(sample * 32767.0))) + + wav_file.close() +``` + +44100 is the industry standard sample rate - CD quality. If you need to save on +file size, you can adjust it downwards. The standard for low quality is, 8000 or +8kHz. + +WAV files here are using short, 16 bit, signed integers for the sample size. +So, we multiply the floating-point data we have by 32767, the maximum value for +a short integer. + +> It is theoretically possible to use the floating point -1.0 to 1.0 data +> directly in a WAV file, but not obvious how to do that using the wave module +> in Python. + +## Generating Spectograms + +I have tried two methods of doing this and both were just fine. I however opted +out to use the [SoX - Sound eXchange, the Swiss Army knife of audio +manipulation](https://linux.die.net/man/1/sox) one because it didn't require +anything else. + +```shell +sox output.wav -n spectrogram -o spectrogram.png +``` + +An example spectrogram of Ludwig van Beethoven Symphony No. 6 First movement. + + + +![Ludwig van Beethoven Symphony No. 6 First movement](/assets/dna-synthesized/symphony-no6-1st-movement.png) + +The other option could also be in combination with +[gnuplot](http://www.gnuplot.info/). This would require an intermediary step, +however. + +```shell +sox output.wav audio.dat +tail -n+3 audio.dat > audio_only.dat +gnuplot audio.gpi +``` + +And input file `audio.gpi` that would be passed to gnuplot looks something like +this. + +``` +# set output format and size +set term png size 1000,280 + +# set output file +set output "audio.png" + +# set y range +set yr [-1:1] + +# we want just the data +unset key +unset tics +unset border +set lmargin 0 +set rmargin 0 +set tmargin 0 +set bmargin 0 + +# draw rectangle to change background color +set obj 1 rectangle behind from screen 0,0 to screen 1,1 +set obj 1 fillstyle solid 1.0 fillcolor rgbcolor "#ffffff" + +# draw data with foreground color +plot "audio_only.dat" with lines lt rgb 'red' +``` + +## Pre-generated sequences + +What I did was take interesting parts from an animal's genome and feed it to a +tone generator script. This then generated a WAV file and I converted those to +MP3, so they can be played in a browser. The last step was creating a +spectrogram based on a WAV file. + +### Niels Bohr quote + + + +![Spectogram](/assets/dna-synthesized/quote/spectogram.png) + +### Mouse + +This is part of a mouse genome `Mus_musculus.GRCm39.dna.nonchromosomal`. You +can get [genom data +here](http://ftp.ensembl.org/pub/release-106/fasta/mus_musculus/dna/). + + + +![Spectogram](/assets/dna-synthesized/mouse/spectogram.png) + +### Bison + +This is part of a bison genome `Bison_bison_bison.Bison_UMD1.0.cdna`. You can +get [genom data +here](http://ftp.ensembl.org/pub/release-106/fasta/bison_bison_bison/cdna/). + + + +![Spectogram](/assets/dna-synthesized/bison/spectogram.png) + +### Taurus + +This is part of a taurus genome `Bos_taurus.ARS-UCD1.2.cdna`. You can get +[genom data +here](http://ftp.ensembl.org/pub/release-106/fasta/bos_taurus/cdna/). + + + +![Spectogram](/assets/dna-synthesized/taurus/spectogram.png) + +## Making a drummer out of a DNA sequence + +To make things even more interesting, I decided to send this data via MIDI to my +[Elektron Model:Samples](https://www.elektron.se/en/model-samples). This is a +really cool piece of equipment that supports MIDI in via USB and 3.5 mm audio +jack. + +Elektron is connected to my MacBook via USB cable and audio out is patched to a +Sony Bluetooth speaker I have that supports 3.5 mm audio in. Elektron doesn't +have internal speakers. + +![](/assets/dna-synthesized/elektron/IMG_0619.jpg) + +![](/assets/dna-synthesized/elektron/IMG_0620.jpg) + +![](/assets/dna-synthesized/elektron/IMG_0622.jpg) + +For communicating with Elektron, I choose `pygame` Python module that has MIDI +built in. With this, it was rather simple to send notes to the device. All I did +was map MIDI notes to the actual Nucleotides. + +Before all of this I also checked Audio MIDI Setup app under MacOS and checked +MIDI Studio by pressing ⌘-2. + +![](/assets/dna-synthesized/elektron/midi-studio.jpg) + +The whole script that parses and send notes to the Elektron looks like this. + +```python +import pygame.midi +import time + +pygame.midi.init() + +print(pygame.midi.get_default_output_id()) +print(pygame.midi.get_device_info(0)) + +player = pygame.midi.Output(1) +player.set_instrument(2) + +def send_note(note, velocity): + global player + player.note_on(note, velocity) + time.sleep(0.3) + player.note_off(note, velocity) + + +nucleotide_midi_map = { + 'A': 60, + 'C': 90, + 'G': 160, + 'T': 180, # is D +} + +with open("quote.fa") as f: + sequence = f.read().replace('\n', '') + +for nucleotide in [char for char in sequence]: + print("Playing nucleotide {} with MIDI note {}".format( + nucleotide, nucleotide_midi_map[nucleotide])) + send_note(nucleotide_midi_map[nucleotide], 127) + +del player +pygame.midi.quit() +``` + + + +All of this could be made much more interesting if I choose different +instruments for different Nucleotides, or doing more funky stuff with Elektron. +But for now, this should be enough. It is just a proof of concept. Something to +play around with. + +## Going even further + +As you probably notice, the end results are quite similar to each other. This is +to be expected because we are operating only with 4 notes essentially. What +could make this more interesting is using something like +[Supercollider](https://supercollider.github.io/) to create more interesting +sounds. By transposing notes or using effects based on repeated data in a +sequence. Possibilities are endless. + +It is really astonishing what can be achieved with a little bit of code and an +idea. I could see this becoming an interesting background soundscape instrument +if done properly. It could replace random note generator with something more +intriguing, biological, natural. + +I actually find the results fascinating. I took some time and listened to this +music of nature. Even though it's quite the same, it's also quite different. +The subtle differences on repeat kind of creates music on its own. Makes you +wonder. It kind of puts Occam’s Razor in its place. Nature for sure loves to +make things as energy efficient as possible. diff --git a/content/posts/2022-08-13-algae-spotted-on-river-sava.md b/content/posts/2022-08-13-algae-spotted-on-river-sava.md new file mode 100644 index 0000000..ba2dd2b --- /dev/null +++ b/content/posts/2022-08-13-algae-spotted-on-river-sava.md @@ -0,0 +1,31 @@ +--- +title: Aerial photography of algae spotted on river Sava +url: aerial-photography-of-algae-spotted-on-river-sava.html +date: 2022-08-13T12:00:00+02:00 +type: post +draft: false +--- + +This is a bit of a different post than I usually write, but quite interesting +one to me. River Sava has plenty of hydropower plants located down the stream. +This makes regulating the strength of a current easier than normally. Because of +lower stream strength and high temperatures, algae has formed on the river. +This is the first time I've seen something like this in my whole life. + +Below are some photographs taken from a DJI drone capturing the event. + +![Algae on Sava](/assets/algae-sava/dji-algae-0.jpg) + +![Algae on Sava](/assets/algae-sava/dji-algae-1.jpg) + +![Algae on Sava](/assets/algae-sava/dji-algae-2.jpg) + +![Algae on Sava](/assets/algae-sava/dji-algae-3.jpg) + +![Algae on Sava](/assets/algae-sava/dji-algae-4.jpg) + +![Algae on Sava](/assets/algae-sava/dji-algae-5.jpg) + +I will try to get more photos of this in the future days and if something +intriguing shows up will post it again on the blog. + diff --git a/content/posts/2022-10-06-state-of-web-technologies-in-year-2022.md b/content/posts/2022-10-06-state-of-web-technologies-in-year-2022.md new file mode 100644 index 0000000..e5a0b74 --- /dev/null +++ b/content/posts/2022-10-06-state-of-web-technologies-in-year-2022.md @@ -0,0 +1,304 @@ +--- +title: State of Web Technologies and Web development in year 2022 +url: state-of-web-technologies-and-web-development-in-year-2022.html +date: 2022-10-06T12:00:00+02:00 +type: post +draft: false +--- + +## Initial thoughts + +*This post is a critique on the current state of web development. It is an +opinionated post! I will learn more about this in the future, and probably +slightly change my mind about some of the things I criticize.* + +I have started working on a hobby project about two weeks ago, and I wanted to +use that situation as a learning one. Trying new things, new technologies, new +tools. I always considered myself to be an adventurous person when it comes to +technology. I never shy away from trying new languages, new operating systems +etc. Likewise, I find the whole experience satisfying, and it tickles that part +of my brain that finds discovery the highest of the mountains to climb. + +What I always wanted to make was a coding game, that you would play in a browser +(just to eliminate building binaries for each operating system) where you would +level up your character and go into these scriptable battles. You know, RPG +elements. + +So, the natural way to go would be some sort of SPA (single page application) +with basic routing and some state management. Nothing crazy. + +> **Before we move on**, I have to be transparent. Take my views on this with +> a grain of salt. I have only scratched the surface with these technologies, +> and my knowledge is full of gaps. This is my experience using some of these +> products for the first time or in a limited capacity. + +Having this out of the way, I got myself a fresh pot of coffee and down the +rabbit hole I went. + +## Giving React JS a spin + +I first tried [React JS](https://reactjs.org/). I kind of like it. Furthermore, +I have worked with libraries like this in the past and also wrote a couple of +them (nothing compared to that level), but I had the basic understanding of what +was going on. I rolled up a project quickly and had basic things done in a +matter of two hours, which was impressive. + +I prefer using [Tailwind CSS](https://tailwindcss.com/) for my styling +pleasures, and integrating that was also a painless experience. It was actually +nice to see that some things got better with time. In about 2 minutes I got +Tailwind working, and I was able to use classes at my disposal. All that +`postcss` stuff was taken care of by adding a couple of things in config files +(all described really well in their documentation). + +It is not that different from Vue which I have had more encounters with in the +past People will probably call me a lunatic for saying this. But you know, it is +the truth. Same same, but different. I still believe that using libraries like +this is beneficial. I am not a JavaScript purist. They all have their quirks, +but at the end of the day, I truly believe it’s worth it. + +## Bundlers and Transpilers + +I still reject calling [Typescript](https://www.typescriptlang.org/) to +[JavaScript](https://www.javascript.com/) conversion a "compilation process". I +call them [transpilers](https://devopedia.org/transpiler), and I don’t care! 😈 + +And if you want to fight this, take a look at this little chart and be mad at +it! + +![Compiling vs Transpiling](/assets/state-of-web/compiling-vs-transpiling.png) + +The first one that I ever used was [webpack](https://webpack.js.org/), and it +was an absolute horrific experience. Saying this, it is an absolutely fantastic +tool. I felt more like a config editor than actually a programmer. To be fair, +I am a huge fan of [make](https://www.gnu.org/software/make/), and you can do as +you wish with this information. I like my build systems simple. + +Also, isn’t it interesting that we need something like +[Babel](https://babeljs.io/) to make JavaScript code work in a browser that has +only one client side scripting available, which is by no accident also +JavaScript. Why? I know why it’s needed, but seriously, why. + +I haven’t used Babel for years now. Or if I did, it was packaged together by +some other bundler thingy. Which does not make things better, but at least I +didn’t need to worry about it. + +I really don’t like complicated build systems. I really don’t like abstracting +code and making things appear magical. The older I get, the more I appreciate +clear and clean, expressive code. No one-liners, if possible. + +But I have to give props to [Vite](https://vitejs.dev/)! This was one of the +best developer experiences I have ever had. Granted, it still has magical +properties. And yes, it still is a bundler and abstracts things to the nth +degree. But at least it didn’t force me to configure 700 lines of JSON. And I +know that this makes me a hypocrite. You can’t have it all. Nonetheless, my +reasoning here is, if using bundlers is inevitable, then at least they should +provide an excellent developer experience. + +I also noticed that now the catch-all phrase is “blazingly fast” and “lightning +fast” and “next generation” and stuff like that. I mean, yeah, tools should get +faster with time. But saying that starting a project now takes 2 seconds instead +of 20 seconds is something that is a break it or make it kind of a deal is +ridiculous. I don’t mind waiting a couple of seconds every couple of days. I +also don’t create 700 projects every day, and also who does? This argument has +no bite. All I want is a decent reload time (~100ms is more than good enough for +me) and that is it. + +You don’t need to sell me benefits if I only get them when I start a fresh +project, and then try to convince me that this is somehow changing the fate of +the universe. First of all, it is not. And second, if this is your only argument +for your tool, I would advise you to maybe re-focus your efforts to something +else. Vite says that startup times are really fast. And if that would be the +only thing differentiating it from other tools, I would ignore it. But it has +some really compelling features like [Hot Module +Replacement](https://www.geeksforgeeks.org/reactjs-hot-module-replacement/) that +really works well. It was a joy to use. + +So, I will be definitely using Vite in the future. + +## Jam Stack, Mach Stack no snack + +Let's get a couple of the acronyms out of the way, so we all know what we are +talking about: + +- Jam Stack - JavaScript, API and Markup +- Mach Stack - Microservices, API-first, Cloud-Native SaaS, Headless + +It is so hard to follow all these new trendy things happening around you, that +it makes you have a massive **FOMO** all the time. But on the other hand, you +also don’t want to be that old fart that doesn’t move with the times and still +writes his trusty jQuery code while listening to Blink 182 All the small things +on full blast. It’s a good song, don’t get me wrong, but there are other songs +out there. + +I have to admit. [Vercel](https://vercel.com/) is really cool! Love the +simplicity of the service. You could compare it to +[Netlify](https://www.netlify.com/). I haven’t tried Netlify extensively, but +from a couple of experimental deployments I still prefer Vercel. It is much more +streamlined, but maybe this is bias in me. I really like Vercel’s Analytics, +which give you a [Core Web Vitals report](https://web.dev/vitals/) in their +admin console. Kind of cool, I’m not going to lie. + +This whole idea about frontend and backend merging into [SSR (server-side +rendering)](https://www.debugbear.com/blog/server-side-rendering) looks so good +on paper. It almost doesn’t come with any major flaws. + +But when it comes to the actual implementation, there is much to be desired. +I’m going to lump [Next.js](https://nextjs.org/) and +[Nuxt.js](https://nuxtjs.org/) together because they are essentially the same +thing, just a different library. + +Now comes the reality. Mixing backend and frontend in this manner creates this +weird mental model where you kind of rely on magical properties of these +libraries. You relinquish control over to them for better developer experience. +But is that really true? Initially, I was so stoked about it. However, the more +I used them, the more I felt uncomfortable. I felt dirty, actually. Maybe this +is because I come from old ways of doing things where you control every step of +request, and allowing something to hijack it feels like blasphemy. + +More than that, some pretty significant technical issues arose from this. How do +you do JWT token authentication? You put it in `api` folder and then do some +fetching and storing into local state management. But doing this also requires +some tinkering with await/async stuff on the React/Vue side of things. And then +you need to write middleware for it. And the more I look at it, the more I see +that this whole thing was not meant to be used like this, and it all feels and +looks like a huge hack. + +The issue I have with this is that they over-promise and under-deliver. They +want to be an all-in-one replacement for everything, and they don’t deliver on +this promise. And how could they?! We have to be fair. It is an impossible task. + +They sell you [NoOps](https://www.geeksforgeeks.org/overview-of-noops/), but +when you need to accomplish something a little bit more out of the scope of +Hello World, you have to make hacky decisions to make it work. And having a +deployment strategy that relies on many moving parts is never a good idea. +Abstracting too much is usually a sign of bad architecture. + +Lately, this has become a huge trend that will for sure bite us in the future. +And let’s not get it twisted. By doing this, PaaS providers like +[AWS](https://aws.amazon.com/), [GCS](https://cloud.google.com/), etc. obscure +their billing, and you end up paying more than you really should. And even if +that is not an issue, it comes down to the principle of things. AWS is known for +having multiple “currencies“ inside their projects like write operations, read +operations, etc. which add up, and it creates this impossible to track billing +scheme. It all behaves suspiciously like a pay-to-win game you could find on +mobile phones that scams you out of your money. + +And as far as I am concerned, the most important thing was me not coding the +functionalities for the game I want to make. I was battling libraries and cloud +providers. How to deploy, what settings are relevant. Bad documentation or +multiple versions of achieving the same thing. You are getting bombarded by all +this information, and you don’t really have any control over it. +Production-ready code becomes a joke, essentially. Especially if you tend to +work on that project for a prolonged period of time. + +All of these options end up creating a fatigue. What to choose, what not to +choose. Unnecessary worrying about if the stack will still be deemed worthy in +six months. There is elegance in simplicity. + +> JavaScript UI frameworks and libraries work in cycles. Every six months or +> so, a new one pops up, claiming that it has revolutionized UI development. +> Thousands of developers adopt it into their new projects, blog posts are +> written, Stack Overflow questions are asked and answered, and then a newer +> (and even more revolutionary) framework pops up to usurp the throne. +> — Ian Allen + +![To many options](/assets/state-of-web/2008-vs-2020.png) + +And this jab at these libraries and cloud providers is not done out of malice. +It is a real concern that I have about them. In my life, I have seen +technologies come and go, but the basics always stick around. So surrendering +all the power you have to a library or a cloud provider is in my opinion a +stupid move. + +## Tailwind CSS still rocks! + +You know, many people say negative things about Tailwind. And after a lot of +deliberation, I came to the conclusion that Tailwind is good for two types of +developers. Tailwind is good for a complete noob or a senior developer. A +complete noob doesn’t really care about inner workings of CSS, and a senior +developer also doesn’t care about CSS. Well, at least, not anymore. And +developers in between usually have the biggest issues with it. Not always of +course, but in a lot of cases. + +I like the creature comforts of Tailwind. Being utility first would make me +argue that it is actually more similar to [Sass](https://sass-lang.com/) or +[Less](https://lesscss.org/) than something like Bootstrap. Not technically, but +ideologically. After I started using it, I never looked back. I use it every +time I need to do something web related. + +Writing CSS for general things feels like going several steps back. Instead of +focusing on what you are actually trying to achieve, you focus on notations like +[BEM](https://en.bem.info/methodology/css/), code structuring, optimizing HTML +size. Just doing things that make 0.1% difference. You know that saying: Early +optimization is the root of all evil. Exactly that. + +I am also not saying that Tailwind is the cure for everything. Sometimes custom +CSS is necessary. But from what I found out in using it for almost two years in +a production environment (on a site getting quite a lot of traffic and +constantly being changed), I can say without any reservations that Tailwind +saved our asses countless times. We would be rewriting CSS all the time without +it. And I don’t really think writing CSS is the best way to spend my time. + +I have also noticed that people who criticize Tailwind the most never actually +used it in a real project that has a long lifetime with plenty of changes that +will happen in the future. + +But you know, whatever floats your boat! + +## Code maintainability + +Somehow, people also stopped talking about maintenance. If you constantly try to +catch the latest and greatest train, you are by that logic always trying new +things. Which is a good thing if you want to learn about technologies and try +them. But for the production environment, you have to have a stable stack that +doesn’t change every 6 months. + +You can lock dependencies for sure. Nevertheless, the hype train moves along +anyway. And the mindset this breeds goes against locking the code. This +bleeding-edge rolling release cycle is not helping. That is why enterprise +solutions usually look down on these popular stacks and only do bare minimum to +appear hip and cool. + +With that said, I still think that progress is good, but should be taken with a +grain of salt. If your project is something that should be built once and then +rarely updated, going with the latest stack is a possible way to go. But, if you +are working on a project that lasts for years, you should probably approach it +with some level of caution. Web development is often times too volatile. + +## Web development has a marketing issue + +I noticed that almost every project now has this marketing spin put on it. +Everything is blazingly fast now. I get it, they are competing for your +attention, but what happened to just being truthful and not inflating reality. + +And in order to appeal to mass market, they leave things out of their marketing +materials. These open-source projects are now behaving more and more like +companies do. Which is a scary thought on its self. + +And we are also seeing a rise in a concept of building a company in the open, +which is a good thing, don't get me wrong. But when it is using open-source to +lure people and then lock them in their ecosystem, there is where I have issues +with it. + +This might be because I have been using GNU/Linux for 20 years now and have been +so beholden for my success to open-source that I see issues when open-source is +being used to trick people into a false sense of security that these projects +are built in the spirit of open-source. Because there is a difference. They are +NOT! They have a really specific goal in mind. And the open-source is being used +as a delivery system. Which is in my opinion disgusting! + +## Conclusion + +I will end my post with this. Web development is running now in circles. People +are discovering [RPC](https://www.tutorialspoint.com/remote-procedure-call-rpc) +now and this is the now the next big thing. [GraphQL](https://graphql.org/) is +so passé. And I am so tired of it all. Of blazingly fast libraries, of all these +new technologies that are actually just a remake of old ones. Of just the +general spirit of the web. I will just use what I already know. Which worked 10 +years ago and will work 10 years after this. I will adopt a couple of little +tools like Vite. But I will not waste my time on this anymore. + +It was a good exercise to get in touch with what’s new now. Nothing really +changed that much. FOMO is now cured! Now I have to get my ass back to actually +code and make the project that I wanted to make in the first place. + diff --git a/content/posts/2022-10-16-that-sound-that-machine-makes-when-struggling.md b/content/posts/2022-10-16-that-sound-that-machine-makes-when-struggling.md new file mode 100644 index 0000000..47a6212 --- /dev/null +++ b/content/posts/2022-10-16-that-sound-that-machine-makes-when-struggling.md @@ -0,0 +1,66 @@ +--- +title: Microsoundtrack — That sound that machine makes when struggling +url: that-sound-that-machine-makes-when-struggling.html +date: 2022-10-16T12:00:00+02:00 +type: post +draft: false +--- + +A couple of months ago, I got an idea about micro soundtracks. In this concept, +you are the observer, director, and audience in this tiny movies. + +What you do is to attempt to imagine what would be happening around you based on +a title of the song and let the song help you fill the void in your story. + +I made these songs is Logic Pro X. Every year or so I do this kind of thing and +make a couple of songs similar to this. But this is the first time I am posting +about it. + +You can listen to the whole set on +[Youtube](https://www.youtube.com/watch?v=_5oXBhSmF3c) or scroll down the page +and there are embedded players for each song. + +## A bunch of inter-dimensional people with loud clocks + +A group of inter-dimensional people are going up and down the elevator with you +while having loud clocks around their necks. Each clock ticks on a different +frequency. A lot of other sounds are getting drawn into your dimension, +resulting in a strange merging of dimensions. + + + +## Two black holes conversing about the weather + +You are a traveler in a spaceship flying very close to two colliding black holes +having a discussion about the weather while tearing each other apart. During all +this your ship is getting pulled into the event horizon of both black holes, +putting a lot of strain on your spaceship. + + + +## A planet where every organism is a plant + +You land on a planet where every living organism is a plant and among those +plants some of them are highly intelligent, and you were asked to make first +contact with the native species. Your visit takes place in a giant cave where +you are meeting these plants, and they are talking to you. + + + +## Bio implants having a fit and reprogramming your brain + +In a distant future where everybody has bio implants, you have just received +your first one, which happens to be a brain implant. Something goes wrong, and +your implant is starting to misbehave, and you are experiencing brain +malfunctions. You are on the streets at night a couple of hours after your +procedure. You can feel your sanity breaking down. + + + +## Cow animation + +I also made this little cow animation. Go into full screen to see the effects in +more details. + + + diff --git a/content/posts/2023-01-26-trying-to-build-a-new-kind-of-terminal-emulator.md b/content/posts/2023-01-26-trying-to-build-a-new-kind-of-terminal-emulator.md new file mode 100644 index 0000000..27e227a --- /dev/null +++ b/content/posts/2023-01-26-trying-to-build-a-new-kind-of-terminal-emulator.md @@ -0,0 +1,253 @@ +--- +title: Trying to build a New kind of terminal emulator for the modern age +url: trying-to-build-a-new-kind-of-terminal-emulator.html +date: 2023-01-26T12:00:00+02:00 +type: post +draft: false +--- + +Over the past few weeks, I have been really thinking about terminal emulators, +how we interact with computers, the separation of text-based programs and GUI +ones. To be perfectly honest, I got pissed off one evening when I was cleaning +up files on my computer. Normally, I go into console and do `ncdu` and check +where the junk is. Then I start deleting stuff. Without any discrimination, +usually. But when it comes to screenshots, I have learned that it's good to keep +them somewhere near if I need to refer to something that I was doing. I am an +avid screenshot taker. So at that point I checked Pictures folder and also did a +basic search `find . -type f -name "*.jpg"` for all the JPEG files in my home +directory and immediately got pissed off. Why can’t I see thumbnails in my +terminal? I know why, but why in the year of 2022 this is still a problem. I am +used to traversing my disk via terminal. I am faster, and I am more comfortable +this way. But when it comes to visualization, I then need to revert to GUI +applications and again find the same file to see it. I know that programs like +`feh` and `sxiv` are available, but I would just like to see the preview. Like +[Jupyter notebook](https://jupyter.org/) or something similar. Just having it +inline. Part of a result. + +It also didn’t help that I was spending some time with the [Plan +9](https://plan9.io/plan9/) Operating system. More specifically +[9FRONT](http://9front.org/). The way that [ACME editor](http://acme.cat-v.org/) +handles text editing is just wonderful. Different and fresh somehow, even though +it’s super old. + +So, I went on a lookout for an interesting way of visualizing results of some +query. I found these applications to be outstanding examples of how not to be a +captive of a predetermined way of doing things. + +- [Wolfram Mathematica](https://www.wolfram.com/mathematica/) +- [Jupyter notebooks](https://jupyter.org/) +- [Plan 9 / 9FRONT](http://www.9front.org) +- [Temple OS](https://templeos.org/) +- [Emacs](https://www.gnu.org/software/emacs/) + +My idea is not as out there as ACME is, but it is a spin on the terminal +emulators. I like the modes that Vi/Vim provides you with. I like the way the +Emacs does its own `M-x` `M-c`. Furthermore, I really like how Mathematica and +Jupyter present the data in a free flowing form. And I love how Temple OS is +basically a C interpreter on some level. + +> **Note:** This is part 1 of the journey. Nowhere finished yet. I am just +> tinkering with this at the moment. This whole thing can easily spectacularly +> fail. + +So I started. I knew that I wanted to have the couple of modes, but I didn’t +like the repetition of keystrokes, so the only option was to have some sort of +toggle and indicate to the user that they are in a special mode. Like Vi does +for Normal and Visual mode. + +These modes would for the first version be: + +- *Preview mode* (toggle with Ctrl + P) + - When this mode would be enabled, the `ls` command would try to find images + from the results and display thumbnails from them in the terminal itself. + No ASCII art. Proper images. In a grid! +- *Detach mode* (toggle with Ctrl + D) + - When this mode would be enabled, every command would open a new window + and execute that command in it. This would be useful for starting `htop` + in a separate window. + +The reason for having these modes togglable is to not ask for previews every +time. You enable a mode and until you disable it, it behaves that way. Purely +out of ergonomic reasons. + +I would like to treat every terminal I open as a session mentally. When I start +using the terminal, I start digging deeper into the issue I am trying to +resolve. And while I am doing this, I would like to open detached windows +etc. A lot of these things can be done easily with something like +[i3](https://i3wm.org/), but also that pull you out of the context of what you +were doing. I would like to orchestrate everything from one single point. + +In planning for this project, I knew that I would need to use a language like C +and a library such as [SDL2](https://www.libsdl.org/) in order to achieve the +desired results. I had considered other options, but ultimately determined that +[SDL2](https://www.libsdl.org/) was the best fit based on its capabilities and +reputation in the programming community. + +At first, I thought the idea of a hardware accelerated terminal was a bit of a +joke. It seemed like such a niche and unnecessary feature, especially given the +fact that terminal emulators have been around for decades and have always relied +on software rendering. But to be fair, [Alacritty](https://alacritty.org/) is +doing the same thing. Well, they are doing a remarkable job at it. + +So, I embarked on a journey. Everything has to start somewhere. For me, it +started with creating a window! It has to start somewhere. 🙂 + +```c +// Oh, Hi Mark! +// Create the window, obviously. +SDL_Window *window = SDL_CreateWindow( + WINDOW_TITLE, SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, + WINDOW_WIDTH, WINDOW_HEIGHT, + SDL_WINDOW_RESIZABLE | SDL_WINDOW_OPENGL | SDL_WINDOW_SHOWN); +``` + +I continued like this to get some text displayed on the screen. + +I noted that +[`TTF_RenderText_Solid`](https://wiki.libsdl.org/SDL_ttf/TTF_RenderText_Solid) +rendered text really poorly. There were no antialiasing at all. In my wisdom, I +never checked the documentation. Well, that was a fail. To uneducated like me: +`TTF_RenderText_Solid` renders Latin1 text at fast quality to a new 8-bit +surface. So, that's why the texts looked like shit. No wonder. + +Remarks on `TTF_RenderText_Solid`: This function will allocate a new 8-bit, +palettized surface. The surface's 0 pixel will be the colorkey, giving a +transparent background. The 1 pixel will be set to the text color. + +After I replaced it with +[`TTF_RenderText_LCD`](https://wiki.libsdl.org/SDL_ttf/TTF_RenderText_LCD) which +renders Latin1 text at LCD subpixel quality to a new ARGB surface, the text +started looking good. Really make sure you read the documentation. It’s actually +good. As a side note, you can find all the documentation regarding [SDL2 on +their Wiki](https://wiki.libsdl.org/). + +After that was done, I started working on displaying other things like `Preview` +and `Detach` modes. This wasn’t really that hard. In SDL2 you can check all the +available events with `while (SDL_PollEvent(&event) > 0)` and have a bunch of +switch statements to determine which key is currently being pressed. More about +keys, [SDLKey](https://documentation.help/SDL/sdlkey.html) and mroe about +pooling the events on +[SDL_PollEvent](https://documentation.help/SDL/sdlpollevent.html). + +```c +while (SDL_PollEvent(&event) > 0) +{ + switch (event.type) + { + case SDL_QUIT: + running = false; + break; + + case SDL_TEXTINPUT: + if (!meta_key_pressed) + { + strncat(input_prompt_text, event.text.text, 1); + update_input_prompt = true; + } + break; + } +} +``` + +After that was somewhat working correctly, I started creating a struct that +would hold all the commands and results and I call them Cells. Yes, I stole that +naming idea from Jupyter. + +```c +typedef struct +{ + char *command; + char *result; + SDL_Surface *surface; + SDL_Texture *texture; + SDL_Rect rect; +} Cell; +``` + +I am at a place now where I am starting to implement scrolling. This will for +sure be fun to code. Memory management in C is super easy. 😂 + +I have also added a simple [INI file like +configuration](https://en.wikipedia.org/wiki/INI_file) support. It is done in an +[STB style of +header](https://github.com/nothings/stb/blob/master/docs/stb_howto.txt) and maps +to specific options supported by the terminal. It is not universal, and the code +below demonstrates how I will use it in the future. + +```c +#ifndef CONFIG_H +#define CONFIG_H + +/* +# This is a comment + +# This is the first configuration option +dettach=value11111 + +# This is the second configuration option +preview=value22222 + +# This is the third configuration option +debug=value33333 +*/ + +// Define a struct to hold the configuration options +typedef struct +{ + char dettach[256]; + char preview[256]; + char debug[256]; +} Config; + +// Read the configuration file and return the options as a struct +extern Config read_config_file(const char *filename) +{ + // Create a struct to hold the configuration options + Config config = {0}; + + // Open the configuration file + FILE *file = fopen(filename, "r"); + + // Read each line from the file + char line[256]; + while (fgets(line, sizeof(line), file)) + { + // Check if this line is a comment or empty + if (line[0] == '#' || line[0] == '\n') + continue; + + // Parse the line to get the option and value + char option[128], value[128]; + if (sscanf(line, "%[^=]=%s", option, value) != 2) + continue; + + // Set the value of the appropriate option in the config struct + if (strcmp(option, "dettach") == 0) + { + strncpy(config.option1, value, sizeof(config.option1)); + } + else if (strcmp(option, "preview") == 0) + { + strncpy(config.option2, value, sizeof(config.option2)); + } + else if (strcmp(option, "debug") == 0) + { + strncpy(config.option3, value, sizeof(config.option3)); + } + } + + // Close the configuration file + fclose(file); + + // Return the configuration options + return config; +} + +#endif +``` + +This is as far as I managed to get for now. I have a daily job and this +prohibits me to work on these things full time. But I should probably get back +and finish this. At least have a simple version working out, so I can start +testing it on my machines. Fingers crossed. 🕵️‍♂️ + diff --git a/content/posts/2023-05-16-rekindling-my-love-for-programming.md b/content/posts/2023-05-16-rekindling-my-love-for-programming.md new file mode 100644 index 0000000..3c2267b --- /dev/null +++ b/content/posts/2023-05-16-rekindling-my-love-for-programming.md @@ -0,0 +1,74 @@ +--- +title: Rekindling my love for programming and enjoying the act of creating +url: rekindling-my-love-for-programming.html +date: 2023-05-16T12:00:00+02:00 +type: post +draft: false +--- + +Programming can be a challenging and rewarding experience, but sometimes it's +easy to feel burnt out or disinterested. I have lost the passion for coding over +the past couple of months and it looked like I will never enjoy the coding as +much as I did. + +I was feeling burnt out with programming. I thought taking a break from it and +focusing on other activities that I enjoy might be helpful. This way, I could +come back to programming with a fresh perspective and renewed energy. I also +thought about learning a new programming language or technology to keep things +interesting and challenging. + +However, what I didn't realize was that learning a new language or technology +wasn't going to solve the underlying issue. I needed to take a step back and +re-evaluate why I had lost my passion for programming in the first place. This +involved taking a deep look into what I was doing that resulted in this rut. + +Sometimes, it's easy to get caught up in the hype of new technologies or +languages, and we can feel like we're missing out if we're not constantly +learning and experimenting. However, it's important to remember that the latest +and greatest isn't always the best fit for our projects or our +interests. Instead of constantly chasing the next big thing, it can be helpful +to focus on what truly interests us and what we're passionate about. This can +help us stay motivated and engaged with our work, rather than feeling like we're +just going through the motions. + +I expressed that I had lost my passion for coding over the past couple of +months, and I realized that the reason behind it was my tendency to spread +myself too thin and not focus on completing interesting projects. In order to +regain my passion for coding, I need to focus on projects that truly interest me +and give me a sense of purpose and motivation. + +Recently, I have been playing World of Warcraft more frequently and have become +interested in developing addons for the game. + +This quickly resulted in me creating three addons that improve the quality of +life, and I subsequently developed a more useful add-on that encapsulates all +the others I made. + +I found it interesting that this action sparked a new interest in me. +Additionally, I discovered the Lua language, which reminded me that coding +should be fun rather than just a struggle with a language. It should be pure, +unadulterated fun. + +I wasn't fighting the syntax, nor was I focused on finding the most optimal +solution. I simply created things without the pressure of making them the best +they could possibly be. + +This made me realize that I actually adore simple languages that get out of the +way and let you express what you want to do. It forced me to rethink a lot about +what I use and what I actually enjoy. + +I have decided to stick to the basics. For a scripting language, I will use +Lua. For networking, I will use Golang. And for any special needs, I will rely +on C. I do not require Rust, Nim, or Zig. This selection is more than sufficient +for my needs. I have to stay true to this simplicity. There is something to the +Occam's Razor. + +I've been struggling with a lack of creativity lately, but now I'm experiencing +a real change. I realized I needed to take a step back and stop actively trying +to address the issue. I needed to stop worrying and overthinking it. I simply +needed some time. Looking back, I don't think I've taken any significant time +off in the last 10 years. + +Suddenly, I find myself with the energy and passion to complete multiple small +projects. It doesn't feel like a chore at all. Who knew I needed WoW to +kickstart everything. Inspiration really does come from the strangest places. diff --git a/content/posts/2023-05-22-crafting-stories-in-zed-editor.md b/content/posts/2023-05-22-crafting-stories-in-zed-editor.md new file mode 100644 index 0000000..dc22e95 --- /dev/null +++ b/content/posts/2023-05-22-crafting-stories-in-zed-editor.md @@ -0,0 +1,88 @@ +--- +title: From General Zod to Superman - Crafting Stories in Zed Editor +url: crafting-stories-in-zed-editor.html +date: 2023-05-22T12:00:00+02:00 +type: post +draft: false +--- + +Pretentious title! Good start! I have nothing to add to this discussion. I just +like this editor and wanted to write something here that will remind me to use +it again in a while when/if it becomes available for Linux. + +**TLDR:** I think this code editor is very cool and has a massive potential. I +hope they don’t mess up with adding a plugin ecosystem to it! + +Out of morbid curiosity, I started using the [Zed editor](https://zed.dev/) on +my Mac. Zed is a high-performance, multiplayer code editor developed by the +creators of Atom and Tree-sitter. Written in Rust so it has to be blazingly +fast! 😊 It's a joke, calm down. + +Over the past year, I have switched between [Helix +editor](https://helix-editor.com/) and [VS +Code](https://code.visualstudio.com/), but for the last couple of months, I have +been using Helix exclusively. + +I've been genuinely impressed by Zed. When you open a file, it automatically +detects its type and downloads the corresponding [LSP (language +server)](https://en.wikipedia.org/wiki/Language_Server_Protocol). The list of +supported languages is not extensive, but it's still impressive. It's a great +example of how to create a product that stays out of your way. + +![Zed editor](/assets/zed/zed-1.png?style=bigimg) + +For C development it downloaded [clangd](https://clangd.llvm.org/) and setting +up missing dependencies in code was rather easy. For this project I use +[SDL2](https://www.libsdl.org/) for rendering terminal emulator. It’s a hobby +project, don’t worry about it. + +If you are going to give this a try and you are using C, I suggest checking two +files in the root of your project folder. If you don't have them, create them. + +**compile_flags.txt** + +``` +-I/opt/homebrew/include +-I/opt/homebrew/include/SDL2 +``` + +Easy way of checking what the appropriate includes for a specific library is to +use `pkg-config` and in my case `pkg-config SDL2 --cflags-only-I`. But this is +nothing new to C/C++ devs. Just a noter for people who are using Visual Studio. + +**.clang-format** + +``` +ColumnLimit: 220 +BasedOnStyle: Mozilla +``` + +I prefer Mozilla coding style for C so you can set that up. + +They really have something special here. Although there is no version available +for Linux yet, I will stick to Helix. This impressive piece of engineering is, +above all, an amazing example of craftsmanship. + +They have a bunch of amazing integrated functionalities like live desktop +sharing, code sharing in a live coding session. There is a lot of pretentious +marketing speak there but the product is still amazing! + +For me the speed and the simplicity of the product was the most impressive +thing. You get that: it just works feeling. A rare thing in 2023. + +![Zed editor](/assets/zed/zed-2.png?style=bigimg) + +They also managed to add [Github Copilot](https://github.com/features/copilot) +in a non obtrusive way. To me, everything feels very intentional and +specifically selected. It's minimal yet maximally effective. + + + +It is a perfect balance between VS Code, Jetbrains IDE’s and something like VIM +or Helix. + +I just hope they **DON’T** add plugin support and keep it like it is. They as a +vendor should add stuff to it with great deliberation and thought. And this way +the product will stay fast and focused. That’s my two cents. + +Amazing job! diff --git a/content/posts/2023-05-23-i-was-wrong-about-git-workflows.md b/content/posts/2023-05-23-i-was-wrong-about-git-workflows.md new file mode 100644 index 0000000..e82f50b --- /dev/null +++ b/content/posts/2023-05-23-i-was-wrong-about-git-workflows.md @@ -0,0 +1,71 @@ +--- +title: I think I was completely wrong about Git workflows +url: i-was-wrong-about-git-workflows.html +date: 2023-05-23T12:00:00+02:00 +type: post +draft: false +tags: [] +--- + +I have been using some approximation of [Git +Flow](https://jeffkreeftmeijer.com/git-flow/) for years now and never really +questioned it to be honest. When I create a repo I create develop branch and set +it as default one and then merge to master from there. Seems reasonable enough. + +One thing that I have learned is that long living branches are the devil. They +always end up making a huge mess when they need to be merged eventually into +master. So by that reason, what is the develop branch if not the longest living +feature branch. And from my personal experience there was never a situation +where I wasn’t sweating bullets when I had to merge develop back to master. + +This realisation started to give me pause. So why the hell am I doing this, and +is there a better way. Well the solution was always there. And it comes in a +form of [git tags](https://git-scm.com/book/en/v2/Git-Basics-Tagging). + +So what are git tags? Git tags are references to specific points in a Git +repository's history. They are used to mark important milestones, such as +releases or significant commits, making it easier to identify and access +specific versions of a project. + +Somehow we have all hijacked the meaning of the master branch that it has to be +the most releasable version of code. And this is also where the confusing about +versioning the software kicks in. Because master branch implicitly says that we +are dealing with the rolling release type of a software. And by having a develop +branch we are hacking around this confusion. With a separation of develop and +master we lock functionalities into place and forcing a stable vs development +version of the software. + +But if that is true and the long living branches are the devil then why have +develop at all. I think that most of this comes to how continuous integration is +being done. There usually is no granular access to tags and CD software deploys +what is present on a specific branch, may that be master for production and +develop for staging. This is a gross simplification and by having this in place +we have completely removed tagging as a viable option to create a fix point in +software cycle that says, this is the production ready code. + +One cool thing about tags are that you can checkout a specific tag. So they +behave very similarly as branches in that regard. And you don’t have the +overhead of having two mainstream branches. + +So what is the solution? One approach is to use development workflow, where all +changes are made on the smaller branches and continuously merged into +master. Where the software is ready to be pushed to production you tag the +master branch. This approach eliminates the need for long-lived branches and +simplifies the development process. It also encourages developers to make small, +incremental changes that can be tested and deployed quickly. However, this +approach may not be suitable for all projects or teams that heavily rely on +automated deployment based on branch names only. + +This also requires that developers always keep production in mind. No more +living on an island of the develop branch. All your actions and code need to be +ready to meet production standards on a much smaller timescale. + +I think that we have complicated the workflow in an honest attempt to make +things more streamlined but in the process of doing this, we have inadvertently +made our lives much more complicated. + +In conclusion, it's important to re-evaluate our workflows from time to time to +see if they still make sense and if there are better alternatives available. +Long-living branches can be problematic, and using tags to mark important +milestones can simplify the development process. + diff --git a/content/posts/2023-05-31-re-inventing-task-runner-that-i-actually-used-daily.md b/content/posts/2023-05-31-re-inventing-task-runner-that-i-actually-used-daily.md new file mode 100644 index 0000000..fd44605 --- /dev/null +++ b/content/posts/2023-05-31-re-inventing-task-runner-that-i-actually-used-daily.md @@ -0,0 +1,159 @@ +--- +title: "Re-Inventing Task Runner That I Actually Used Daily" +url: re-inventing-task-runner-that-i-actually-used-daily.html +date: 2023-05-31T12:21:10+02:00 +type: post +draft: false +--- + +Couple of months ago I had this brilliant idea of re-inventing the wheel by +making an alternative for make. And so I went. Boldly into the battle. And to my +big surprise my attempt resulted in not a completely useless piece of software. + +My initial requirements were quite simple but soon grow into something more +ambitious. And looking back I should have stuck to the simple version. My +laziness was on my side this time though. Because I haven’t implemented some of +the features I now realise I really didn’t need them and they would bog the +whole program and make it be something it was never meant to be. + +My basic requirements were following: + +- Syntax should be a tiny bit inspired by Rake and Rakefiles. +- Should borrow the overall feel of a unit test experience. +- Using something like Python would be a bit of an overkill. +- The program must be statically compiled, so it can run on same architecture + without libc, musl dependencies or things like that. +- Install ruby for rake is a bit overkill and can not be done with certain + really lightweight distributions like Alpine Linux. This tool would be usable + on such lightweight systems for remote debugging. +- I want to use it for more than just compiling things. I want to use it as an + entry-point into a project, and I want this to help me indirectly document the + project as well. +- It should be an abstraction over bash shell or the default system shell. + - Each task essentially becomes its own shell instance. +- Must work on Linux and macOS systems. +- By default, running `erd` list all the available tasks (when I use make, I + usually put a disclaimer that you should check Makefile to see all available + target). +- Should support passing arguments when you run it from a shell. +- Normal variable as the same as environmental variables. There is no + distinction. Every variable is also essentially an environment variable and + can be used by other programs. +- State between tasks is not shared, and this makes this “pure” shell instances. +- Should be single-threaded for the start and later expanded with `@spawn` + command. +- Variables behave like macros and are preprocessed before evaluation. +- Should support something like `assure` that would check if programs like C + compiler or Python (whatever the project requires) are installed on a machine. + +Quite a reasonable list of requirements. I do this things already in my +Makefiles or/and Bash scripts. But I would like to avoid repeating myself every +time I start working on something new. + +So I started with the following syntax. + +```ruby +@env on + +# Override the default shell. +@shell /bin/bash + +# Assure that program is installed. +@assure docker-compose pip python3 + +# Load local dotenv files (these are then globally available). +@dotenv .env +@dotenv .env.sample +@dotenv some_other_file + +# This are local variables but still accessible in tasks. +@var HI = "hey" +@var TOKEN = "sometoken" +@var EMAIL = "m@m.com" +@var PASSWORD = "pass" +@var EDITOR = "vim" + +@task dev "Test chars .:'}{]!//" does + echo "..." $HI +end + +@task clean "Cleans the obj files" does + rm .obj +end + +@task greet "Greets the user" does + echo "Hi user $TOKEN or $WINDOWID $EMAIL" +end + +@task stack "Starts Docker stack" does + docker-compose -f stack.yml up +end + +@task todo "Shows all todos in source files and count them" does + grep -ir "TODO|FIXME" . | wc -l +end + +@task test1 "For testing 1" does + unknown-command + echo "test1" + ls -lha +end + +@task test2 "For testing 2" does + echo "test1" + ls -lha + docker-compose -f samples/stack.yml up +end +``` + +One thing that I really like about Errand. Yes, this is what it is called. And +it is available at https://git.mitjafelicijan.com/errand.git/about/. Moving +on. One thing that I really like is that a task is a persistent shell. By that I +mean, that the whole task, even if it contains multiple command in one shell. +In make each line in a target is that and you need to combine lines or add `\` +at the end of the line. + +```bash +# How you do this things in make. +target: + source .venv/bin/activate \ + python script.py +``` + +This solves this problem. Consider each task and what is being executed in that +task a shell that will only close when all the tasks are completed. + +By self-documenting I mean that if you are in a directory with `Errandfile` in, +if you only type `erd` and press enter it should by default display all the +possible targets. In make i was doing this by having a first target be something +like `default` that echos the message “Check Makefile for all available target.” +Because all of the tasks in Errand require a message I use that to display let’s +call it table of contents. + +Because I don’t use any external dependencies this whole thing can be statically +compiled. So that also checked one of the boxes. + +It works on Linux and on a Mac so that’s also a bonus. I don’t believe this +would work on Windows machines because of the way that I use shell instances. By +you could use something like Windows Subsystem for Linux and run it in +there. That is a valid option. + +To finish this essay off, how was it to use it in “real life”. I have to be +honest. Some of the missing features still bother me. `@dotenv` directive is +still missing and I need to implement this ASAP. + +Another thing that needs to happen is support for streaming output. Currently +commands like `docker-compose` that runs in foreground mode is not compatible +with Errand. So commands that stream output are an issue. I need to revisit how +I initiate shell and how I read stdout and stderr. But that shouldn’t be a +problem. + +I have been very satisfied with this thing. I am pleasantly surprised by how +useful it is. I really wanted to test this in the wild before I commit to it. I +have more abandoned project than Google and it’s bringing a massive shame to my +family at this point. So I wanted to be sure that this is even useful. And it +actually is. Quite surprised at myself. + +I really need to package this now and write proper docs. And maybe rewrite +tokeniser. Its atrocious right now. Site to behold! But that is an issue for +another time. diff --git a/content/posts/2023-07-01-bringing-all-of-my-projects-together-under-one-umbrella.md b/content/posts/2023-07-01-bringing-all-of-my-projects-together-under-one-umbrella.md new file mode 100644 index 0000000..9059b00 --- /dev/null +++ b/content/posts/2023-07-01-bringing-all-of-my-projects-together-under-one-umbrella.md @@ -0,0 +1,281 @@ +--- +title: "Bringing all of my projects together under one umbrella" +url: bringing-all-of-my-projects-together-under-one-umbrella.html +date: 2023-07-01T18:49:07+02:00 +type: post +draft: false +--- + +## What is the issue anyway? + +Over the years, I have accumulated a bunch of virtual servers on my +[DigitalOcean](https://www.digitalocean.com/) account for small experimental +projects I dabble in. And this has resulted in quite a bill. I mean, I wouldn't +care if these projects were actually being used. But there were just being there +unused and wasting resources. Which makes this an unnecessary burden for me. + +Most of them are just small HTML pages that have an endpoint or two to read data +from or to, and for that reason I wrote servers left and right. To be honest, +all of those things could have been done with [CGI +scripts](https://en.wikipedia.org/wiki/Common_Gateway_Interface) and that would +have been more than enough. + +Recently, I decided to stop language hopping and focus on a simpler stack which +includes C, Go and Lua. And I can accomplish all the things I am interested in. + +## Finding a web server replacement + +Usually I had [Nginx](https://nginx.org/en/) in front of these small web servers +and I had to manage SSL certificates and all that jazz. I am bored with these +things. I don't want to manage any of this bullshit anymore. + +So the logical move forward was to find a solid alternative for this. I have +ended up on [Caddy server](https://caddyserver.com/). I've used it in the past +but kind of forgotten about it. What I really like about it is an ease of use +and a bunch of out of the box functionalities that come with it. + +These are the _pitch_ points from their website: + +- **Secure by Default**: Caddy is the only web server that uses HTTPS by + default. A hardened TLS stack with modern protocols preserves privacy and + exposes MITM attacks. +- **Config API**: As its primary mode of configuration, Caddy's REST API makes + it easy to automate and integrate with your apps. +- **No Dependencies**: Because Caddy is written in Go, its binaries are entirely + self-contained and run on every platform, including containers without libc. +- **Modular Stack**: Take back control over your compute edge. Caddy can be + extended with everything you need using plugins. + +I had just a few requirements: + +- Automatic SSL +- Static file server +- Basic authentication +- CGI script support + +And the vanilla version does all of it, but CGI scripts. But that can easily be +fixed with their modular approach. You can do this on their website and build a +custom version of the server, or do it with Docker. + +This is a `Dockerfile` I used to build a custom server. + +```Dockerfile +FROM caddy:builder AS builder + +RUN xcaddy build \ + --with github.com/aksdb/caddy-cgi + +FROM caddy:latest +RUN apk add --no-cache nano + +COPY --from=builder /usr/bin/caddy /usr/bin/caddy +``` + +## Getting rid of all the unnecessary virtual machines + +The next step was to get a handle on the number of virtual servers I have all +over the place. + +I decided to move all the projects and services into two main VMs: + +- personal server (still Nginx) + - git server + - static file server + - personal blog +- projects server (Caddy server) + - personal experiments + - other projects + +I will focus on projects' server in this post since it's more interesting. + +## Testing CGI scripts + +The first thing I tested was how CGI scripts work under Caddy. This is +particularly import to me because almost all of my experiments and mini projects +need this to work. + +To configure Caddy server, you must provide the server with a configuration +file. By default, it's called `Caaddyfile`. + +```caddyfile +{ + order cgi before respond +} + +examples.mitjafelicijan.com { + cgi /bash-test /opt/projects/examples/bash-test.sh + cgi /tcl-test /opt/projects/examples/tcl-test.tcl + cgi /lua-test /opt/projects/examples/lua-test.lua + cgi /python-test /opt/projects/examples/python-test.py + + root * /opt/projects/examples + file_server +} +``` + +- The order is very important. Make sure that `order cgi before respond` is at + the top of the configuration file. +- Also, when you run with Caddy v2, make sure you provide `adapter` argument + like this `/usr/bin/caddy run --watch --environ --config /etc/caddy/Caddyfile + --adapter caddyfile`. Otherwise, Caddy will try to use a different format for + config file. + +I did a small batch of tests with [Bash](https://www.gnu.org/software/bash/), +[Tcl](https://www.tcl-lang.org/), [Lua](https://www.lua.org/) and +[Python](https://www.python.org/). Here is a cheat sheet if you need it. + +Let's get Bash out of the way first. + +```bash +#!/usr/bin/bash + +printf "Content-type: text/plain\n\n" + +printf "Hello from Bash\n\n" +printf "PATH_INFO [%s]\n" $PATH_INFO +printf "QUERY_STRING [%s]\n" $QUERY_STRING +printf "\n" + +for i in {0..9..1}; do + printf "> %s\n" $i +done + +exit 0 +``` + +This one is for Tcl script. + +```tcl +#!/usr/bin/tclsh + +puts "Content-type: text/plain\n" + +puts "Hello from Tcl\n" +puts "PATH_INFO \[$env(PATH_INFO)\]" +puts "QUERY_STRING \[$env(QUERY_STRING)\]" +puts "" + +for {set i 0} {$i < 10} {incr i} { + puts "> $i" +} +``` + +And for all you Python enjoyers. + +```python +#!/usr/bin/python3 + +import os + +print("Content-type: text/plain\n") + +print("Hello from Python\n") +print("PATH_INFO [{}]".format(os.environ['PATH_INFO'])) +print("QUERY_STRING [{}]".format(os.environ['QUERY_STRING'])) +print("") + +for i in range(10): + print("> {}".format(i)) +``` + +And for the final example, Lua. + +```lua +#!/usr/bin/lua + +print("Content-type: text/plain\n") + +print("Hello from Lua\n") +print(string.format("PATH_INFO [%s]", os.getenv("PATH_INFO"))) +print(string.format("QUERY_STRING [%s]", os.getenv("QUERY_STRING"))) +print() + +for i = 0, 9 do + print(string.format("> %d", i)) +end +``` + +## Basic authentication + +One thing was also to have an option for some sort of authentication, and +something like [Basic access +authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) would +be more than enough. + +Thankfully, Caddy supports this out of the box already. Below is an updated +example. + +```Caddyfile +{ + order cgi before respond +} + +examples.mitjafelicijan.com { + cgi /bash-test /opt/projects/examples/bash-test.sh + cgi /tcl-test /opt/projects/examples/tcl-test.tcl + cgi /lua-test /opt/projects/examples/lua-test.lua + cgi /python-test /opt/projects/examples/python-test.py + + root * /opt/projects/examples + file_server + + basicauth * { + bob $2a$14$/wCgaf9oMnmQa20txB76u.nI1AldGMBT/1J7fXCfgOiRShwz/JOkK + } +} +``` + +`basicauth *` matches everything under this domain/sub-domain and protects it +with Basic Authentication. + +- `bob` is the username +- `hash` is the password + +To generate these passwords, execute `caddy hash-password` and this will prompt +you to insert a password twice and spit out a hashed password that you can put +in your configuration file. + +Restart the server and you are ready to go. + +## Making Caddy a service with systemd + +After the tests were successful, I copied `caddy` to `/usr/bin/caddy` and copied +`Caddyfile` to `/etc/caddy/Caddyfile`. + +Now off to the systemd. Each systemd service requires you to create a service +file. + +- I created a `/etc/systemd/system/caddy.service` and put the following content + in the file. + +```systemd +[Unit] +Description=Caddy +Documentation=https://caddyserver.com/docs/ +After=network.target network-online.target +Requires=network-online.target + +[Service] +Type=notify +User=root +Group=root +ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile --adapter caddyfile +ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force --adapter caddyfile +TimeoutStopSec=5s +LimitNOFILE=1048576 +LimitNPROC=512 +PrivateTmp=true +ProtectSystem=full +AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE + +[Install] +WantedBy=multi-user.target +``` + +- You might need to reload systemd with `systemctl daemon-reload`. +- Then I enabled the service with `systemctl enable caddy.service`. +- And then I started the service with `systemctl start caddy.service`. + +This was about all that I needed to do to get it running. Now I can easily add +new subdomains and domains to the main configuration file and be done with +it. No manual Let's Encrypt shenanigans needed. diff --git a/content/posts/2023-07-08-who-knows-what-the-world-will-look-like-tomorrow.md b/content/posts/2023-07-08-who-knows-what-the-world-will-look-like-tomorrow.md new file mode 100644 index 0000000..46e6167 --- /dev/null +++ b/content/posts/2023-07-08-who-knows-what-the-world-will-look-like-tomorrow.md @@ -0,0 +1,99 @@ +--- +title: "Who knows what the world will look like tomorrow" +url: who-knows-what-the-world-will-look-like-tomorrow.html +date: 2023-07-08T18:49:07+02:00 +type: post +draft: false +--- + +This site has gone through a lot of changes over the years. From being written +in Flask and Bottle to moving on to static site generators. I have used and +tested probably 10s of them my now. From homebrew solutions to the biggest and +the baddest. From Bash scripts to Node.js disasters. I've seen some things, no +doubt. Not all bad. + +I'have been closely observing the web and where the trends are going, and I +don't like what I see. Instead of internet being this weird place where +experimentation is happening, it all became stale and formulized. Boring, +actually. Really boring. And sad. Where is that old, revolutionary FU spirit I +remember? It's still there, I know. But it's being drowned by the voices of +mediocrity and formulaic boredom. + +It almost feels like that the internet stopped for 10 years and only now +something has started happening. With all the insanity around the world. People +hating people without actual reasons, just because it's fashionable to hate and +crowd is saying so. Sad state of affairs. + +All this is contributing to this overall negativity masked as apathy. Everybody +walking in lockstep. Instead of being creative are bold, we are just +re-inventing the world and making the same mistakes. Maybe, just maybe, some +things are good enough and there is no need to try to be too smart for our own +good. After N-attempts, maybe something should click inside our heads to maybe +say: "This thing, opinion, etc. is actually really good, and even after several +attempts it still holds." + +The older I get, the more careful I am of my own thoughts and why I think the +way I think. More and more, I try to understand people with opposite +opinions. Far from perfect, but closer to bearable. And then I see people +hearing or reading a thing on internet and let's fucking goooooo! Strong +opinions are a sign of a weak and uneducated mind. I am more and more sure of +this. + +It's gotten to a point where you can with great certainty deduce a person's +personality based on one or two opinions. How boring have we become. No wonder +people can't talk to each other. These would be very quick conversations anyway. + +I just got remembered of a song, "Hi Ren". The ending talks about being stiff +and not being able to dance. Such an amazing metaphor. And we as people have +gone so far, we can't even walk or even crawl normally anymore. We have +forgotten that the most beautiful things in life have a great deal of +uncertainty about them. We want instant gratification. Not only that, but we +want absolute obedience. Complete control over others, because we have zero +control of ourselves. And all the lies we could tell ourselves will not help us +in this situation. + +It is funny how I catch myself from time to time being a complete idiot. It's +like having an outer body experience. I can see myself being an idiot, and +cannot stop myself. It serves as a learning lesson to stop before speaking. To +think before saying. And to crawl before walking. + +So there is still time. We can dance once more. All we need to do is stop for a +second. Me and you. Us two is a start. Let's not try to change the world, but +rather nudge ourselves just a tiny bit. And if we only did that. Each of us +nudged ourselves a small, tiny bit, the world would heal. If we would just put +down the phones and ignored Internet for a day or two. Put visiting websites +that feed on us on hold. Listened to just one sentence and try to understand it +from a person who we completely disagree with. I truly believe that this is +possible. + +Life is about suffering and joy. And instead of wishing suffering on others and +excepting joy for yourselves, we should for a brief moment want suffering for +ourselves and wish joy on others. Wouldn't that be an amazing sight to see? + +I caught myself hating on Rust. And I deeply thought about it afterward. Why did +I do it? It is obviously not for me. So why the hell was I being so negative +towards it? I think that I know the answer. I was negative because that is +easy. Because it's much easier to hate on things than to say to yourself: "Well, +you know what? This is not for me. I will focus on creation and not +destruction. This is who I want to be. This is what fills me with joy and +purpose." Where joy is keeping me happy and purpose scares the shit out of me +and keeps me honest. This is who I want to be. Admit to myself when I am wrong +and accept the faults that I have without reservation and with courage march on. + +I just realized that this blog post is a sort of therapy for me. It's +cathartic. Going thought the history of this site and remembering all the +decisions and annoyances that came with it. When I was cursing at the tools. And +time moved on, and the site is still here. It serves as a reminder that +perseverance wins at the end. If we just let things go. + +This came with a decision that simplifying life and removing all the unnecessary +negativity is key. Rather than worrying about what the internet is saying, what +the world is trying to take from you, you are the only one who can say no. And +create instead of destroy. + +I don't have an ending for this post, so I will say this. We live in the most +amazing times in the recorded history, and we should be internally grateful for +it. Create and study, this should be my mantra. Just create and let the world +happen. And you feel yourself to be too certain, stop and check how deep in the +shit you are already. Strong opinions are a sign of a weak and uneducated +mind. Hate and disdain is for the weak. diff --git a/content/posts/2023-07-10-fix-screen-tearing-on-debian-12-xorg-and-i3.md b/content/posts/2023-07-10-fix-screen-tearing-on-debian-12-xorg-and-i3.md new file mode 100644 index 0000000..1de0ffe --- /dev/null +++ b/content/posts/2023-07-10-fix-screen-tearing-on-debian-12-xorg-and-i3.md @@ -0,0 +1,22 @@ +--- +title: "Fix screen tearing on Debian 12 Xorg and i3" +url: fix-screen-tearing-on-debian-12-xorg-and-i3.html +date: 2023-07-10T04:21:48+02:00 +type: note +draft: false +--- + +I have been experiencing some issues with Intel® Integrated HD Graphics 3000 +under Debian 12 with Xorg and i3. Using `picom` compositor didn't help. To fix +this issue create new file `/etc/X11/xorg.conf.d/20-intel.conf` as root and put +the following in the file. + +``` +Section "Device" + Identifier "Intel Graphics" + Driver "intel" + Option "TearFree" "true" +EndSection +``` + +Reboot the system and that should be it. diff --git a/content/posts/2023-07-10-online-radio-streaming-with-mpv-from-terminal.md b/content/posts/2023-07-10-online-radio-streaming-with-mpv-from-terminal.md new file mode 100644 index 0000000..821a80f --- /dev/null +++ b/content/posts/2023-07-10-online-radio-streaming-with-mpv-from-terminal.md @@ -0,0 +1,14 @@ +--- +title: "Online radio streaming with MPV from terminal" +url: online-radio-streaming-with-mpv-from-terminal.html +date: 2023-07-10T03:34:45+02:00 +type: note +draft: false +--- + +Recently I have been using my Thinkpad x220 more and there are some constraints +I have faced with it. CPU is not as powerful as on my main machine and I really +want to listen to some music while using the machine. Browsers really are bloat. + +Check out this site https://streamurl.link/ and copy the stream url and then do +`mpv streamlink`. -- cgit v1.2.3