From 43b0708769eb61392050045b881f8e6ba39c5b66 Mon Sep 17 00:00:00 2001 From: Mitja Felicijan Date: Fri, 26 May 2023 00:40:40 +0200 Subject: Massive update to posts, archetypes Added a archetypes for creating notes and posts so it auto-populates fields. Fixed existing posts so they align with the rule of 80 columns now. --- content/notes/cachebusting-in-hugo.md | 2 +- content/notes/catv-weechat-config.md | 2 +- content/notes/convert-mkv.md | 2 +- content/notes/dcss-materials.md | 69 +++++ content/notes/download-youtube-videos.md | 2 +- content/notes/extend-lua-with-custom-c.md | 2 +- content/notes/fix-plan9-bootloader.md | 2 +- content/notes/fresh-9front-desktop.md | 2 +- content/notes/git-push-multiple-origins.md | 2 +- content/notes/install-plan9port-linux.md | 2 +- content/notes/mass-set-permission.md | 2 +- content/notes/mount-plan9-over-network.md | 2 +- content/notes/non-blocking-shell-exec-csharp.md | 2 +- content/notes/parse-rss-with-lua.md | 2 +- content/notes/plan9-screenshot.md | 2 +- content/notes/preview-troff-man-pages.md | 2 +- content/notes/run-9front-in-qemu.md | 2 +- content/notes/show-xterm-colors.md | 2 +- content/notes/something.md | 11 + content/notes/tmux-sane-defaults.md | 2 +- content/notes/write-iso-usb.md | 5 +- content/pages/books.md | 2 +- content/pages/curriculum-vitae.md | 2 +- ...01-13-most-likely-to-succeed-in-year-of-2011.md | 29 +- .../posts/2012-03-09-led-technology-not-so-eco.md | 24 +- .../posts/2013-10-24-wireless-sensor-networks.md | 50 +++- .../2015-11-10-software-development-pitfalls.md | 180 +++++++++--- .../2017-03-07-golang-profiling-simplified.md | 28 +- ...04-17-what-i-ve-learned-developing-ad-server.md | 114 ++++++-- ...ng-python-web-applications-with-visual-tools.md | 41 ++- content/posts/2017-08-11-simple-iot-application.md | 223 +++++++++++---- ...digitalocean-spaces-object-storage-with-fuse.md | 117 ++++++-- ...01-03-encoding-binary-data-into-dna-sequence.md | 133 ++++++--- .../2019-10-14-simplifying-and-reducing-clutter.md | 63 +++- ...g-sentiment-analysis-for-clickbait-detection.md | 43 ++- .../2020-03-22-simple-sse-based-pubsub-server.md | 109 +++++-- ...0-03-27-create-placeholder-images-with-sharp.md | 36 ++- ...nge-case-of-elasticsearch-allocation-failure.md | 61 +++- ...30-my-love-and-hate-relationship-with-nodejs.md | 99 ++++++- content/posts/2020-05-05-remote-work.md | 63 +++- .../2020-08-15-systemd-disable-wake-onmouse.md | 42 ++- content/posts/2020-09-06-esp-and-micropython.md | 68 +++-- content/posts/2020-09-08-bind-warning-on-login.md | 28 +- content/posts/2020-09-09-digitalocean-sync.md | 92 ++++-- .../posts/2021-01-24-replacing-dropbox-with-s3.md | 52 +++- content/posts/2021-01-25-goaccess.md | 90 ++++-- content/posts/2021-06-26-simple-world-clock.md | 34 ++- ...from-internet-consumer-to-full-hominum-again.md | 100 ++++++- content/posts/2021-08-01-linux-cheatsheet.md | 3 +- ...n-based-riced-up-distribution-for-developers.md | 164 ++++++++--- ...021-12-25-running-golang-application-as-pid1.md | 122 ++++++-- .../2021-12-30-wap-mobile-web-before-the-web.md | 79 +++-- .../posts/2022-06-30-trying-out-helix-editor.md | 42 ++- ...22-07-05-what-would-dna-sound-if-synthesized.md | 134 ++++++--- .../2022-08-13-algae-spotted-on-river-sava.md | 12 +- ...10-06-state-of-web-technologies-in-year-2022.md | 317 ++++++++++++++++----- ...2022-10-10-sentiment-based-on-political-bias.md | 84 ------ .../2022-10-11-handling-massive-worlds-in-godot.md | 82 ------ ...hat-sound-that-machine-makes-when-struggling.md | 40 ++- ...ing-to-build-a-new-kind-of-terminal-emulator.md | 140 ++++++--- ...5-10-push-to-multiple-origins-at-once-in-git.md | 15 +- ...023-05-16-rekindling-my-love-for-programming.md | 72 ++++- .../2023-05-22-crafting-stories-in-zed-editor.md | 61 ++-- .../2023-05-23-i-was-wrong-about-git-workflows.md | 68 ++++- 64 files changed, 2554 insertions(+), 925 deletions(-) create mode 100644 content/notes/dcss-materials.md create mode 100644 content/notes/something.md delete mode 100644 content/posts/2022-10-10-sentiment-based-on-political-bias.md delete mode 100644 content/posts/2022-10-11-handling-massive-worlds-in-godot.md (limited to 'content') diff --git a/content/notes/cachebusting-in-hugo.md b/content/notes/cachebusting-in-hugo.md index 65d7769..34e1684 100644 --- a/content/notes/cachebusting-in-hugo.md +++ b/content/notes/cachebusting-in-hugo.md @@ -1,7 +1,7 @@ --- title: Cache busting in Hugo url: cachebusting-in-hugo.html -date: 2023-05-01 +date: 2023-05-01T12:00:00+02:00 type: notes draft: false tags: [hugo, cachebusting] diff --git a/content/notes/catv-weechat-config.md b/content/notes/catv-weechat-config.md index 3e6e612..ea53120 100644 --- a/content/notes/catv-weechat-config.md +++ b/content/notes/catv-weechat-config.md @@ -1,7 +1,7 @@ --- title: "#cat-v on weechat configuration" url: catv-weechat-config.html -date: 2023-05-09 +date: 2023-05-09T12:00:00+02:00 type: notes draft: false tags: [irc, weechat, cat-v] diff --git a/content/notes/convert-mkv.md b/content/notes/convert-mkv.md index bafdfb3..0e1987a 100644 --- a/content/notes/convert-mkv.md +++ b/content/notes/convert-mkv.md @@ -1,7 +1,7 @@ --- title: Convert all MKV files into other formats url: convert-mkv.html -date: 2023-05-14 +date: 2023-05-14T12:00:00+02:00 type: notes draft: false tags: [ffmpeg, mkv, webm, mp4] diff --git a/content/notes/dcss-materials.md b/content/notes/dcss-materials.md new file mode 100644 index 0000000..e446224 --- /dev/null +++ b/content/notes/dcss-materials.md @@ -0,0 +1,69 @@ +--- +title: Dungeon Crawl Stone Soup Materials +url: dcss-materials.html +date: 2023-05-25T22:00:00+02:00 +type: notes +draft: false +tags: [dcss] +--- + +An amazing game deserves an amazing guide. All this material can be find in +some for on another on [craw's](https://github.com/crawl/crawl) official +repository. + +- [DCSS Quickstart](/notes/dcss-quickstart.pdf) - Very short introduction to the game +- [DCSS Manual](/notes/dcss_manual.pdf) - Extensive manual about the game + +![Dungeon Crawl Stone Soup](/notes/dcss.jpg) + +**Movement and Exploration** + +- You can move around with the numpad (try numlock on and off), vi-keys, or clicking with the mouse. Arrow keys work, though you can't move diagonally with them. Pressing Shift and a direction will move until you see/hit something. +- Pressing `>` will take you down a staircase, and `<` to go up a staircase. +- You can open doors by walking into them, and close them with `C`. +- You can autoexplore by pressing `o`. +- You can re-view recent messages with `Ctrl-p`. + +**Monsters and Combat** + +- You can pick up items with `,` or `g`. +- Wield weapons with `w`. Weapons have different stats. + - (You may also engage in Unarmed Combat, though it isn't very effective when untrained). +- Attack monsters in melee by walking in their direction (or with Ctrl-direction). +- You can wait with `.` or `s`, passing your turn - such as to get monsters into a corridor with you. +- You can rest with `5`, waiting until you are fully healed, or something noteworthy happens. +- Either mouseover and rightclick, or use `x` then `v` on the monster to examine monsters. Monsters with a red border are 'dangerous' relative to your current XP level (XL). +- Quiver (often ranged) actions for further use with `Q`. +- You can fire ranged weapons manually with `f`, or auto-target your quiver with `p` or `Shift-Tab`. Throwing weapons can be thrown immediately, while launchers (like bows) need to be wielded first. + +**Items and Inventory** + +- View your inventory by pressing `i`. Most item related commands can also be done with this menu. +- You can wear amour with `W;` amour gives `AC`, while heavier body armour reduces `EV`. +- Autoexplore will automatically pick up useful items, such as potions and scrolls, if you aren't in danger. +- You can read scrolls with `r` and drink ("quaff") potions with `q`. +- Equipment items may have brands, with special properties. Branded equipment is blue when unidentified. +- Equipment items may be artifacts, often with unique properties, and are unmodifiable. They are written in white. +- You can evoke wands with `V`. +- You can put on jewelry with `P`, and remove it with `R`. +- Gold is used in shops, which can be interacted with by either `>` or `<`. + +**Magic and Spellcasting** + +- Once you find a spellbook, you can memorize spells with `M`. +- You need to be the same XL as the spell's spell level in order to learn it, in addition to training magical skill (to lower failure rate). +- Cast spells by pressing `z`, then the letter assigned to the spell. You may also Quiver a spell and then use it like a ranged weapon (with Shift-Tab). +- You can view your memorized spells by pressing `I` (capital-i) or `z`. +- Like HP, you can recover MP by resting (with 5). +- Many spells can be positioned more effectively, or combined with other spells, in order to get (more effective) use out of them. +- Heavier body amour and shields hamper spellcasting. + +**Gods and Divine Abilities** + +- You may look at a god's overview by praying at their altar (with `>` or `<`). After praying, you can worship the god by pressing Enter afterwards. +- Gods all have unique features about them. Trog, the god of the tutorial, is also the god of rage and bloodshed, and so despises spellcasting. +- Gods like and dislike different things. Most gods either like killing things (like Trog) or exploring new areas (like Elyvilon), rewarding you piety (divine favor) for doing so. +- You should learn to use and even rely on divine abilities often, as they are usually very strong. Trog's Berserk gives you 1.5x health, 1.5x speed (to all valid actions), and a big damage boost. Note that Berserk prevents most actions other than move and melee attack, and runs out very quickly if you aren't attacking. And after berserk ends, you are slowed down and can't berserk again for a short time. +- In addition, the vast majority of abilities consume piety in the process. Regardless, this ability is very cheap, and the benefits are incredible, so don't hold back! +- Pressing `^` will let you view your current god, abilities, and piety. + diff --git a/content/notes/download-youtube-videos.md b/content/notes/download-youtube-videos.md index 455c550..6d26d12 100644 --- a/content/notes/download-youtube-videos.md +++ b/content/notes/download-youtube-videos.md @@ -1,7 +1,7 @@ --- title: Download list of YouTube files url: download-youtube-videos.html -date: 2023-05-13 +date: 2023-05-13T12:00:00+02:00 type: notes draft: false tags: [youtube, yt-dlp, ffmpeg, webm, mp4] diff --git a/content/notes/extend-lua-with-custom-c.md b/content/notes/extend-lua-with-custom-c.md index 2151237..eaa55a1 100644 --- a/content/notes/extend-lua-with-custom-c.md +++ b/content/notes/extend-lua-with-custom-c.md @@ -1,7 +1,7 @@ --- title: Extend Lua with custom C functions using Clang url: extend-lua-with-custom-c.html -date: 2023-05-23 +date: 2023-05-23T12:00:00+02:00 type: notes draft: false tags: [lua, clang, c] diff --git a/content/notes/fix-plan9-bootloader.md b/content/notes/fix-plan9-bootloader.md index 675eabc..fdd9425 100644 --- a/content/notes/fix-plan9-bootloader.md +++ b/content/notes/fix-plan9-bootloader.md @@ -1,7 +1,7 @@ --- title: Fix bootloader not being written in Plan9 url: fix-plan9-bootloader.html -date: 2023-05-11 +date: 2023-05-11T12:00:00+02:00 type: notes draft: false tags: [plan9, bootloader] diff --git a/content/notes/fresh-9front-desktop.md b/content/notes/fresh-9front-desktop.md index 71edd04..840126a 100644 --- a/content/notes/fresh-9front-desktop.md +++ b/content/notes/fresh-9front-desktop.md @@ -1,7 +1,7 @@ --- title: My brand new Plan9/9front desktop url: fresh-9front-desktop.html -date: 2023-05-24 +date: 2023-05-24T12:00:00+02:00 type: notes draft: false tags: [plan9, 9front] diff --git a/content/notes/git-push-multiple-origins.md b/content/notes/git-push-multiple-origins.md index 14b6ed0..929a5dc 100644 --- a/content/notes/git-push-multiple-origins.md +++ b/content/notes/git-push-multiple-origins.md @@ -1,7 +1,7 @@ --- title: Push to multiple origins at once in Git url: git-push-multiple-origins.html -date: 2023-05-06 +date: 2023-05-06T12:00:00+02:00 type: notes draft: false tags: [git] diff --git a/content/notes/install-plan9port-linux.md b/content/notes/install-plan9port-linux.md index 7e8994d..b74d2c0 100644 --- a/content/notes/install-plan9port-linux.md +++ b/content/notes/install-plan9port-linux.md @@ -1,7 +1,7 @@ --- title: Install Plan9port on Linux url: install-plan9port-linux.html -date: 2023-05-12 +date: 2023-05-12T12:00:00+02:00 type: notes draft: false tags: [plan9, plan9port, linux] diff --git a/content/notes/mass-set-permission.md b/content/notes/mass-set-permission.md index 84ec97a..b57374c 100644 --- a/content/notes/mass-set-permission.md +++ b/content/notes/mass-set-permission.md @@ -1,7 +1,7 @@ --- title: Change permissions of matching files recursively url: mass-set-permission.html -date: 2023-05-16 +date: 2023-05-16T12:00:00+02:00 type: notes draft: false tags: [linux, find, chmod] diff --git a/content/notes/mount-plan9-over-network.md b/content/notes/mount-plan9-over-network.md index 9e9875e..f643395 100644 --- a/content/notes/mount-plan9-over-network.md +++ b/content/notes/mount-plan9-over-network.md @@ -1,7 +1,7 @@ --- title: Mount Plan9 over network url: mount-plan9-over-network.html -date: 2023-05-07 +date: 2023-05-07T12:00:00+02:00 type: notes draft: false tags: [plan9, linux, 9pfs] diff --git a/content/notes/non-blocking-shell-exec-csharp.md b/content/notes/non-blocking-shell-exec-csharp.md index 6de081e..cf43716 100644 --- a/content/notes/non-blocking-shell-exec-csharp.md +++ b/content/notes/non-blocking-shell-exec-csharp.md @@ -1,7 +1,7 @@ --- title: Execute not blocking async shell command in C# url: non-blocking-shell-exec-csharp.html -date: 2023-05-22 +date: 2023-05-22T12:00:00+02:00 type: notes draft: false tags: [csharp, async, shell] diff --git a/content/notes/parse-rss-with-lua.md b/content/notes/parse-rss-with-lua.md index 1757d45..1efeae5 100644 --- a/content/notes/parse-rss-with-lua.md +++ b/content/notes/parse-rss-with-lua.md @@ -1,7 +1,7 @@ --- title: Parse RSS feeds with Lua url: parse-rss-with-lua.html -date: 2023-05-23 +date: 2023-05-23T12:00:00+02:00 type: notes draft: false tags: [lua, rss] diff --git a/content/notes/plan9-screenshot.md b/content/notes/plan9-screenshot.md index fe21ee4..c8a8f23 100644 --- a/content/notes/plan9-screenshot.md +++ b/content/notes/plan9-screenshot.md @@ -1,7 +1,7 @@ --- title: Take a screenshot in Plan9 url: plan9-screenshot.html -date: 2023-05-10 +date: 2023-05-10T12:00:00+02:00 type: notes draft: false tags: [plan9, screenshot] diff --git a/content/notes/preview-troff-man-pages.md b/content/notes/preview-troff-man-pages.md index acda0d6..ad94256 100644 --- a/content/notes/preview-troff-man-pages.md +++ b/content/notes/preview-troff-man-pages.md @@ -1,7 +1,7 @@ --- title: Previews how man page written in Troff will look like url: preview-troff-man-pages.html -date: 2023-05-15 +date: 2023-05-15T12:00:00+02:00 type: notes draft: false tags: [troff, man] diff --git a/content/notes/run-9front-in-qemu.md b/content/notes/run-9front-in-qemu.md index 0001162..3bf2ff5 100644 --- a/content/notes/run-9front-in-qemu.md +++ b/content/notes/run-9front-in-qemu.md @@ -1,7 +1,7 @@ --- title: Run 9front in Qemu url: run-9front-in-qemu.html -date: 2023-05-05 +date: 2023-05-05T12:00:00+02:00 type: notes draft: false tags: [plan9, qemu] diff --git a/content/notes/show-xterm-colors.md b/content/notes/show-xterm-colors.md index fc7280b..008966b 100644 --- a/content/notes/show-xterm-colors.md +++ b/content/notes/show-xterm-colors.md @@ -1,7 +1,7 @@ --- title: Display xterm color palette url: write-iso-usb.html -date: 2023-05-25 +date: 2023-05-25T12:00:00+02:00 type: notes draft: false tags: [linux, xterm, palette] diff --git a/content/notes/something.md b/content/notes/something.md new file mode 100644 index 0000000..435ea8c --- /dev/null +++ b/content/notes/something.md @@ -0,0 +1,11 @@ +--- +title: "Something" +url: something.html +date: 2023-05-26T00:37:57+02:00 +type: notes +draft: true +tags: [] +--- + +Start here ... + diff --git a/content/notes/tmux-sane-defaults.md b/content/notes/tmux-sane-defaults.md index f23e19d..bd76d28 100644 --- a/content/notes/tmux-sane-defaults.md +++ b/content/notes/tmux-sane-defaults.md @@ -1,7 +1,7 @@ --- title: Sane defaults for tmux with more visible statusbar url: tmux-sane-defaults.html -date: 2023-05-25 +date: 2023-05-25T12:00:00+02:00 type: notes draft: false tags: [tmux] diff --git a/content/notes/write-iso-usb.md b/content/notes/write-iso-usb.md index 860e6a7..9e379fd 100644 --- a/content/notes/write-iso-usb.md +++ b/content/notes/write-iso-usb.md @@ -1,7 +1,7 @@ --- title: Write ISO to USB Key url: write-iso-usb.html -date: 2023-05-08 +date: 2023-05-08T12:00:00+02:00 type: notes draft: false tags: [linux, iso, usb] @@ -12,3 +12,6 @@ Write ISO to USB key. Nothing fancy here. ```sh sudo dd if=iso_file.iso of=/dev/sdX bs=4M status=progress conv=fdatasync ``` + + + diff --git a/content/pages/books.md b/content/pages/books.md index 473efab..9f58ec5 100644 --- a/content/pages/books.md +++ b/content/pages/books.md @@ -1,6 +1,6 @@ --- title: Books I've read so far -date: 2022-08-27 +date: 2022-08-27T12:00:00+02:00 url: books.html draft: false --- diff --git a/content/pages/curriculum-vitae.md b/content/pages/curriculum-vitae.md index 762234e..4fee27c 100644 --- a/content/pages/curriculum-vitae.md +++ b/content/pages/curriculum-vitae.md @@ -1,6 +1,6 @@ --- title: Curriculum Vitae -date: 2022-08-27 +date: 2022-08-27T12:00:00+02:00 url: curriculum-vitae.html draft: false --- diff --git a/content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md b/content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md index fdd0075..bdf965a 100644 --- a/content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md +++ b/content/posts/2011-01-13-most-likely-to-succeed-in-year-of-2011.md @@ -1,19 +1,35 @@ --- title: Most likely to succeed in the year of 2011 url: most-likely-to-succeed-in-year-of-2011.html -date: 2011-01-13 +date: 2011-01-13T12:00:00+02:00 draft: false --- -The year of 2010 was definitely the year of Geo-location. The market responded beautifully and lots of very cool services were launched. We all have to thank the mobile market for such extensive adoption. With new generations of mobile phones that are not only buffed with high-tech hardware but are also affordable. We can now manage tasks that were not so long time ago, almost Star Trek’ish. And all this had and has great influence on the destination to which we are going now. +The year of 2010 was definitely the year of Geo-location. The market responded +beautifully and lots of very cool services were launched. We all have to thank +the mobile market for such extensive adoption. With new generations of mobile +phones that are not only buffed with high-tech hardware but are also affordable. +We can now manage tasks that were not so long time ago, almost Star Trek’ish. +And all this had and has great influence on the destination to which we are +going now. -Reading all this articles about new innovation about new thriving technologies makes me wonder what’s the next step. The future is the mesh, like Lisa Gansky said in her book The Mesh. +Reading all this articles about new innovation about new thriving technologies +makes me wonder what’s the next step. The future is the mesh, like Lisa Gansky +said in her book The Mesh. -Many still have conservative views on distributed systems. The problems with security of information. Fear of not controlling every aspect of information flow. I am very opened to distributed systems and heterogeneous applications, and I think this is the correct and best way to proceed. +Many still have conservative views on distributed systems. The problems with +security of information. Fear of not controlling every aspect of information +flow. I am very opened to distributed systems and heterogeneous applications, +and I think this is the correct and best way to proceed. -This year will definitely be about communication platforms. Mobile to mobile. Machine to mobile and vice versa. All the tech is available and ready to put into action. Wireless is today’s new mantra. And the concept of semantic web is now ready for industry. +This year will definitely be about communication platforms. Mobile to mobile. +Machine to mobile and vice versa. All the tech is available and ready to put +into action. Wireless is today’s new mantra. And the concept of semantic web +is now ready for industry. -Applications and developers now can gain access to new layers of systems and can prepare and build solutions to meet the high quality needs of market. The speed is everything now. +Applications and developers now can gain access to new layers of systems and +can prepare and build solutions to meet the high quality needs of market. The +speed is everything now. My vote goes to “Machine to Machine” and “Embedded Systems”! @@ -22,3 +38,4 @@ My vote goes to “Machine to Machine” and “Embedded Systems”! - [COOS Project (connectivity initiative)](http://www.coosproject.org/maven-site/1.0.0/project-info.html) - [Community for machine-to-machine](http://m2m.com/index.jspa) - [Embedded system](http://en.wikipedia.org/wiki/Embedded_system) + diff --git a/content/posts/2012-03-09-led-technology-not-so-eco.md b/content/posts/2012-03-09-led-technology-not-so-eco.md index 514b1c3..6e5aab9 100644 --- a/content/posts/2012-03-09-led-technology-not-so-eco.md +++ b/content/posts/2012-03-09-led-technology-not-so-eco.md @@ -1,20 +1,32 @@ --- title: LED technology might not be as eco-friendly as you think url: led-technology-not-so-eco.html -date: 2012-03-09 +date: 2012-03-09T12:00:00+02:00 draft: false --- -There is a lot of talk about LED technology. It is beginning to infiltrate industry at a fast rate, and it’s a challenge for designers and also engineers. I wondered when a weakness will be revealed. Then I stomped on an article talking about harm in using LED technology. It looks like this magical technology is not so magical and eco-friendly. +There is a lot of talk about LED technology. It is beginning to infiltrate +industry at a fast rate, and it’s a challenge for designers and also engineers. +I wondered when a weakness will be revealed. Then I stomped on an article +talking about harm in using LED technology. It looks like this magical +technology is not so magical and eco-friendly. -A new study from the University of California indicates that LED lights contain toxic metals, and should be produced, used and disposed of carefully. Besides the lead and nickel, the bulbs and their associated parts were also found to contain arsenic, copper, and other metals that have been linked to different cancers, neurological damage, kidney disease, hypertension, skin rashes and other illnesses in humans, and to ecological damage in waterways. +A new study from the University of California indicates that LED lights +contain toxic metals, and should be produced, used and disposed of carefully. +Besides the lead and nickel, the bulbs and their associated parts were also +found to contain arsenic, copper, and other metals that have been linked to +different cancers, neurological damage, kidney disease, hypertension, skin +rashes and other illnesses in humans, and to ecological damage in waterways. -Since then, I haven’t yet found any regulation for disposal of LED lights or any other regulation or standard. This might be a problem in the future. And it is a massive drawback. This might have quite an impact on consumer market. - -Nevertheless, there is a potential, and I am sure the market will adapt. I also hope I will be reading documents regarding solution for this concern soon. +Since then, I haven’t yet found any regulation for disposal of LED lights or +any other regulation or standard. This might be a problem in the future. And +it is a massive drawback. This might have quite an impact on consumer market. +Nevertheless, there is a potential, and I am sure the market will adapt. I +also hope I will be reading documents regarding solution for this concern soon. **Additional resources:** - [Recycling and Disposal of Light Bulbs](http://ezinearticles.com/?Recycling-and-Disposal-of-Light-Bulbs&id=1091304) - [How to Dispose of a Low-Energy Light Bulb](http://www.ehow.com/how_7483442_dispose-lowenergy-light-bulb.html) + diff --git a/content/posts/2013-10-24-wireless-sensor-networks.md b/content/posts/2013-10-24-wireless-sensor-networks.md index e8948fb..dabdebc 100644 --- a/content/posts/2013-10-24-wireless-sensor-networks.md +++ b/content/posts/2013-10-24-wireless-sensor-networks.md @@ -1,25 +1,55 @@ --- title: Wireless sensor networks url: wireless-sensor-networks.html -date: 2013-10-24 +date: 2013-10-24T12:00:00+02:00 draft: false --- -Zigbee networks have this wonderful capability to self-heal, which means they can reorder connections between them if one of them is inoperable. This works our of the box when you deploy them. But you have to have in mind that achieving this is not as easy as you would think. None of it is plug&play. So to make your life a bit easier, here are some pointers which, I hope, will help you. +Zigbee networks have this wonderful capability to self-heal, which means they +can reorder connections between them if one of them is inoperable. This works +our of the box when you deploy them. But you have to have in mind that +achieving this is not as easy as you would think. None of it is plug&play. +So to make your life a bit easier, here are some pointers which, I hope, +will help you. -- Be careful when you are ordering your equipment abroad. There are many rules and regulations you need to comply before you get your Xbee radios. What they do is they wait until you prove that you won’t use the technology for some kind of evil take over control of the world project :). For this, they have EAR (Export Administration Regulations) which basically means “This product may require a license to export from the United States.”. -- I don’t know if this applies for every country, but when we purchased our Xbee radios from Mouser, this was mandatory! What we needed to do was to print out a form and write information about our company and send them a copy via email. With this document, we proved that we are a legitimate company. -- When you complete your purchase and send all the documentation, you are not clear yet. Then customs will take it from there :). There will be some additional costs. Before purchasing, make sure you have as much information about costs as possible. Because it can get costly in the end. -- I suggest you use companies from your country. You can seriously cut your costs. Here in Slovenia, the best option so far as I know is Farnell. And based on my personal experience, they rock! All I need to say! -- Make plans when ordering larger quantities. Do not, I say, do not make your orders in December! :) Believe me! You will have problems with stock they can provide for you. So, we were forced to buy some things from Mouser, which was extremely painful because of all the regulations you need to obey when importing goods from the USA. -- Make sure that firmware version on your Xbee radios is exactly the same! Do not get creative!!! I propose using templates. You can get template by exporting settings/profile in X-CTU application. Make sure you have enabled “Upgrade firmware” so you can be sure each radio has the same firmware. -- And again: make plans! Plan everything! In months advanced! You will thank me later :) +- Be careful when you are ordering your equipment abroad. There are many rules + and regulations you need to comply before you get your Xbee radios. What they + do is they wait until you prove that you won’t use the technology for some + kind of evil take over control of the world project :). For this, they have + EAR (Export Administration Regulations) which basically means “This product + may require a license to export from the United States.”. +- I don’t know if this applies for every country, but when we purchased our + Xbee radios from Mouser, this was mandatory! What we needed to do was to + print out a form and write information about our company and send them a + copy via email. With this document, we proved that we are a legitimate + company. +- When you complete your purchase and send all the documentation, you are not + clear yet. Then customs will take it from there :). There will be some + additional costs. Before purchasing, make sure you have as much information + about costs as possible. Because it can get costly in the end. +- I suggest you use companies from your country. You can seriously cut your + costs. Here in Slovenia, the best option so far as I know is Farnell. And + based on my personal experience, they rock! All I need to say! +- Make plans when ordering larger quantities. Do not, I say, do not make your + orders in December! :) Believe me! You will have problems with stock they + can provide for you. So, we were forced to buy some things from Mouser, + which was extremely painful because of all the regulations you need to obey + when importing goods from the USA. +- Make sure that firmware version on your Xbee radios is exactly the same! Do + not get creative!!! I propose using templates. You can get template by + exporting settings/profile in X-CTU application. Make sure you have enabled + “Upgrade firmware” so you can be sure each radio has the same firmware. +- And again: make plans! Plan everything! In months advanced! You will thank + me later :) - Test, test, test. Wireless networks can be tricky. -If you are serious, I suggest you buy this book, Building Wireless Sensor Networks. You will get a glimpse of how networks work in lumens terms. It is a good starting point for everybody who wants to build wireless networks. +If you are serious, I suggest you buy this book, Building Wireless Sensor +Networks. You will get a glimpse of how networks work in lumens terms. It is +a good starting point for everybody who wants to build wireless networks. **Additional resources:** - http://www.digi.com/aboutus/export/generalexportinfo - http://doresearch.stanford.edu/research-scholarship/export-controls/export-controlled-or-embargoed-countries-entities-and-persons - http://www.bis.doc.gov/licensing/exportingbasics.htm + diff --git a/content/posts/2015-11-10-software-development-pitfalls.md b/content/posts/2015-11-10-software-development-pitfalls.md index ec7fcb8..57e9736 100644 --- a/content/posts/2015-11-10-software-development-pitfalls.md +++ b/content/posts/2015-11-10-software-development-pitfalls.md @@ -1,72 +1,182 @@ --- title: Software development and my favorite pitfalls url: software-development-pitfalls.html -date: 2015-11-10 +date: 2015-11-10T12:00:00+02:00 draft: false --- -Over the years I had the privilege to work on some very excited projects both in software development field and also in electronics field and every experience taught me some invaluable lessons about how NOT TO approach development. And through this post I will try to point out some absurd, outdated techniques I find the most annoying and damaging during a development cycle. There will be swearing because this topic really gets on my nerves and I never coherently tried to explain them in writing. So if I get heated up, please bear with me. - -As new methods of project management are emerging, underlying processes still stay old and outdated. This is mainly because we as people are unable to completely shift away from these approaches. - -I was always struggling with communication, and many times that cost me a relationship or two because I was not on the ball all the time. Through every experience, I became more convinced that I am the problem and never ever doubted that the problem may be that communication never evolved a single step from emails. And if you think for a second, not many things have changed around this topic. We just have different representations of email (message boards, chats, project management tools). And I believe this is the real issue we are facing now. - -There are many articles written about hyper connectivity and the effects that are a direct result of it. But mainstream does nothing towards it. We are just putting out fires, and we do nothing to prevent it. I am certain this will be a major source of grief in coming years. And what we all can do to avoid this is to change our mindset and experiment on our communication skills, development approaches. We need to maximize possible output that a person can give. And to achieve this we need to listen to them, encourage them. I know that not everybody is a naturally born leader, but with enough practice and encouragement they also can become active participants in leadership. - -There are many talks now about methodologies such as Scrum, Kanban, Cleanroom and they all fucking piss me of :). These are all boxes that imprison people and take away their freedom of thought. This is a straightforward mindfuck / amputation of creativity. - -Let me list a couple of things that I find really destructive and bad for a project and in a long run company. +Over the years I had the privilege to work on some very excited projects both +in software development field and also in electronics field and every experience +taught me some invaluable lessons about how NOT TO approach development. And +through this post I will try to point out some absurd, outdated techniques I +find the most annoying and damaging during a development cycle. There will be +swearing because this topic really gets on my nerves and I never coherently +tried to explain them in writing. So if I get heated up, please bear with me. + +As new methods of project management are emerging, underlying processes still +stay old and outdated. This is mainly because we as people are unable to +completely shift away from these approaches. + +I was always struggling with communication, and many times that cost me a +relationship or two because I was not on the ball all the time. Through every +experience, I became more convinced that I am the problem and never ever +doubted that the problem may be that communication never evolved a single step +from emails. And if you think for a second, not many things have changed around +this topic. We just have different representations of email (message boards, +chats, project management tools). And I believe this is the real issue we are +facing now. + +There are many articles written about hyper connectivity and the effects that +are a direct result of it. But mainstream does nothing towards it. We are just +putting out fires, and we do nothing to prevent it. I am certain this will be +a major source of grief in coming years. And what we all can do to avoid this +is to change our mindset and experiment on our communication skills, +development approaches. We need to maximize possible output that a person can +give. And to achieve this we need to listen to them, encourage them. I know +that not everybody is a naturally born leader, but with enough practice and +encouragement they also can become active participants in leadership. + +There are many talks now about methodologies such as Scrum, Kanban, Cleanroom +and they all fucking piss me of :). These are all boxes that imprison people +and take away their freedom of thought. This is a straightforward +mindfuck / amputation of creativity. + +Let me list a couple of things that I find really destructive and bad for a +project and in a long run company. ## Ping emails -Ping emails are emails you have to write as soon as you receive an email. Its sole purpose is to inform the sender that you received their email, and you are working on it. Its result is only to calm down the sender that their task is being dealt with. It’s intent basically is, I did my job by sending you this email, so I am on clear grounds. I categorize this email as fuck you email. This is one of the most irritating types of emails I need to write. This is the ultimate control freak show you can experience, and it gives the sender a false feeling of control. Newsflash: We do not live in 1982 where there was a possibility that email never reached the destination. I really hate this from the bottom of my heart. - -They should be like: “Yes, I am fucking alive, and I am at your service my leash!”. I guess if I would reply like this, I wouldn’t have to write any more of this kind of messages. +Ping emails are emails you have to write as soon as you receive an email. Its +sole purpose is to inform the sender that you received their email, and you +are working on it. Its result is only to calm down the sender that their task +is being dealt with. It’s intent basically is, I did my job by sending you +this email, so I am on clear grounds. I categorize this email as fuck you email. +This is one of the most irritating types of emails I need to write. This is +the ultimate control freak show you can experience, and it gives the sender a +false feeling of control. Newsflash: We do not live in 1982 where there was a +possibility that email never reached the destination. I really hate this from +the bottom of my heart. + +They should be like: “Yes, I am fucking alive, and I am at your service my +leash!”. I guess if I would reply like this, I wouldn’t have to write any +more of this kind of messages. ## Everybody is a project manager -Well, this is a tough one. I noticed that as soon as you let people to give their suggestions, you are basically screwed. There is a truth in the saying: “Give low expectations and deliver little more than you promised.”. +Well, this is a tough one. I noticed that as soon as you let people to give +their suggestions, you are basically screwed. There is a truth in the saying: +“Give low expectations and deliver little more than you promised.”. -People tend to take a role of a manager as soon as they are presented with an opportunity. And by getting angry at them, you only provoke yourself. They are not at fault. You just need to tell them they are only giving suggestions and not tasks at the beginning and everything will be alright. But if you give them a feeling that they are in control, you will have immense problems explaining why their features are not in current release. +People tend to take a role of a manager as soon as they are presented with an +opportunity. And by getting angry at them, you only provoke yourself. They are +not at fault. You just need to tell them they are only giving suggestions and +not tasks at the beginning and everything will be alright. But if you give +them a feeling that they are in control, you will have immense problems +explaining why their features are not in current release. -Project mission must be always leading project requirements and any deviation from it will result in major project butchering. And by this, I mean that the project will get its own path, and you will be left with half done software that helps nobody. Clear mission goals and clean execution will allow you to develop software will clear intent. +Project mission must be always leading project requirements and any deviation +from it will result in major project butchering. And by this, I mean that the +project will get its own path, and you will be left with half done software +that helps nobody. Clear mission goals and clean execution will allow you to +develop software will clear intent. ## We are never wrong -I find this type of arrogance the worst. We must always conduct ourselves that we are infallible and cannot make mistakes. As soon as a procedure or process is established, there is no room for changes or improvements. This is the most idiotic thing someone can say of think. I think that processes need to involve and change over time. This is imperative and need to have in your organization if you want to improve and develop company. We all need to grow balls and change everything in order to adapt to current situations. Being a prisoner of predefined processes kills creativity. - -I am constantly trying new software for project managing and communication. I believe every team has its own dynamic, and it needs to be discovered organically and naturally through many experiments. By putting the team in a box, you are amputating their creativity and therefore minimizing their potential. But if you talk to an executive, you will mainly find archetypical thinking and a strong need to compartmentalize everything from business processes to resource management. And this type of management that often displays micromanagement techniques only works for short periods (couple of years) and then employees either leave the company or become basically retarded drones on autopilot. +I find this type of arrogance the worst. We must always conduct ourselves that +we are infallible and cannot make mistakes. As soon as a procedure or process +is established, there is no room for changes or improvements. This is the most +idiotic thing someone can say of think. I think that processes need to involve +and change over time. This is imperative and need to have in your organization +if you want to improve and develop company. We all need to grow balls and change +everything in order to adapt to current situations. Being a prisoner of +predefined processes kills creativity. + +I am constantly trying new software for project managing and communication. +I believe every team has its own dynamic, and it needs to be discovered +organically and naturally through many experiments. By putting the team in a +box, you are amputating their creativity and therefore minimizing their +potential. But if you talk to an executive, you will mainly find archetypical +thinking and a strong need to compartmentalize everything from business +processes to resource management. And this type of management that often +displays micromanagement techniques only works for short periods (couple of +years) and then employees either leave the company or become basically retarded +drones on autopilot. ## Micromanaging -This basically implies that everybody on the team is an idiot who needs to have a to-do list that they cannot write themselves. How about spoon-feeding the team at launch because besides the team leader, everybody must be a retarded idiot at best? +This basically implies that everybody on the team is an idiot who needs to +have a to-do list that they cannot write themselves. How about spoon-feeding +the team at launch because besides the team leader, everybody must be a +retarded idiot at best? -I prefer milestones as they give developers much more freedom and creativity in developing and not waste their time checking some bizarre to-do list that was not even thought through. Projects constantly change throughout the development cycle, and all you are left at the end is a list of unchecked tasks and the wrath of management why they are not completed. Best WTF moment! +I prefer milestones as they give developers much more freedom and creativity +in developing and not waste their time checking some bizarre to-do list that +was not even thought through. Projects constantly change throughout the +development cycle, and all you are left at the end is a list of unchecked tasks +and the wrath of management why they are not completed. Best WTF moment! ## Human contact — no need for it! -We are vigorously trying to eliminate physical contact by replacing short meetings with software, with no regards that we are not machines. Many times a simple 5-min meeting at morning can solve most of the problems. In rapid development, short bursts of man to man communication is possibly the best way to go. +We are vigorously trying to eliminate physical contact by replacing short +meetings with software, with no regards that we are not machines. Many times +a simple 5-min meeting at morning can solve most of the problems. In rapid +development, short bursts of man to man communication is possibly the best +way to go. -We now have all this software available, and all what we get out of it is a giant clusterfuck. An obstacle and not a solution. So, why we still use them? +We now have all this software available, and all what we get out of it is a +giant clusterfuck. An obstacle and not a solution. So, why we still use them? ## MVP is killing innovation -Many will disagree with me on this one, but I stand strong by this statement. What I noticed in my experience that all this buzz words around us only mislead and capture us in a circle of solving issues that already have a solution, but we are unable to see it without using some fancy word for it. +Many will disagree with me on this one, but I stand strong by this statement. +What I noticed in my experience that all this buzz words around us only +mislead and capture us in a circle of solving issues that already have a +solution, but we are unable to see it without using some fancy word for it. -The toughest thing to do for a developer is to minimize requirements. Well, this is though only for bad developers. Yes, I said it. There are many types of developers out there. And those unable to minimize feature scope are the ones you don’t need on your team. Their only goal is to solve problems that exist only in their heads. And then you have to argue with them, and waste energy on them, instead of developing your awesome product. They are a cancer and I suggest you cut them off. +The toughest thing to do for a developer is to minimize requirements. Well, +this is though only for bad developers. Yes, I said it. There are many types +of developers out there. And those unable to minimize feature scope are the +ones you don’t need on your team. Their only goal is to solve problems that +exist only in their heads. And then you have to argue with them, and waste +energy on them, instead of developing your awesome product. They are a cancer +and I suggest you cut them off. -MVP as an idea is great, but sadly people don’t understand underlying philosophy, and they spent too much time focusing and fixating on something that every sane person with normal IQ will understand without some made up acronym. And the result is a lot of talking and barely no execution. +MVP as an idea is great, but sadly people don’t understand underlying +philosophy, and they spent too much time focusing and fixating on something +that every sane person with normal IQ will understand without some made up +acronym. And the result is a lot of talking and barely no execution. -Well, MVP is not directly killing innovation, but stupid people do when they try to understand it. +Well, MVP is not directly killing innovation, but stupid people do when they +try to understand it. ## Pressure wasteland -You must never allow to be pressured into confirming a deadline if you are not confident. We often feel a need that we are in service of others, which is true to some extent. But it is also true that others are in service to us to some extent. And we forget this all the time. We are all pressured all the time to make decisions just to calm other people down. And when they leave your office you experience WTF moment :) How the hell did they manage to fuck me up again? - -People need to realize that the more pressure you put on somebody, the less they will be able to do. So 5-min update email requests will only resolve in mental breakdown and inability to work that day. Constant poking is probably the only thing I lose my mind instantly. For all you that are doing this: “Stop bothering us with your insecurities and let us do our job. We will do it quicker and better without you breathing down our necks.” - -If this happens to me, I end up with no energy at the end. Don’t you get it? You will get much more from and out of me if you ask me like a human person and not your personal butler. On a long run, you are destroying your relationships and nobody would want to work with you. Your schizophrenic approach will damage only you in a long run. Nobody is anybody’s property. +You must never allow to be pressured into confirming a deadline if you are not +confident. We often feel a need that we are in service of others, which is +true to some extent. But it is also true that others are in service to us to +some extent. And we forget this all the time. We are all pressured all the +time to make decisions just to calm other people down. And when they leave +your office you experience WTF moment :) How the hell did they manage to +fuck me up again? + +People need to realize that the more pressure you put on somebody, the less +they will be able to do. So 5-min update email requests will only resolve in +mental breakdown and inability to work that day. Constant poking is probably +the only thing I lose my mind instantly. For all you that are doing this: +“Stop bothering us with your insecurities and let us do our job. We will do +it quicker and better without you breathing down our necks.” + +If this happens to me, I end up with no energy at the end. Don’t you get it? +You will get much more from and out of me if you ask me like a human person +and not your personal butler. On a long run, you are destroying your +relationships and nobody would want to work with you. Your schizophrenic +approach will damage only you in a long run. Nobody is anybody’s property. ## Conclusion -I am guilty of many things described in this post. And I find it hard sometimes to acknowledge this. And I lie to myself and try vigorously to find some explanation why I do these things. There is always space for growth. And maybe you will also find some of yourself in this post and realize what needs to change for you to evolve. +I am guilty of many things described in this post. And I find it hard sometimes +to acknowledge this. And I lie to myself and try vigorously to find some +explanation why I do these things. There is always space for growth. And +maybe you will also find some of yourself in this post and realize what needs +to change for you to evolve. + diff --git a/content/posts/2017-03-07-golang-profiling-simplified.md b/content/posts/2017-03-07-golang-profiling-simplified.md index 92ff881..7eee2dd 100644 --- a/content/posts/2017-03-07-golang-profiling-simplified.md +++ b/content/posts/2017-03-07-golang-profiling-simplified.md @@ -1,27 +1,40 @@ --- title: Golang profiling simplified url: golang-profiling-simplified.html -date: 2017-03-07 +date: 2017-03-07T12:00:00+02:00 draft: false --- -Many posts have been written regarding profiling in Golang and I haven’t found proper tutorial regarding this. Almost all of them are missing some part of important information and it gets pretty frustrating when you have a deadline and are not finding simple distilled solution. +Many posts have been written regarding profiling in Golang and I haven’t found +proper tutorial regarding this. Almost all of them are missing some part of +important information and it gets pretty frustrating when you have a deadline +and are not finding simple distilled solution. -Nevertheless, after searching and experimenting I have found a solution that works for me and probably should also for you. +Nevertheless, after searching and experimenting I have found a solution that +works for me and probably should also for you. ## Where are my pprof files? -By default pprof files are generated in /tmp/ folder. You can override folder where this files are generated programmatically in your golang code as we will see below in example. +By default pprof files are generated in /tmp/ folder. You can override folder +where this files are generated programmatically in your golang code as we will +see below in example. ## Why is my CPU profile empty? -I have found out that sometimes CPU profile is empty because program was not executing long enough. Programs, that execute too quickly don’t produce pprof file in my cases. Well, file is generated but only contains 4KB of information. +I have found out that sometimes CPU profile is empty because program was not +executing long enough. Programs, that execute too quickly don’t produce pprof +file in my cases. Well, file is generated but only contains 4KB of information. ## Profiling -As you can see from examples we are executing dummy_benchmark functions to ensure some sort of execution. Memory profiling can be done without such a “complex” function. But CPU profiling needs it. +As you can see from examples we are executing dummy_benchmark functions to +ensure some sort of execution. Memory profiling can be done without such a +“complex” function. But CPU profiling needs it. -Both memory and CPU profiling examples are almost the same. Only parameters in main function when calling profile.Start are different. When we set profile.ProfilePath(“.”) we tell profiler to store pprof files in the same folder as our program. +Both memory and CPU profiling examples are almost the same. Only parameters +in main function when calling profile.Start are different. When we set +profile.ProfilePath(“.”) we tell profiler to store pprof files in the same +folder as our program. ### Memory profiling @@ -109,3 +122,4 @@ This will generate PDF document with visualized profile. - [Memory PDF profile example](/assets/go-profiling/golang-profiling-mem.pdf) - [CPU PDF profile example](/assets/go-profiling/golang-profiling-cpu.pdf) + diff --git a/content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md b/content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md index 6fca4ce..1b9be06 100644 --- a/content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md +++ b/content/posts/2017-04-17-what-i-ve-learned-developing-ad-server.md @@ -1,11 +1,14 @@ --- title: What I've learned developing ad server url: what-i-ve-learned-developing-ad-server.html -date: 2017-04-17 +date: 2017-04-17T12:00:00+02:00 draft: false --- -For the past year and half I have been developing native advertising server that contextually matches ads and displays them in different template forms on variety of websites. This project grew from serving thousands of ads per day to millions. +For the past year and half I have been developing native advertising server +that contextually matches ads and displays them in different template forms +on variety of websites. This project grew from serving thousands of ads per +day to millions. The system is made from couple of core components: @@ -13,29 +16,56 @@ The system is made from couple of core components: - Utils - cronjobs and queue management tools, - Dashboard UI. -Initial release was using [MongoDB](https://www.mongodb.com/) for full-text search but was later replaced by [Elasticsearch](https://www.elastic.co/) for better CPU utilization and better search performance. This provided us with many amazing functionalities of [Elasticsearch](https://www.elastic.co/). You should check it out if you do any search related operations. +Initial release was using [MongoDB](https://www.mongodb.com/) for full-text +search but was later replaced by [Elasticsearch](https://www.elastic.co/) +for better CPU utilization and better search performance. This provided us +with many amazing functionalities of [Elasticsearch](https://www.elastic.co/). +You should check it out if you do any search related operations. -Because the premise of the server is to provide native ad experience, they are rendered on the client side via simple templating engine. This ensures that ads can be displayed number of different ways based on the visual style of the page. And this makes JavaScript client library quite complex. +Because the premise of the server is to provide native ad experience, they +are rendered on the client side via simple templating engine. This ensures +that ads can be displayed number of different ways based on the visual style +of the page. And this makes JavaScript client library quite complex. -So now that you know basic information about the product lets get into the lessons we learned. +So now that you know basic information about the product lets get into the +lessons we learned. ## Aggregate everything -After beta version was released everything (impressions, clicks, etc) was written in nanosecond resolution in the database. At that time we were using [PostgreSQL](https://www.postgresql.org/) and database quickly grew way above 200GB in disk space. And that was problematic. Statistics took disturbingly long time to aggregate. Also using indexes on stats table in database was no help after we reached 500 million datapoints. +After beta version was released everything (impressions, clicks, etc) was +written in nanosecond resolution in the database. At that time we were using +[PostgreSQL](https://www.postgresql.org/) and database quickly grew way above +200GB in disk space. And that was problematic. Statistics took disturbingly +long time to aggregate. Also using indexes on stats table in database was no +help after we reached 500 million datapoints. -> There is a marketing product information and there is real life experience. And the tend to be quite the opposite. +> There is a marketing product information and there is real life experience. +And the tend to be quite the opposite. -This was the reason that now everything is aggregated on daily basis and this data is then fed to Elastic in form of daily summary. With this we achieved we can now track many more dimensions such as zone, channel and platform information. And with this information we can now adapt occurrences of ads on specific places more precisely. +This was the reason that now everything is aggregated on daily basis and this +data is then fed to Elastic in form of daily summary. With this we achieved we +can now track many more dimensions such as zone, channel and platform information. +And with this information we can now adapt occurrences of ads on specific +places more precisely. -We have also adapted [Redis](https://redis.io/) as a full-time citizen in our stack. Because Redis also stores information on a local disk we have some sort of backup if server would accidentally suffer some failure. +We have also adapted [Redis](https://redis.io/) as a full-time citizen in our +stack. Because Redis also stores information on a local disk we have some sort +of backup if server would accidentally suffer some failure. -All the real-time statistics for ad serving and redirecting is presented as counters in Redis instance and daily extracted and pushed to Elastic. +All the real-time statistics for ad serving and redirecting is presented as +counters in Redis instance and daily extracted and pushed to Elastic. ## Measure everything -The thing about software is that we really don't know how well it is performing under load until such load is presented. When testing locally everything is fine but when on production things tend to fall apart. +The thing about software is that we really don't know how well it is performing +under load until such load is presented. When testing locally everything is +fine but when on production things tend to fall apart. -As a solution for this we are measuring everything we can. Function execution time (by encapsulating functions with timers), server performance (cpu, memory, disk, etc), Nginx and [uWSGI](https://uwsgi-docs.readthedocs.io/) performance. We sacrifice a bit of performance for the sake of this information. And we store all this information for later analysis. +As a solution for this we are measuring everything we can. Function execution +time (by encapsulating functions with timers), server performance (cpu, memory, +disk, etc), Nginx and [uWSGI](https://uwsgi-docs.readthedocs.io/) performance. +We sacrifice a bit of performance for the sake of this information. And we +store all this information for later analysis. **Example of function execution time** @@ -69,17 +99,28 @@ As a solution for this we are measuring everything we can. Function execution ti } ``` -We have also started profiling with [cProfile](https://pymotw.com/2/profile/) and then visualizing with [KCachegrind](http://kcachegrind.sourceforge.net/). This provides much more detailed look into code execution. +We have also started profiling with [cProfile](https://pymotw.com/2/profile/) +and then visualizing with [KCachegrind](http://kcachegrind.sourceforge.net/). +This provides much more detailed look into code execution. ## Cache control is your friend -Because we use Javascript library for rendering ads we rely on this script extensively and when in need we need to be able to change behavior of the script quickly. +Because we use Javascript library for rendering ads we rely on this script +extensively and when in need we need to be able to change behavior of the +script quickly. -In our case we can not simply replace javascript url in html code. It usually takes a day or two for the guys who maintain sites to change code or add ?ver=xxx attribute. And this makes rapid deployment and testing very difficult and time consuming. There is a limitation of how much you can test locally. +In our case we can not simply replace javascript url in html code. It usually +takes a day or two for the guys who maintain sites to change code or add +?ver=xxx attribute. And this makes rapid deployment and testing very difficult +and time consuming. There is a limitation of how much you can test locally. -We are now in the process of integrating [Google Tag Manager](https://www.google.com/analytics/tag-manager/) but couple of websites are developed on ASP.net platform that have some problems with tag manager. With a solution below we are certain that we are serving latest version of the script. +We are now in the process of integrating [Google Tag Manager](https://www.google.com/analytics/tag-manager/) +but couple of websites are developed on ASP.net platform that have some +problems with tag manager. With a solution below we are certain that we are +serving latest version of the script. -And it only takes one mistake and users have the script cached and in case of caching it for 1 year you probably know where the problem is. +And it only takes one mistake and users have the script cached and in case of +caching it for 1 year you probably know where the problem is. ```nginx # nginx ➜ /etc/nginx/sites-available/default @@ -102,7 +143,10 @@ location /static/ { } ``` -Also be careful when redirecting to url in your python code. We noticed that if we didn't precisely setup cache control and expire headers in response we didn't get the request on the server and therefore couldn't measure clicks. So when redirecting do as follows and there will be no problems. +Also be careful when redirecting to url in your python code. We noticed that +if we didn't precisely setup cache control and expire headers in response we +didn't get the request on the server and therefore couldn't measure clicks. +So when redirecting do as follows and there will be no problems. ```python # python ➜ bottlepy web micro-framework @@ -113,22 +157,42 @@ response.set_header("Location", url) return response ``` -> Cache control in browsers is quite aggressive and you need to be precise to avoid future problems. We learned that lesson the hard way. +> Cache control in browsers is quite aggressive and you need to be precise +to avoid future problems. We learned that lesson the hard way. ## Learn NGINX -When deciding on a web server we went with Nginx as a reverse proxy for our applications. We adapted micro-service oriented architecture early in the project to ensure when we scale we can easily add additional servers to our cluster. And Nginx was crucial to perform load balancing and static content delivery. +When deciding on a web server we went with Nginx as a reverse proxy for our +applications. We adapted micro-service oriented architecture early in the +project to ensure when we scale we can easily add additional servers to our +cluster. And Nginx was crucial to perform load balancing and static content +delivery. -At first our config file was quite simple and later grew larger. After patching and adding new settings I sat down and learned more about the guts of Nginx. This proved to be very useful and we were able to squeeze much more out of our setup. So I advise you to take your time and read through the [documentation](https://nginx.org/en/docs/). This saved us a lot of headache. Googling for solutions only goes so far. +At first our config file was quite simple and later grew larger. After patching +and adding new settings I sat down and learned more about the guts of Nginx. +This proved to be very useful and we were able to squeeze much more out of our +setup. So I advise you to take your time and read through the +[documentation](https://nginx.org/en/docs/). This saved us a lot of headache. +Googling for solutions only goes so far. ## Use Redis/Memcached -As explained above we are using caching basically for everything. It is the corner stone of our services. At first we were very careful about the quantity of things we stored in [Redis](https://redis.io/). But we later found out that the memory footprint is very low even when storing large amount of data in it. +As explained above we are using caching basically for everything. It is the +corner stone of our services. At first we were very careful about the quantity +of things we stored in [Redis](https://redis.io/). But we later found out that +the memory footprint is very low even when storing large amount of data in it. -So we gradually increased our usage to caching whole HTML outputs of dashboard. This improved our performance in order of magnitude. And by using native TTL support this goes hand in hand with our needs. +So we gradually increased our usage to caching whole HTML outputs of dashboard. +This improved our performance in order of magnitude. And by using native TTL +support this goes hand in hand with our needs. -The reason why we choose [Redis](https://redis.io/) over [Memcached](https://memcached.org/) was the nature of scalability of Redis out of the box. But all this can be achieved with Memcached. +The reason why we choose [Redis](https://redis.io/) over [Memcached](https://memcached.org/) +was the nature of scalability of Redis out of the box. But all this can be +achieved with Memcached. ## Conclusion -There are a lot more details that could have been written and every single topic in here deserves it's own post but you probably got the idea about the problems we faced. +There are a lot more details that could have been written and every single +topic in here deserves it's own post but you probably got the idea about +the problems we faced. + diff --git a/content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md b/content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md index 466d838..911e6e0 100644 --- a/content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md +++ b/content/posts/2017-04-21-profiling-python-web-applications-with-visual-tools.md @@ -1,11 +1,14 @@ --- title: Profiling Python web applications with visual tools url: profiling-python-web-applications-with-visual-tools.html -date: 2017-04-21 +date: 2017-04-21T12:00:00+02:00 draft: false --- -I have been profiling my software with KCachegrind for a long time now and I was missing this option when I am developing API's or other web services. I always knew that this is possible but never really took the time and dive into it. +I have been profiling my software with KCachegrind for a long time now and I +was missing this option when I am developing API's or other web services. I +always knew that this is possible but never really took the time and dive +into it. Before we begin there are some requirements. We will need to: @@ -25,7 +28,8 @@ We will be dividing this post into two main categories: ## Simple web-service -Let's use virtualenv so we won't pollute our base system. If you don't have virtualenv installed on your system you can install it with pip command. +Let's use virtualenv so we won't pollute our base system. If you don't have +virtualenv installed on your system you can install it with pip command. ```bash # let's install virtualenv globally @@ -69,7 +73,8 @@ $ pip install bottle $ deactivate ``` -We are now ready to write simple web service. Let's create file app.py and paste code bellow in this newly created file. +We are now ready to write simple web service. Let's create file app.py and +paste code bellow in this newly created file. ```python # -*- coding: utf-8 -*- @@ -121,7 +126,8 @@ if __name__ == '__main__': # open browser 'http://0.0.0.0:4000' ``` -When browser hits awesome\_random\_number() function profile is created in prof/ subfolder. +When browser hits awesome\_random\_number() function profile is created in +prof/ subfolder. ## Visualize profile @@ -133,17 +139,27 @@ $ pyprof2calltree -i awesome_random_number.prof # this creates 'awesome_random_number.prof.log' file in the same folder ``` -This file can be opened with visualizing tools listed above. In this case we will be using Profilling Viewer under MacOS. You can open image in new tab. As you can see from this example there is hierarchy of execution order of your code. +This file can be opened with visualizing tools listed above. In this case we +will be using Profilling Viewer under MacOS. You can open image in new tab. +As you can see from this example there is hierarchy of execution order of +your code. ![Profilling Viewer](/assets/python-profiling/profiling-viewer.png) -> Make sure you convert output of the cProfile output every time you want to refresh and take a look at your possible optimizations because cProfile updates .prof file every time browser hits the function. +> Make sure you convert output of the cProfile output every time you want to +refresh and take a look at your possible optimizations because cProfile +updates .prof file every time browser hits the function. -This is just a simple example but when you are developing real-life applications this can be very illuminating, especially to see which parts of your code are bottlenecks and need to be optimized. +This is just a simple example but when you are developing real-life applications +this can be very illuminating, especially to see which parts of your code are +bottlenecks and need to be optimized. ## Update 2017-04-22 -Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this awesome web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/) that directly takes output from [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) module. +Reddit user [mvt](https://www.reddit.com/user/mvt) also recommended this +awesome web based profile visualizer [SnakeViz](https://jiffyclub.github.io/snakeviz/) +that directly takes output from [cProfile](https://docs.python.org/2/library/profile.html#module-cProfile) +module.
Comment from discussion Profiling Python web applications with visual tools.
@@ -160,7 +176,8 @@ $ snakeviz awesome_random_number.prof ![SnakeViz](/assets/python-profiling/snakeviz.png) -Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better way for installing pip software by targeting user level instead of using sudo. +Reddit user [ccharles](https://www.reddit.com/user/ccharles) suggested a better +way for installing pip software by targeting user level instead of using sudo.
Comment from discussion Profiling Python web applications with visual tools.
@@ -182,4 +199,6 @@ $ echo $PATH $ pip install snakeviz --user ``` -Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can use [pipsi](https://github.com/mitsuhiko/pipsi). +Or as suggested by [mvt](https://www.reddit.com/user/mvt) you can +use [pipsi](https://github.com/mitsuhiko/pipsi). + diff --git a/content/posts/2017-08-11-simple-iot-application.md b/content/posts/2017-08-11-simple-iot-application.md index b34fe6b..a0ac4be 100644 --- a/content/posts/2017-08-11-simple-iot-application.md +++ b/content/posts/2017-08-11-simple-iot-application.md @@ -1,21 +1,40 @@ --- title: Simple IOT application supported by real-time monitoring and data history url: simple-iot-application.html -date: 2017-08-11 +date: 2017-08-11T12:00:00+02:00 draft: false --- ## Initial thoughts -I have been developing these kind of application for the better part of my last 5 years and people keep asking me how to approach developing such application and I will give a try explaining it here. - -IOT applications are really no different than any other kind of applications. We have data that needs to be collected and visualized in some form of tables or charts. The main difference here is that most of the times these data is collected by some kind of device foreign to developer that mainly operates in web domain. But fear not, it's not that different than writing some JavaScript. - -There are many devices able to transmit data via wireless or wired network by default but for the sake of example we will be using commonly known Arduino with wireless module already on the board → [Arduino MKR1000](https://store.arduino.cc/arduino-mkr1000). - -In order to make this little project as accessible to others as possible I will try to make it as inexpensive as possible. And by this I mean that I will avoid using hosted virtual servers and will be using my own laptop as a server. But you must buy Arduino MKR1000 to follow steps below. But if you would want to deploy this software I would suggest using [DigitalOcean](https://www.digitalocean.com) → smallest VPS is only per month making this one of the most affordable option out there. Please notice that this software will not run on stock web hosting that only supports LAMP (Linux, Apache, MySQL, and PHP). - -_But before we begin please take notice that this is strictly experimental code and not well optimized and there are much better ways in handling some aspects of the application but that requires much deeper knowledge of technology that is not needed for an example like this._ +I have been developing these kind of application for the better part of my last +5 years and people keep asking me how to approach developing such application +and I will give a try explaining it here. + +IOT applications are really no different than any other kind of applications. +We have data that needs to be collected and visualized in some form of tables +or charts. The main difference here is that most of the times these data is +collected by some kind of device foreign to developer that mainly operates in +web domain. But fear not, it's not that different than writing some JavaScript. + +There are many devices able to transmit data via wireless or wired network by +default but for the sake of example we will be using commonly known Arduino +with wireless module already on the board → [Arduino MKR1000](https://store.arduino.cc/arduino-mkr1000). + +In order to make this little project as accessible to others as possible I will +try to make it as inexpensive as possible. And by this I mean that I will avoid +using hosted virtual servers and will be using my own laptop as a server. But +you must buy Arduino MKR1000 to follow steps below. But if you would want to +deploy this software I would suggest using +[DigitalOcean](https://www.digitalocean.com) → smallest VPS is only per month +making this one of the most affordable option out there. Please notice that +this software will not run on stock web hosting that only supports LAMP +(Linux, Apache, MySQL, and PHP). + +But before we begin please take notice that this is strictly experimental code +and not well optimized and there are much better ways in handling some aspects +of the application but that requires much deeper knowledge of technology that +is not needed for an example like this. **Development steps** @@ -23,23 +42,33 @@ _But before we begin please take notice that this is strictly experimental code 2. Prototype C++ code that will read "sensor data" and transmit it to API. 3. Data visualization with charts → extends Python web application. -Step 1. and 3. will share the same web application. One route will be dedicated to API and another to serving HTML with chart. +Step 1. and 3. will share the same web application. One route will be dedicated +to API and another to serving HTML with chart. -Schema below represents what we will try to achieve and how different parts correlates to each other. +Schema below represents what we will try to achieve and how different parts +correlates to each other. ![Overview](/assets/iot-application/simple-iot-application-overview.svg) ## Simple Python API -I have always been a fan of simplicity so we will be using [Bottle: Python Web Framework](https://bottlepy.org/docs/dev/). It is a single file web framework that seriously simplifies working with routes, templating and has built-in web server that satisfies our need in this case. +I have always been a fan of simplicity so we will be using [Bottle: Python Web Framework](https://bottlepy.org/docs/dev/). +It is a single file web framework that seriously simplifies working with routes, +templating and has built-in web server that satisfies our need in this case. -First we need to install bottle package. This can be done by downloading ```bottle.py``` and placing it in the root of your application or by using pip software ```pip install bottle --user```. +First we need to install bottle package. This can be done by downloading +```bottle.py``` and placing it in the root of your application or by using +pip software ```pip install bottle --user```. -If you are using Linux or MacOS then Python is already installed. If you will try to test this on Windows please install [Python for Windows](https://www.python.org/downloads/windows/). There may be some problems with path when you will try to launch ```python webapp.py``` so please take care of this before you continue. +If you are using Linux or MacOS then Python is already installed. If you will +try to test this on Windows please install [Python for Windows](https://www.python.org/downloads/windows/). +There may be some problems with path when you will try to launch +```python webapp.py``` so please take care of this before you continue. ### Basic web application -Most basic bottle application is quite simple. Paste code below in ```webapp.py``` file and save. +Most basic bottle application is quite simple. Paste code below in +```webapp.py``` file and save. ```python # -*- coding: utf-8 -*- @@ -67,29 +96,60 @@ if __name__ == "__main__": ) ``` -To run this simple application you should open command prompt or terminal on your machine and go to the folder containing your file and type ```python webapp.py```. If everything goes ok then open your web browser and point it to ```http://0.0.0.0:5000```. +To run this simple application you should open command prompt or terminal on +your machine and go to the folder containing your file and type +```python webapp.py```. If everything goes ok then open your web browser and +point it to ```http://0.0.0.0:5000```. -If you would like change the port of your application (like port 80) and not use root to run your app this will present a problem. The TCP/IP port numbers below 1024 are privileged ports → this is a security feature. So in order of simplicity and security use a port number above 1024 like I have used port 5000. +If you would like change the port of your application (like port 80) and not +use root to run your app this will present a problem. The TCP/IP port numbers +below 1024 are privileged ports → this is a security feature. So in order of +simplicity and security use a port number above 1024 like I have used port +5000. -If this fails at any time please fix it before you continue, because nothing below will work otherwise. +If this fails at any time please fix it before you continue, because nothing +below will work otherwise. -We use 0.0.0.0 as default host so that this app is available over your local network. If you find your local ip ```ifconfig``` and try accessing this site with your phone (if on same network/router as your machine) this should work as well (example of such ip ```http://192.168.1.15:5000```). This is a must have because Arduino will be accessing this application to send it's data. +We use 0.0.0.0 as default host so that this app is available over your local +network. If you find your local ip ```ifconfig``` and try accessing this site +with your phone (if on same network/router as your machine) this should work +as well (example of such ip ```http://192.168.1.15:5000```). This is a must +have because Arduino will be accessing this application to send it's data. ### Web application security -There is a lot to be said about security and is a topic of many books. Of course all this can not be written here but to just establish some basic security → you should always use SSL with your application. Some fantastic free certificates are available by [Let's Encrypt - Free SSL/TLS Certificates](https://letsencrypt.org). With SSL certificate installed you should then make use of HTTP headers and send your "API key" via a header. If your key is send via header then this key is encrypted by SSL and send encrypted over the network. Never send your api keys by GET parameter like ```http://example.com/?api_key=somekeyvalue```. The problem that this kind of sending presents is that this key is visible in logs and by network sniffers. +There is a lot to be said about security and is a topic of many books. Of course +all this can not be written here but to just establish some basic +security → you should always use SSL with your application. Some fantastic free +certificates are available by [Let's Encrypt - Free SSL/TLS Certificates](https://letsencrypt.org). +With SSL certificate installed you should then make use of HTTP headers and +send your "API key" via a header. If your key is send via header then this +key is encrypted by SSL and send encrypted over the network. Never send your +api keys by GET parameter like ```http://example.com/?api_key=somekeyvalue```. +The problem that this kind of sending presents is that this key is visible in +logs and by network sniffers. There is a fantastic article describing some aspects about security: [11 Web Application Security Best Practices](https://www.keycdn.com/blog/web-application-security-best-practices/). Please check it out. ### Simple API for writing data-points -We will now be using boilerplate code from example above and extend it to be able to write data received by API to local storage. For example use I will use SQLite3 because it plays well with Python and can store quite large amount of data. I have been using it to collect gigabytes of data in a single database without any corruption or problems → your experience may vary. +We will now be using boilerplate code from example above and extend it to be +able to write data received by API to local storage. For example use I will +use SQLite3 because it plays well with Python and can store quite large amount +of data. I have been using it to collect gigabytes of data in a single database +without any corruption or problems → your experience may vary. -To avoid learning SQLite I will be using [Dataset: databases for lazy people](https://dataset.readthedocs.io/en/latest/index.html). This package abstracts SQL and simplifies writing and reading data from database. You should install this package with pip software ```pip install dataset --user```. +To avoid learning SQLite I will be using [Dataset: databases for lazy people](https://dataset.readthedocs.io/en/latest/index.html). +This package abstracts SQL and simplifies writing and reading data from +database. You should install this package with pip software +```pip install dataset --user```. -Because API will use POST method I will be testing if code works correctly by using [Restlet Client for Google Chrome](https://chrome.google.com/webstore/detail/restlet-client-rest-api-t/aejoelaoggembcahagimdiliamlcdmfm). This software also allows you to set headers → for basic security with API_KEY. +Because API will use POST method I will be testing if code works correctly by +using [Restlet Client for Google Chrome](https://chrome.google.com/webstore/detail/restlet-client-rest-api-t/aejoelaoggembcahagimdiliamlcdmfm). +This software also allows you to set headers → for basic security with API_KEY. -To quickly generate passwords or API keys I usually use this nifty website [RandomKeygen](https://randomkeygen.com/). +To quickly generate passwords or API keys I usually use this nifty website +[RandomKeygen](https://randomkeygen.com/). Copy and paste code below over your previous code in file ```webapp.py```. @@ -144,31 +204,53 @@ if __name__ == "__main__": ) ``` -To run this simply go to folder containing python file and run ```python webapp.py``` from terminal. If everything goes ok you should have simple API available via POST method on /api route. +To run this simply go to folder containing python file and run +```python webapp.py``` from terminal. If everything goes ok you should have +simple API available via POST method on /api route. -After testing the service with Restlet Client you should be able to view your data in a database file ```data.db```. +After testing the service with Restlet Client you should be able to view +your data in a database file ```data.db```. ![REST settings example](/assets/iot-application/iot-rest-example.png) -You can also check the contents of new database file by using desktop client for SQLite → [DB Browser for SQLite](http://sqlitebrowser.org/). +You can also check the contents of new database file by using desktop client +for SQLite → [DB Browser for SQLite](http://sqlitebrowser.org/). ![SQLite database example](/assets/iot-application/iot-sqlite-db.png) -Table structure is as simple as it can be. We have ts (timestamp) and value (value from Arduino). As you can see timestamp is generated on API side. If you would happen to have atomic clock on Arduino it would be then better to generate and send timestamp with the value. This would be particularity useful if we would be collecting sensor data at a higher frequency and then sending this data in bulk to API. +Table structure is as simple as it can be. We have ts (timestamp) and +value (value from Arduino). As you can see timestamp is generated on API +side. If you would happen to have atomic clock on Arduino it would be then +better to generate and send timestamp with the value. This would be +particularity useful if we would be collecting sensor data at a higher +frequency and then sending this data in bulk to API. -If you will deploy this app with uWSGI and multi-threaded, use DSN (Data Source Name) url with ```?check_same_thread=False```. +If you will deploy this app with uWSGI and multi-threaded, use +DSN (Data Source Name) url with ```?check_same_thread=False```. -Ok, now that we have some sort of a working API with some basic security so unwanted people can not post data to your database can we proceed further and try to program Arduino to send data to API. +Ok, now that we have some sort of a working API with some basic security +so unwanted people can not post data to your database can we proceed further +and try to program Arduino to send data to API. ## Sending data to API with Arduino MKR1000 -First of all you should have MKR1000 module and microUSB cable to proceed. If you have ever done any work with Arduino you should know that you also need [Arduino IDE](https://www.arduino.cc/en/Main/Software). On provided link you should be able to download and install IDE. Once that task is completed and you have successfully run blink example you should proceed to the next step. +First of all you should have MKR1000 module and microUSB cable to proceed. +If you have ever done any work with Arduino you should know that you also +need [Arduino IDE](https://www.arduino.cc/en/Main/Software). On provided link +you should be able to download and install IDE. Once that task is completed +and you have successfully run blink example you should proceed to the next step. -In order to use wireless capabilities of MKR1000 you need to first install [WiFi101 library](https://www.arduino.cc/en/Reference/WiFi101) in Arduino IDE. Please check before you install, you may already have it installed. +In order to use wireless capabilities of MKR1000 you need to first install +[WiFi101 library](https://www.arduino.cc/en/Reference/WiFi101) in Arduino IDE. +Please check before you install, you may already have it installed. -Code below is a working example that sends data to API. Before you try to test your code make sure you have run Python web application. Then change settings for wifi, api endpoint and api_key. If by some reason code bellow doesn't work for you please leave a comment and I'll try to help. +Code below is a working example that sends data to API. Before you try to test +your code make sure you have run Python web application. Then change settings +for wifi, api endpoint and api_key. If by some reason code bellow doesn't work +for you please leave a comment and I'll try to help. -Once you have opened IDE and copied this code try to compile and upload it. Then open "Serial monitor" to see if any output is presented by Arduino. +Once you have opened IDE and copied this code try to compile and upload it. +Then open "Serial monitor" to see if any output is presented by Arduino. ```c #include @@ -225,7 +307,6 @@ void setup() { } void loop() { - WiFiClient client; if (client.connect(server, port)) { @@ -251,32 +332,54 @@ void loop() { // waits for x seconds and continue looping delay(timeout); - } ``` -As seen from example you can notice that Arduino is generating random integer between [ 0 .. 1000 ]. You can easily replace this with a temperature sensor or any other kind of sensor. +As seen from example you can notice that Arduino is generating random integer +between [ 0 .. 1000 ]. You can easily replace this with a temperature sensor +or any other kind of sensor. -Now that we have API under the hood and Arduino is sending demo data we can now focus on data visualization. +Now that we have API under the hood and Arduino is sending demo data we can +now focus on data visualization. ## Data visualization -Before we continue we should examine our project folder structure. Currently we only have two files in our project: +Before we continue we should examine our project folder structure. Currently +we only have two files in our project: _simple-iot-app/_ * _webapp.py_ * _data.db_ -We will now add HTML template that will contain CSS and JavaScript code inline for the simplicity reason. And for the bottle framework to be able to scan root application folder for templates we will add ```bottle.TEMPLATE_PATH.insert(0, "./")``` in ```webapp.py```. By default bottle framework uses ```views/``` subfolder to store templates. This is not the ideal situation and if you will use bottle to develop web applications you should use native behavior and store templates in it's predefined folder. But for the sake of example we will over-ride this. Be careful to fully replace your code with new code that is provided below. Avoid partially replacing code in file :) Also new code for reading data-points is provided in Python example below. - -First we add new route to our web application. It should be trigger when browser hits root of application ```http://0.0.0.0:5000/```. This route will do nothing more than render ```frontend.html``` template. This is done by ```return bottle.template("frontend.html")```. Check code below to further examine how exactly this is done. - -Now we will expand ```/api``` route and use different methods to write or read data-points. For writing data-point we will use POST method and for reading points we will use GET method. GET method will return JSON object with latest readings and historical data. - -There is a fantastic JavaScript library for plotting time-series charts called [MetricsGraphics.js](https://www.metricsgraphicsjs.org) that is based on [D3.js](https://d3js.org/) library for visualizing data. - -Data schema required by MetricsGraphics.js → to achieve this we need to transform data from database into this format: +We will now add HTML template that will contain CSS and JavaScript code inline +for the simplicity reason. And for the bottle framework to be able to scan root +application folder for templates we will add +```bottle.TEMPLATE_PATH.insert(0, "./")``` in ```webapp.py```. By default bottle +framework uses ```views/``` subfolder to store templates. This is not the ideal +situation and if you will use bottle to develop web applications you should use +native behavior and store templates in it's predefined folder. But for the sake +of example we will over-ride this. Be careful to fully replace your code with +new code that is provided below. Avoid partially replacing code in file :) Also +new code for reading data-points is provided in Python example below. + +First we add new route to our web application. It should be trigger when browser +hits root of application ```http://0.0.0.0:5000/```. This route will do nothing +more than render ```frontend.html``` template. This is done by +```return bottle.template("frontend.html")```. Check code below to further +examine how exactly this is done. + +Now we will expand ```/api``` route and use different methods to write or +read data-points. For writing data-point we will use POST method and for +reading points we will use GET method. GET method will return JSON object +with latest readings and historical data. + +There is a fantastic JavaScript library for plotting time-series charts +called [MetricsGraphics.js](https://www.metricsgraphicsjs.org) that is +based on [D3.js](https://d3js.org/) library for visualizing data. + +Data schema required by MetricsGraphics.js → to achieve this we need to +transform data from database into this format: ```json [ @@ -291,7 +394,9 @@ Data schema required by MetricsGraphics.js → to achieve this we need to transf ] ``` -Web application is now complete and we only need ```frontend.html``` that we will develop now. If you would try to start web app now and go to root app this will return error because we don't have frontend.html yet. +Web application is now complete and we only need ```frontend.html``` that we +will develop now. If you would try to start web app now and go to root app +this will return error because we don't have frontend.html yet. ```python # -*- coding: utf-8 -*- @@ -372,7 +477,9 @@ if __name__ == "__main__": ) ``` -And now finally we can implement ```frontend.html```. Create file with this name and copy code below. When you are done you can start web application. Steps for this part are listed below the code. +And now finally we can implement ```frontend.html```. Create file with this +name and copy code below. When you are done you can start web application. +Steps for this part are listed below the code. ```html @@ -465,9 +572,11 @@ Ok, lets now start application and start feeding it data. 2. connect Arduino MKR1000 to power source 3. open browser and go to ```http://0.0.0.0:5000``` -If everything goes well you should be seeing new data-points rendered on chart every 5 seconds. +If everything goes well you should be seeing new data-points rendered on +chart every 5 seconds. -If you navigate to ```http://0.0.0.0:5000``` you should see rendered chart as shown on picture below. +If you navigate to ```http://0.0.0.0:5000``` you should see rendered chart +as shown on picture below. ![Application output](/assets/iot-application/iot-app-output.png) @@ -475,9 +584,12 @@ Complete application with all the code is available for [download](/assets/iot-a ## Conclusion -I hope this clarifies some aspects of IOT application development. Of course this is a minimal example and is far from what can be done in real life with some further dive into other technologies. +I hope this clarifies some aspects of IOT application development. Of course +this is a minimal example and is far from what can be done in real life with +some further dive into other technologies. -If you would like to continue exploring IOT world here are some interesting resources for you to examine: +If you would like to continue exploring IOT world here are some interesting +resources for you to examine: * [Reading Sensors with an Arduino](https://www.allaboutcircuits.com/projects/reading-sensors-with-an-arduino/) * [MQTT 101 – How to Get Started with the lightweight IoT Protocol](http://www.hivemq.com/blog/how-to-get-started-with-mqtt) @@ -485,3 +597,4 @@ If you would like to continue exploring IOT world here are some interesting reso * [Internet of Things (IoT) Tutorials](http://www.tutorialspoint.com/internet_of_things/) Any comment or additional ideas are welcomed in comments below. + diff --git a/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md b/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md index 75f7799..bfc77ee 100644 --- a/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md +++ b/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md @@ -1,36 +1,67 @@ --- title: Using DigitalOcean Spaces Object Storage with FUSE url: using-digitalocean-spaces-object-storage-with-fuse.html -date: 2018-01-16 +date: 2018-01-16T12:00:00+02:00 draft: false --- -Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced new product called [Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) which is Object Storage very similar to Amazon's S3. This really peaked my interest, because this was something I was missing and even the thought of going over the internet for such functionality was in no interest to me. Also in fashion with their previous pricing this also is very cheap and pricing page is a no-brainer compared to AWS or GCE. [Prices are clearly and precisely defined and outlined](https://www.digitalocean.com/pricing/). You must love them for that :) +Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced +new product called [Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) +which is Object Storage very similar to Amazon's S3. This really peaked my +interest, because this was something I was missing and even the thought of +going over the internet for such functionality was in no interest to me. Also +in fashion with their previous pricing this also is very cheap and pricing +page is a no-brainer compared to AWS or GCE. [Prices are clearly and precisely defined and outlined](https://www.digitalocean.com/pricing/). +You must love them for that :) ## Initial requirements * Is it possible to use them as a mounted drive with FUSE? (tl;dr YES) -* Will the performance degrade over time and over different sizes of objects? (tl;dr NO&YES) -* Can storage be mounted on multiple machines at the same time and be writable? (tl;dr YES) - -> Let me be clear. This scripts I use are made just for benchmarking and are not intended to be used in real-life situations. Besides that, I am looking into using this approaches but adding caching service in front of it and then dumping everything as an object to storage. This could potentially be some interesting post of itself. But in case you would need real-time data without eventual consistency please take this scripts as they are: not usable in such situations. +* Will the performance degrade over time and over different sizes of objects? + (tl;dr NO&YES) +* Can storage be mounted on multiple machines at the same time and be writable? + (tl;dr YES) + +> Let me be clear. This scripts I use are made just for benchmarking and are +> not intended to be used in real-life situations. Besides that, I am looking +> into using this approaches but adding caching service in front of it and then +> dumping everything as an object to storage. This could potentially be some +> interesting post of itself. But in case you would need real-time data without +> eventual consistency please take this scripts as they are: not usable in such +> situations. ## Is it possible to use them as a mounted drive with FUSE? -Well, actually they can be used in such manor. Because they are similar to [AWS S3](https://aws.amazon.com/s3/) many tools are available and you can find many articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse). +Well, actually they can be used in such manor. Because they are similar to +[AWS S3](https://aws.amazon.com/s3/) many tools are available and you can find +many articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse). -To make this work you will need DigitalOcean account. If you don't have one you will not be able to test this code. But if you have an account then you go and [create new Droplet](https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb®ion=ams3&distro=debian&distroImage=debian-9-x64&options=private_networking,install_agent). If you click on this link you will already have preselected Debian 9 with smallest VM option. +To make this work you will need DigitalOcean account. If you don't have one you +will not be able to test this code. But if you have an account then you go and +[create new Droplet](https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb®ion=ams3&distro=debian&distroImage=debian-9-x64&options=private_networking,install_agent). +If you click on this link you will already have preselected Debian 9 with +smallest VM option. -* Please be sure to add you SSH key, because we will login to this machine remotely. -* If you change your region please remember which one you choose because we will need this information when we try to mount space to our machine. +* Please be sure to add you SSH key, because we will login to this machine + remotely. +* If you change your region please remember which one you choose because we + will need this information when we try to mount space to our machine. -Instuctions on how to use SSH keys and how to setup them are available in article [How To Use SSH Keys with DigitalOcean Droplets](https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets). +Instuctions on how to use SSH keys and how to setup them are available in article +[How To Use SSH Keys with DigitalOcean Droplets](https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets). ![DigitalOcean Droplets](/assets/do-fuse/fuse-droplets.png) -After we created Droplet it's time to create new Space. This is done by clicking on a button [Create](https://cloud.digitalocean.com/spaces/new) (right top corner) and selecting Spaces. Choose pronounceable ```Unique name``` because we will use it in examples below. You can either choose Private or Public, it doesn't matter in our case. And you can always change that in the future. +After we created Droplet it's time to create new Space. This is done by clicking +on a button [Create](https://cloud.digitalocean.com/spaces/new) (right top +corner) and selecting Spaces. Choose pronounceable ```Unique name``` because we +will use it in examples below. You can either choose Private or Public, it +doesn't matter in our case. And you can always change that in the future. -When you have created new Space we should [generate Access key](https://cloud.digitalocean.com/settings/api/tokens). This link will guide to the page when you can generate this key. After you create new one, please save provided Key and Secret because Secret will not be shown again. +When you have created new Space we should [generate Access key](https://cloud.digitalocean.com/settings/api/tokens). +This link will guide to the page when you can generate this key. After you +create new one, please save provided Key and Secret because Secret will not +be shown again. ![DigitalOcean Spaces](/assets/do-fuse/fuse-spaces.png) @@ -59,9 +90,14 @@ s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tm echo "Hello cruel world" > /mnt/hello.txt ``` -After all this you can return to your browser and go to [DigitalOcean Spaces](https://cloud.digitalocean.com/spaces) and click on your created space. If file hello.txt is present you have successfully mounted space to your machine and wrote data to it. +After all this you can return to your browser and go to +[DigitalOcean Spaces](https://cloud.digitalocean.com/spaces) and click on your +created space. If file hello.txt is present you have successfully mounted +space to your machine and wrote data to it. -I choose the same region for my Droplet and my Space but you don't have to. You can have different regions. What this actually does to performance I don't know. +I choose the same region for my Droplet and my Space but you don't have to. +You can have different regions. What this actually does to performance I +don't know. Additional information on FUSE: @@ -70,7 +106,9 @@ Additional information on FUSE: ## Will the performance degrade over time and over different sizes of objects? -For this task I didn't want to just read and write text files or uploading images. I actually wanted to figure out if using something like SQlite is viable in this case. +For this task I didn't want to just read and write text files or uploading +images. I actually wanted to figure out if using something like SQlite is +viable in this case. ### Measurement experiment 1: File copy @@ -96,13 +134,17 @@ n=0; while (( n++ < 100 )); do (time cp 1MB.dat /mnt/1MB.$n.dat) |& tee -a 1MB.r n=0; while (( n++ < 100 )); do (time cp 10MB.dat /mnt/10MB.$n.dat) |& tee -a 10MB.results.txt; done ``` -Files of size 100MB were not successfully transferred and ended up displaying error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted). +Files of size 100MB were not successfully transferred and ended up displaying +error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted). -As I suspected, object size is not really that important. Sadly I don't have the time to test performance over periods of time. But if some of you would do it please send me your data. I would be interested in seeing results. +As I suspected, object size is not really that important. Sadly I don't have +the time to test performance over periods of time. But if some of you would do +it please send me your data. I would be interested in seeing results. **Here are plotted results** -You can download [raw result here](/assets/do-fuse/copy-benchmarks.tsv). Measurements are in seconds. +You can download [raw result here](/assets/do-fuse/copy-benchmarks.tsv). +Measurements are in seconds.
@@ -135,11 +177,18 @@ You can download [raw result here](/assets/do-fuse/copy-benchmarks.tsv). Measure })(); -As far as these tests show, performance is quite stable and can be predicted which is fantastic. But this is a small test and spans only over couple of hours. So you should not completely trust them. +As far as these tests show, performance is quite stable and can be predicted +which is fantastic. But this is a small test and spans only over couple of +hours. So you should not completely trust them. ### Measurement experiment 2: SQLite performanse -I was unable to use database file directly from mounted drive so this is a no-go as I suspected. So I executed code below on a local disk just to get some benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT for 1000 times to generate statistics. As you can see performance of SQLite is quite amazing. You could then potentially just copy file to mounted drive and be done with it. +I was unable to use database file directly from mounted drive so this is a +no-go as I suspected. So I executed code below on a local disk just to get +some benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, +INSERTMANY, FETCHALL, COMMIT for 1000 times to generate statistics. As you can +see performance of SQLite is quite amazing. You could then potentially just +copy file to mounted drive and be done with it. ```python import time @@ -218,7 +267,11 @@ result_time = CLOSE = end_time - start_time print("CLOSE: %g seconds" % (result_time)) ``` -You can download [raw result here](/assets/do-fuse/sqlite-benchmarks.tsv). And again, these results are done on a local block storage and do not represent capabilities of object storage. With my current approach and state of the test code these can not be done. I would need to make Python code much more robust and check locking etc. +You can download [raw result here](/assets/do-fuse/sqlite-benchmarks.tsv). And +again, these results are done on a local block storage and do not represent +capabilities of object storage. With my current approach and state of the test +code these can not be done. I would need to make Python code much more robust +and check locking etc.