From bb3fdcd40fe3c297f01afcc446ab33bf0ba3d986 Mon Sep 17 00:00:00 2001 From: Mitja Felicijan Date: Sun, 17 Feb 2019 22:44:22 +0100 Subject: update --- public/404.html | 33 + public/curriculum-vitae.html | 27 + public/encoding-binary-data-into-dna-sequence.html | 170 ++++ public/files/copy-benchmarks.tsv | 101 ++ public/files/dev101-git-topics-1.png | Bin 0 -> 13317 bytes public/files/dev101-git-topics-2.png | Bin 0 -> 12260 bytes public/files/dna-sequence/benchmarks.ods | Bin 0 -> 21911 bytes public/files/dna-sequence/chart-encoding-speed.png | Bin 0 -> 14201 bytes public/files/dna-sequence/chart-file-sizes.png | Bin 0 -> 12391 bytes public/files/dna-sequence/dna-basics.jpg | Bin 0 -> 165883 bytes public/files/dna-sequence/quote.png | Bin 0 -> 1068 bytes public/files/dna-sequence/sample-binary-file.png | Bin 0 -> 66417 bytes public/files/dna-sequence/sample.png | Bin 0 -> 65930 bytes public/files/fuse-droplets.png | Bin 0 -> 42891 bytes public/files/fuse-spaces.png | Bin 0 -> 32450 bytes public/files/golang-profiling-cpu.pdf | Bin 0 -> 16518 bytes public/files/golang-profiling-mem.pdf | Bin 0 -> 19221 bytes public/files/iot-app-output.png | Bin 0 -> 23767 bytes public/files/iot-rest-example.png | Bin 0 -> 33912 bytes public/files/iot-sqlite-db.png | Bin 0 -> 199821 bytes public/files/kcachegrind.png | Bin 0 -> 88486 bytes public/files/profiling-viewer.png | Bin 0 -> 173672 bytes public/files/simple-iot-application-overview.svg | 2 + public/files/simple-iot-application.zip | Bin 0 -> 6406 bytes public/files/snakeviz.png | Bin 0 -> 59601 bytes public/files/sqlite-benchmarks.tsv | 1001 ++++++++++++++++++++ ...-why-i-choose-classic-vms-and-digitalocean.html | 27 + public/golang-profiling-simplified.html | 94 ++ ...y-destroyed-the-joy-of-product-development.html | 27 + public/index.html | 27 + ...-python-web-applications-with-visual-tools.html | 139 +++ public/robots.txt | 2 + public/simple-iot-application.html | 340 +++++++ public/software-development-pitfalls.html | 27 + public/the-bullshit-web-developments-pov.html | 27 + ...gitalocean-spaces-object-storage-with-fuse.html | 190 ++++ public/what-i-ve-learned-developing-ad-server.html | 78 ++ .../what-its-like-to-be-a-software-developer.html | 27 + 38 files changed, 2339 insertions(+) create mode 100644 public/404.html create mode 100644 public/curriculum-vitae.html create mode 100644 public/encoding-binary-data-into-dna-sequence.html create mode 100644 public/files/copy-benchmarks.tsv create mode 100644 public/files/dev101-git-topics-1.png create mode 100644 public/files/dev101-git-topics-2.png create mode 100644 public/files/dna-sequence/benchmarks.ods create mode 100644 public/files/dna-sequence/chart-encoding-speed.png create mode 100644 public/files/dna-sequence/chart-file-sizes.png create mode 100644 public/files/dna-sequence/dna-basics.jpg create mode 100644 public/files/dna-sequence/quote.png create mode 100644 public/files/dna-sequence/sample-binary-file.png create mode 100644 public/files/dna-sequence/sample.png create mode 100644 public/files/fuse-droplets.png create mode 100644 public/files/fuse-spaces.png create mode 100644 public/files/golang-profiling-cpu.pdf create mode 100644 public/files/golang-profiling-mem.pdf create mode 100644 public/files/iot-app-output.png create mode 100644 public/files/iot-rest-example.png create mode 100644 public/files/iot-sqlite-db.png create mode 100644 public/files/kcachegrind.png create mode 100644 public/files/profiling-viewer.png create mode 100644 public/files/simple-iot-application-overview.svg create mode 100644 public/files/simple-iot-application.zip create mode 100644 public/files/snakeviz.png create mode 100644 public/files/sqlite-benchmarks.tsv create mode 100644 public/gce-aws-docker-and-why-i-choose-classic-vms-and-digitalocean.html create mode 100644 public/golang-profiling-simplified.html create mode 100644 public/how-we-successfully-destroyed-the-joy-of-product-development.html create mode 100644 public/index.html create mode 100644 public/profiling-python-web-applications-with-visual-tools.html create mode 100644 public/robots.txt create mode 100644 public/simple-iot-application.html create mode 100644 public/software-development-pitfalls.html create mode 100644 public/the-bullshit-web-developments-pov.html create mode 100644 public/using-digitalocean-spaces-object-storage-with-fuse.html create mode 100644 public/what-i-ve-learned-developing-ad-server.html create mode 100644 public/what-its-like-to-be-a-software-developer.html (limited to 'public') diff --git a/public/404.html b/public/404.html new file mode 100644 index 0000000..829eda8 --- /dev/null +++ b/public/404.html @@ -0,0 +1,33 @@ + + + + + + Page Not Found + + + + +
+

404

+

Page Not Found

+

The specified file was not found on this website. Please check the URL for mistakes and try again.

+

Why am I seeing this?

+

This page was generated by the Firebase Command-Line Interface. To modify it, edit the 404.html file in your project's configured public directory.

+
+ + diff --git a/public/curriculum-vitae.html b/public/curriculum-vitae.html new file mode 100644 index 0000000..8b1cb16 --- /dev/null +++ b/public/curriculum-vitae.html @@ -0,0 +1,27 @@ +Curriculum Vitae

Curriculum Vitae

Mitja Felicijan
Embedded systems developer
mitja.felicijan@gmail.com
Slovenia, EU

Technical experience

  • Key languages: Golang, Python, C/C++, Bash, C#, Java, Perl.
  • Platforms: GNU/Linux, Arch, Debian, Gentoo, Red Hat, Custom distros.
  • Fields of study: Zigbee, KNX, Modbus, Machine to Machine, Embedded systems, Operating systems, Distributed systems, IOT, RDBMS, Algorithms, Database engine design, SQL, NoSQL, NewSQL, Big data analytics, Machine learning, Prediction algorithms, Realtime analytics, Systems automation.

Major projects

  • SMS marketing system (2007)
  • Yacht management software (2008)
  • Smart Home Gateway (2009)
  • Moxa UPort 1130 USB to RS485 Universal Linux driver (2009)
  • Remote management of electricity meter (2009)
  • Remote management of blood pressure monitor (2010)
  • Infomat automation system (2010)
  • GPS Tourist - GIS Software (2011)
  • Minimal GNU/Linux distribution for embedded platforms (2011)
  • Digital Jukebox system (2012)
  • NanoCloudLogger - Machine to Machine (2012)
  • Street Lightning System (2012)
  • Smart cabins with hardware sensor management (2013)
  • Contextual advertising server (2015)
  • Network accessible database engine for caching and in-memory storage (2016)
  • Tick database engine specifically designed for storing and processing large amount of sensor data with high write throughput (2016)
  • Wireless industrial lighting management system - hardware and software (2016)
  • Minimal configuration reverse proxy (2017)
  • Industrial IOT platform for deployment on on-premise (2018)

Employment history

  • Freelancer (2001 – Present)
  • Software developer at Mobinia (2005 – 2007)
  • CTO at Milk (2007 – 2009)
  • Founder and CTO of UTS (2009 – 2014)
  • Founding member of Origami Group (2014 – 2017)
  • Senior Software Engineer at TSmedia (2015 - 2017)

Awards

  • Regional Award for Innovation by Chamber of Commerce and Industry of Slovenia for project Intelligent system management and regulation of Street Lighting, 2010
  • National Award for Innovation by Chamber of Commerce and Industry of Slovenia for project Intelligent system management and regulation of Street Lighting, 2010

Key responsibilities

  • Responsible for embedded platforms development.
  • Responsible for hardware design and driver development.
  • Responsible for the designing, develop and test the systems.
  • Responsible for implementation of the systems.
  • Responsible for writing and maintaining user and technical documents.
  • Responsible for development and maintenance of the project.
  • Responsible for code revision, testing and output.
  • Work on the enhancement suggested by the customers and fixes the bugs reported.)
\ No newline at end of file diff --git a/public/encoding-binary-data-into-dna-sequence.html b/public/encoding-binary-data-into-dna-sequence.html new file mode 100644 index 0000000..0d99625 --- /dev/null +++ b/public/encoding-binary-data-into-dna-sequence.html @@ -0,0 +1,170 @@ +Encoding binary data into DNA sequence

Encoding binary data into DNA sequence

Published on by Mitja Felicijan

Table of contents

  1. Initial thoughts
  2. Glossary
  3. Data encoding
  4. Quick history of DNA
  5. What is DNA?
  6. Encode binary data into DNA sequence
    1. Basic Encoding
    2. FASTA file format
    3. PNG encoded DNA sequence
  7. Encoding text file in practice
  8. Toolkit for encoding data
    1. dnae-encode
    2. dnae-png
  9. Benchmarks
  10. References

Initial thoughts

Imagine a world where you could go outside and take a leaf from a tree and put it through your personal DNA sequencer and get data like music, videos or computer programs from it. Well, this is all possible now. It was not done on a large scale because it is quite expensive to create DNA strands but it’s possible.

Encoding data into DNA sequence is relatively simple process once you understand the relationship between binary data and nucleotides and scientists have been making large leaps in this field in order to provide viable long-term storage solution for our data that would potentially survive our specie if case of global disaster. We could imprint all the world’s knowledge into plants and ensure the survival of our knowledge.

More optimistic usage for this technology would be easier storage of ever growing data we produce every day. Once machines for sequencing DNA become fast enough and cheaper this could mean the next evolution of storing data and abandoning classical hard and solid state drives in data warehouses.

As we currently stand this is still not viable but it is quite an amazing and cool technology.

My interests in this field are purely in encoding processes and experimental testing mainly because I don’t have the access to this expensive machines. My initial goal was to create a toolkit that can be used by everybody to encode their data into a proper DNA sequence.

Glossary

deoxyribose
A five-carbon sugar molecule with a hydrogen atom rather than a hydroxyl group in the 2′ position; the sugar component of DNA nucleotides.
double helix
The molecular shape of DNA in which two strands of nucleotides wind around each other in a spiral shape.
nitrogenous base
A nitrogen-containing molecule that acts as a base; often referring to one of the purine or pyrimidine components of nucleic acids.
phosphate group
A molecular group consisting of a central phosphorus atom bound to four oxygen atoms.
RGB
The RGB color model is an additive color model in which red, green and blue light are added together in various ways to reproduce a broad array of colors.
GCC
The GNU Compiler Collection is a compiler system produced by the GNU Project supporting various programming languages.

Data encoding

TL;DR: Encoding involves the use of a code to change original data into a form that can be used by an external process [1].

Encoding is the process of converting data into a format required for a number of information processing needs, including:

  • Program compiling and execution
  • Data transmission, storage and compression/decompression
  • Application data processing, such as file conversion

Encoding can have two meanings[1:1]:

  • In computer technology, encoding is the process of applying a specific code, such as letters, symbols and numbers, to data for conversion into an equivalent cipher.
  • In electronics, encoding refers to analog to digital conversion.

Quick history of DNA

  • 1869 - Friedrich Miescher identifies “nuclein”.
  • 1900s - The Eugenics Movement.
  • 1900 – Mendel’s theories are rediscovered by researchers.
  • 1944 - Oswald Avery identifies DNA as the ‘transforming principle’.
  • 1952 - Rosalind Franklin photographs crystallized DNA fibres.
  • 1953 - James Watson and Francis Crick discover the double helix structure of DNA.
  • 1965 - Marshall Nirenberg is the first person to sequence the bases in each codon.
  • 1983 - Huntington’s disease is the first mapped genetic disease.
  • 1990 - The Human Genome Project begins.
  • 1995 - Haemophilus Influenzae is the first bacterium genome sequenced.
  • 1996 - Dolly the sheep is cloned.
  • 1999 - First human chromosome is decoded.
  • 2000 – Genetic code of the fruit fly is decoded.
  • 2002 – Mouse is the first mammal to have its genome decoded.
  • 2003 – The Human Genome Project is completed.
  • 2013 – DNA Worldwide and Eurofins Forensic discover identical twins have differences in their genetic makeup [2].

What is DNA?

Deoxyribonucleic acid, a self-replicating material which is present in nearly all living organisms as the main constituent of chromosomes. It is the carrier of genetic information.

The nitrogen in our DNA, the calcium in our teeth, the iron in our blood, the carbon in our apple pies were made in the interiors of collapsing stars. We are made of starstuff.

– Carl Sagan, Cosmos

The nucleotide in DNA consists of a sugar (deoxyribose), one of four bases (cytosine ©, thymine (T), adenine (A), guanine (G)), and a phosphate. Cytosine and thymine are pyrimidine bases, while adenine and guanine are purine bases. The sugar and the base together are called a nucleoside.

DNA

DNA (a) forms a double stranded helix, and (b) adenine pairs with thymine and cytosine pairs with guanine. (credit a: modification of work by Jerome Walker, Dennis Myts) [3]

Encode binary data into DNA sequence

As an input file you can use any file you want:

  • ASCII files,
  • Compiled programs,
  • Multimedia files (MP3, MP4, MVK, etc),
  • Images,
  • Database files,
  • etc.

Note: If you would copy all the bytes from RAM to file or pipe data to file you could encode also this data as long as you provide file pointer to the encoder.

Basic Encoding

As already mentioned, the Basic Encoding is based on a simple mapping. Since DNA is composed of 4 nucleotides (Adenine, Cytosine, Guanine, Thymine; usually referred using the first letter). Using this technique we can encode

$$ log_2(4) = log_2(2^2) = 2 bits $$

using a single nucleotide. In this way, we are able to use the 4 bases that compose the DNA strand to encode each byte of data.

Two bitsNucleotides
00A (Adenine)
10G (Guanine)
01C (Cytosine)
11T (Thymine)

With this in mind we can simply encode any data by using two-bit to Nucleotides conversion

{ Algorithm 1: Naive byte array to DNA encode }
+procedure EncodeToDNASequence(f) string
+begin
+  enc string
+  while not eof(f) do
+    c byte := buffer[0]                             { Read 1 byte from buffer }
+    bin integer := sprintf('08b', c)                { Convert to string binary }
+    for e in range[0, 2, 4, 6] do
+      if e[0] == 48 and e[1] == 48 then             { 0x00 - A (Adenine) }
+        enc += 'A'
+      else if e[0] == 48 and e[1] == 49 then        { 0x01 - G (Guanine) }
+        enc += 'G'
+      else if e[0] == 49 and e[1] == 48 then        { 0x10 - C (Cytosine) }
+        enc += 'C'
+      else if e[0] == 49 and e[1] == 49 then        { 0x11 - T (Thymine) }
+        enc += 'T'
+  return enc                                        { Return DNA sequence }
+end
+

Another encoding would be Goldman encoding. Using this encoding helps with Nonsense mutation (amino acids replaced by a stop codon) that occurs and is the most problematic during translation because it leads to truncated amino acid sequences, which in turn results in truncated proteins. [4]

Where to store big data? In DNA: Nick Goldman at TEDxPrague

FASTA file format

In bioinformatics, FASTA format is a text-based format for representing either nucleotide sequences or peptide sequences, in which nucleotides or amino acids are represented using single-letter codes. The format also allows for sequence names and comments to precede the sequences. The format originates from the FASTA software package, but has now become a standard in the field of bioinformatics. [5]

The first line in a FASTA file started either with a “>” (greater-than) symbol or, less frequently, a “;” (semicolon) was taken as a comment. Subsequent lines starting with a semicolon would be ignored by software. Since the only comment used was the first, it quickly became used to hold a summary description of the sequence, often starting with a unique library accession number, and with time it has become commonplace to always use “>” for the first line and to not use “;” comments (which would otherwise be ignored).

;LCBO - Prolactin precursor - Bovine
+; a sample sequence in FASTA format
+MDSKGSSQKGSRLLLLLVVSNLLLCQGVVSTPVCPNGPGNCQVSLRDLFDRAVMVSHYIHDLSS
+EMFNEFDKRYAQGKGFITMALNSCHTSSLPTPEDKEQAQQTHHEVLMSLILGLLRSWNDPLYHL
+VTEVRGMKGAPDAILSRAIEIEEENKRLLEGMEMIFGQVIPGAKETEPYPVWSGLPSLQTKDED
+ARYSAFYNLLHCLRRDSSKIDTYLKLLNCRIIYNNNC*
+
+>MCHU - Calmodulin - Human, rabbit, bovine, rat, and chicken
+ADQLTEEQIAEFKEAFSLFDKDGDGTITTKELGTVMRSLGQNPTEAELQDMINEVDADGNGTID
+FPEFLTMMARKMKDTDSEEEIREAFRVFDKDGNGYISAAELRHVMTNLGEKLTDEEVDEMIREA
+DIDGDGQVNYEEFVQMMTAK*
+
+>gi|5524211|gb|AAD44166.1| cytochrome b [Elephas maximus maximus]
+LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV
+EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG
+LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL
+GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX
+IENY
+

FASTA format was extended by FASTQ format from the Sanger Centre in Cambridge.

PNG encoded DNA sequence

| Nucleotides | RGB | Color name |
| | – | - |
| A (Adenine) | (0,0,255) | Blue |
| G (Guanine) | (0,100,0) | Green |
| C (Cytosine) | (255,0,0) | Red |
| T (Thymine) | (255,255,0) | Yellow |

With this in mind we can create a simple algorithm to create PNG representation of a DNA sequence.

{ Algorithm 2: Naive DNA to PNG encode from FASTA file }
+procedure EncodeDNASequenceToPNG(f)
+begin
+  i image
+  while not eof(f) do
+    c char := buffer[0]                             { Read 1 char from buffer }
+    case c of
+      'A': color := RGB(0, 0, 255)                  { Blue }
+      'G': color := RGB(0, 100, 0)                  { Green }
+      'C': color := RGB(255, 0, 0)                  { Red }
+      'T': color := RGB(255, 255, 0)                { Yellow }
+    drawRect(i, [x, y], color)
+  save(i)                                           { Save PNG image }
+end
+

Encoding text file in practice

In this example we will take a simple text file as our input stream for encoding. This file will have a quote from Niels Bohr and saved as txt file.

How wonderful that we have met with a paradox. Now we have some hope of making progress.
― Niels Bohr

First we encode text file into FASTA file.

./dnae-encode -i quote.txt -o quote.fa
+2019/01/10 00:38:29 Gathering input file stats
+2019/01/10 00:38:29 Starting encoding ...
+ 106 B / 106 B [==================================] 100.00% 0s
+2019/01/10 00:38:29 Saving to FASTA file ...
+2019/01/10 00:38:29 Output FASTA file length is 438 B
+2019/01/10 00:38:29 Process took 987.263µs
+2019/01/10 00:38:29 Done ...
+

Output of quote.fa file contains the encoded DNA sequence in ASCII format.

>SEQ1
+GACAGCTTGTGTACAAGTGTGCTTGCTCGCGAGCGGGTACGCGCGTGGGCTAACAAGTGA
+GCCAGCAGGTGAACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGCTGGCGGGTGA
+ACAAGTGTGCCGGTGAGCCAACAAGCAGACAAGTAAGCAGGTACGCAGGCGAGCTTGTCA
+ACTCACAAGATCGCTTGTGTACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGTAT
+GCTTGCTGGCGGACAAGCCAGCTTGTAAGCGGACAAGCTTGCGCACAAGCTGGCAGGCCT
+GCCGGCTCGCGTACAAATTCACAAGTAAGTACGCTTGCGTGTACGCGGGTATGTATACTC
+AACCTCACCAAACGGGACAAGATCGCCGGCGGGCTAGTATACAAGAACGCTTGCCAGTAC
+AACC
+

Then we encode FASTA file from previous operation to encode this data into PNG.

./dnae-png -i quote.fa -o quote.png
+2019/01/10 00:40:09 Gathering input file stats ...
+2019/01/10 00:40:09 Deconstructing FASTA file ...
+2019/01/10 00:40:09 Compositing image file ...
+ 424 / 424 [==================================] 100.00% 0s
+2019/01/10 00:40:09 Saving output file ...
+2019/01/10 00:40:09 Output image file length is 1.1 kB
+2019/01/10 00:40:09 Process took 19.036117ms
+2019/01/10 00:40:09 Done ...
+

After encoding into PNG format this file looks like this.

Encoded Quote in PNG format

The larger the input stream is the larger the PNG file would be.

Compiled basic Hello World C program with GCC would look like.

// gcc -O3 -o sample sample.c
+#include <stdio.h>
+
+main() {
+  printf("Hello, world!\n");
+  return 0;
+}
+

Toolkit for encoding data

I have created a toolkit with two main programs:

  • dnae-encode (encodes file into FASTA file)
  • dnae-png (encodes FASTA file into PNG)

Toolkit with full source code is available on github.com/mitjafelicijan/dna-encoding.

dnae-encode

> ./dnae-encode --help
+usage: dnae-encode --input=INPUT [<flags>]
+
+A command-line application that encodes file into DNA sequence.
+
+Flags:
+      --help             Show context-sensitive help (also try --help-long and --help-man).
+  -i, --input=INPUT      Input file (ASCII or binary) which will be encoded into DNA sequence.
+  -o, --output="out.fa"  Output file which stores DNA sequence in FASTA format.
+  -s, --sequence=SEQ1    The description line (defline) or header/identifier line, gives a name and/or a unique identifier for the sequence.
+  -c, --columns=60       Row characters length (no more than 120 characters). Devices preallocate fixed line sizes in software.
+      --version          Show application version.
+

dnae-png

> ./dnae-png --help
+usage: dnae-png --input=INPUT [<flags>]
+
+A command-line application that encodes FASTA file into PNG image.
+
+Flags:
+      --help              Show context-sensitive help (also try --help-long and --help-man).
+  -i, --input=INPUT       Input FASTA file which will be encoded into PNG image.
+  -o, --output="out.png"  Output file in PNG format that represents DNA sequence in graphical way.
+  -s, --size=10           Size of pairings of DNA bases on image in pixels (lower resolution lower file size).
+      --version           Show application version.
+

Benchmarks

First we generate some binary sample data with dd.

dd if=<(openssl enc -aes-256-ctr  -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) of=1KB.bin bs=1KB count=1 iflag=fullblock
+

Our freshly generated 1KB file looks something like this (its full of garbage data as intended).

Sample binary file 1KB

We create following binary files:

  • 1KB.bin
  • 10KB.bin
  • 100KB.bin
  • 1MB.bin
  • 10MB.bin
  • 100MB.bin

After this we create FASTA files for all the binary files by encoding them into DNA sequence.

./dnae-encode -i 100MB.bin -o 100MB.fa
+

Then we GZIP all the FASTA files to see how much the can be compressed.

gzip -9 < 10MB.fa > 10MB.fa.gz
+

Speed of encoding binary file into FASTA format.

File sizes of encoded files and also GZIP-ed variations.

Download ODS file with benchmarks..

References

  1. https://www.techopedia.com/definition/948/encoding ↩︎ ↩︎

  2. https://www.dna-worldwide.com/resource/160/history-dna-timeline ↩︎

  3. https://opentextbc.ca/biology/chapter/9-1-the-structure-of-dna/ ↩︎

  4. https://arxiv.org/abs/1801.04774 ↩︎

  5. https://en.wikipedia.org/wiki/FASTA_format ↩︎

\ No newline at end of file diff --git a/public/files/copy-benchmarks.tsv b/public/files/copy-benchmarks.tsv new file mode 100644 index 0000000..c7a7af4 --- /dev/null +++ b/public/files/copy-benchmarks.tsv @@ -0,0 +1,101 @@ +10KB 100KB 1MB 10MB +0.15 0.187 0.317 0.653 +0.158 0.237 0.192 0.659 +0.134 0.359 0.236 0.604 +0.136 0.292 0.196 0.501 +4.411 4.479 4.376 0.649 +0.134 0.481 0.265 0.608 +0.146 0.266 0.28 0.516 +4.282 0.307 4.549 0.562 +0.152 0.28 0.229 0.512 +0.162 0.37 0.315 0.652 +0.13 4.735 0.222 5.171 +4.29 8.767 0.283 5.076 +4.555 4.682 0.318 4.941 +4.658 4.691 0.177 9.624 +4.778 4.791 4.415 5.114 +8.794 8.604 0.311 5.223 +4.582 4.727 0.234 9.28 +4.596 4.638 0.212 5.064 +4.7 4.65 4.458 5.221 +8.822 9.159 0.191 5.032 +4.628 4.641 0.324 9.226 +4.6 4.921 0.197 5.22 +8.85 4.58 4.405 5.245 +4.65 9.142 0.215 5.168 +4.884 6.67 0.248 9.273 +4.581 4.594 0.248 5.082 +8.864 4.844 4.502 5.121 +4.704 4.656 0.177 5.173 +4.616 8.883 0.209 9.334 +4.729 4.962 4.366 4.966 +8.918 4.682 0.186 6.702 +4.686 4.58 0.168 5.111 +5.123 8.84 4.747 5.084 +4.846 4.732 8.85 5.065 +8.887 4.639 4.824 9.286 +4.681 8.897 4.791 5.104 +4.649 4.682 4.835 5.194 +8.847 4.663 8.929 5.271 +4.568 4.604 4.762 9.444 +4.657 8.74 4.772 5.076 +4.636 4.724 4.838 5.168 +8.778 4.846 9.065 5.057 +4.995 4.571 5.074 9.314 +2.343 9.222 4.818 5.732 +4.742 4.646 8.909 5.32 +4.82 4.842 4.778 5.167 +8.791 4.66 4.759 5.157 +4.835 8.944 4.804 9.323 +4.599 5.594 8.952 5.299 +4.809 4.628 1.567 5.294 +8.744 4.771 5.59 5.018 +4.71 8.919 4.771 9.257 +4.704 4.7 9.003 5.064 +4.765 4.605 4.781 5.185 +8.866 4.669 4.844 5.392 +4.897 8.925 4.786 9.279 +4.568 5.168 8.893 5.1 +4.679 4.757 5.41 5.232 +8.922 4.702 4.7 1.984 +4.669 8.721 4.906 5.366 +4.707 4.555 8.96 5.245 +8.938 4.615 4.89 5.216 +4.608 4.621 4.677 9.237 +4.58 8.954 4.908 5.194 +4.707 4.575 8.968 5.017 +8.822 4.781 4.882 9.714 +4.674 8.833 4.834 5.02 +5.005 4.689 4.762 5.312 +4.732 4.799 9.111 5.286 +8.894 4.675 4.936 5.185 +4.747 8.764 4.739 9.312 +4.785 4.749 4.845 5.34 +4.656 4.705 9.181 5.256 +8.899 4.601 4.739 5.261 +4.594 8.813 4.576 9.329 +4.585 4.716 8.813 5.343 +8.718 4.723 4.819 5.092 +4.725 4.757 4.83 5.061 +4.737 8.899 4.772 9.488 +4.692 4.717 8.831 5.13 +8.841 4.951 4.787 5.309 +4.66 8.895 4.746 5.228 +4.749 4.595 4.833 5.26 +4.715 4.615 8.928 9.381 +8.849 4.651 4.826 5.289 +4.66 8.897 4.802 5.197 +4.588 4.844 4.883 9.311 +4.753 4.888 9.053 5.072 +8.841 4.737 4.75 5.157 +4.794 8.976 5.063 5.196 +4.544 4.673 9.036 9.335 +8.74 4.654 6.377 5.29 +4.729 4.752 5.001 5.048 +4.654 8.98 4.873 5.544 +4.9 4.606 4.723 5.192 +8.757 4.802 5.427 9.056 +4.859 8.969 4.816 5.3 +4.701 4.662 9.002 5.138 +4.943 4.813 4.894 5.15 +8.772 4.721 4.785 9.168 diff --git a/public/files/dev101-git-topics-1.png b/public/files/dev101-git-topics-1.png new file mode 100644 index 0000000..bb09380 Binary files /dev/null and b/public/files/dev101-git-topics-1.png differ diff --git a/public/files/dev101-git-topics-2.png b/public/files/dev101-git-topics-2.png new file mode 100644 index 0000000..39a370a Binary files /dev/null and b/public/files/dev101-git-topics-2.png differ diff --git a/public/files/dna-sequence/benchmarks.ods b/public/files/dna-sequence/benchmarks.ods new file mode 100644 index 0000000..62a8e30 Binary files /dev/null and b/public/files/dna-sequence/benchmarks.ods differ diff --git a/public/files/dna-sequence/chart-encoding-speed.png b/public/files/dna-sequence/chart-encoding-speed.png new file mode 100644 index 0000000..7fb106d Binary files /dev/null and b/public/files/dna-sequence/chart-encoding-speed.png differ diff --git a/public/files/dna-sequence/chart-file-sizes.png b/public/files/dna-sequence/chart-file-sizes.png new file mode 100644 index 0000000..31bfa66 Binary files /dev/null and b/public/files/dna-sequence/chart-file-sizes.png differ diff --git a/public/files/dna-sequence/dna-basics.jpg b/public/files/dna-sequence/dna-basics.jpg new file mode 100644 index 0000000..c2e7f52 Binary files /dev/null and b/public/files/dna-sequence/dna-basics.jpg differ diff --git a/public/files/dna-sequence/quote.png b/public/files/dna-sequence/quote.png new file mode 100644 index 0000000..09fb01c Binary files /dev/null and b/public/files/dna-sequence/quote.png differ diff --git a/public/files/dna-sequence/sample-binary-file.png b/public/files/dna-sequence/sample-binary-file.png new file mode 100644 index 0000000..1e4622a Binary files /dev/null and b/public/files/dna-sequence/sample-binary-file.png differ diff --git a/public/files/dna-sequence/sample.png b/public/files/dna-sequence/sample.png new file mode 100644 index 0000000..30f12da Binary files /dev/null and b/public/files/dna-sequence/sample.png differ diff --git a/public/files/fuse-droplets.png b/public/files/fuse-droplets.png new file mode 100644 index 0000000..d7ce243 Binary files /dev/null and b/public/files/fuse-droplets.png differ diff --git a/public/files/fuse-spaces.png b/public/files/fuse-spaces.png new file mode 100644 index 0000000..4dcc1c5 Binary files /dev/null and b/public/files/fuse-spaces.png differ diff --git a/public/files/golang-profiling-cpu.pdf b/public/files/golang-profiling-cpu.pdf new file mode 100644 index 0000000..15241cb Binary files /dev/null and b/public/files/golang-profiling-cpu.pdf differ diff --git a/public/files/golang-profiling-mem.pdf b/public/files/golang-profiling-mem.pdf new file mode 100644 index 0000000..822e445 Binary files /dev/null and b/public/files/golang-profiling-mem.pdf differ diff --git a/public/files/iot-app-output.png b/public/files/iot-app-output.png new file mode 100644 index 0000000..1c80589 Binary files /dev/null and b/public/files/iot-app-output.png differ diff --git a/public/files/iot-rest-example.png b/public/files/iot-rest-example.png new file mode 100644 index 0000000..3ed86aa Binary files /dev/null and b/public/files/iot-rest-example.png differ diff --git a/public/files/iot-sqlite-db.png b/public/files/iot-sqlite-db.png new file mode 100644 index 0000000..82e1e29 Binary files /dev/null and b/public/files/iot-sqlite-db.png differ diff --git a/public/files/kcachegrind.png b/public/files/kcachegrind.png new file mode 100644 index 0000000..0dc48ab Binary files /dev/null and b/public/files/kcachegrind.png differ diff --git a/public/files/profiling-viewer.png b/public/files/profiling-viewer.png new file mode 100644 index 0000000..a450513 Binary files /dev/null and b/public/files/profiling-viewer.png differ diff --git a/public/files/simple-iot-application-overview.svg b/public/files/simple-iot-application-overview.svg new file mode 100644 index 0000000..817666d --- /dev/null +++ b/public/files/simple-iot-application-overview.svg @@ -0,0 +1,2 @@ + +
Database
Database
Web application


[Not supported by viewer]
Write
datapoint
[Not supported by viewer]
Read
datapoints
[Not supported by viewer]
Arduino
MKR1000
[Not supported by viewer]
Web browser
Web browser
Route: /
[Not supported by viewer]
Route: /api
[Not supported by viewer] \ No newline at end of file diff --git a/public/files/simple-iot-application.zip b/public/files/simple-iot-application.zip new file mode 100644 index 0000000..46d3205 Binary files /dev/null and b/public/files/simple-iot-application.zip differ diff --git a/public/files/snakeviz.png b/public/files/snakeviz.png new file mode 100644 index 0000000..5bab395 Binary files /dev/null and b/public/files/snakeviz.png differ diff --git a/public/files/sqlite-benchmarks.tsv b/public/files/sqlite-benchmarks.tsv new file mode 100644 index 0000000..daa2c21 --- /dev/null +++ b/public/files/sqlite-benchmarks.tsv @@ -0,0 +1,1001 @@ +DROPTABLE CREATETABLE INSERTMANY FETCHALL COMMIT +0.000732 0.000400 0.008133 0.000065 0.000166 +0.000200 0.000214 0.003105 0.000043 0.000171 +0.000246 0.000170 0.006594 0.000044 0.000101 +0.000182 0.000166 0.003892 0.000043 0.000112 +0.000248 0.000654 0.002308 0.000041 0.000090 +0.000240 0.000184 0.002253 0.000053 0.000110 +0.000698 0.000483 0.003737 0.000041 0.000165 +0.000217 0.000179 0.002470 0.000049 0.000107 +0.000243 0.000160 0.002668 0.000054 0.000340 +0.000196 0.000169 0.002247 0.000040 0.000096 +0.000191 0.000162 0.003522 0.000260 0.000102 +0.000195 0.000188 0.002325 0.000041 0.000132 +0.000194 0.000202 0.002291 0.000039 0.000091 +0.000195 0.000196 0.004114 0.000042 0.000108 +0.000204 0.000200 0.002971 0.000040 0.000106 +0.000227 0.000159 0.002208 0.000039 0.000117 +0.000207 0.000176 0.003558 0.000040 0.000124 +0.000255 0.000179 0.002870 0.000040 0.000125 +0.000209 0.000176 0.002248 0.000040 0.000176 +0.000211 0.000174 0.002661 0.000039 0.000180 +0.000208 0.000219 0.002321 0.000039 0.000151 +0.000212 0.000178 0.002609 0.000040 0.000132 +0.000205 0.000209 0.002666 0.000039 0.000126 +0.000205 0.000176 0.002501 0.000041 0.000133 +0.000243 0.000183 0.002220 0.000037 0.000117 +0.000504 0.000173 0.002230 0.000121 0.000414 +0.000270 0.000200 0.002325 0.000040 0.000154 +0.000208 0.000176 0.002386 0.000038 0.000123 +0.000229 0.000182 0.002245 0.000039 0.000127 +0.000211 0.000176 0.002544 0.000039 0.000136 +0.000204 0.000180 0.002133 0.000037 0.000129 +0.000205 0.000178 0.002330 0.000048 0.000146 +0.000210 0.000178 0.002242 0.000039 0.000109 +0.000210 0.000259 0.002766 0.000039 0.000118 +0.000317 0.000495 0.002237 0.000039 0.000195 +0.000454 0.000246 0.002447 0.000040 0.000172 +0.000936 0.000200 0.002305 0.000057 0.000173 +0.000263 0.000178 0.002251 0.000038 0.000166 +0.000240 0.000183 0.002169 0.000068 0.000176 +0.000251 0.000189 0.002221 0.000038 0.000141 +0.000268 0.000215 0.002322 0.000039 0.000226 +0.000287 0.000223 0.002696 0.000045 0.000247 +0.000362 0.000229 0.002551 0.000043 0.000133 +0.000239 0.000200 0.002621 0.000045 0.000133 +0.000634 0.000208 0.002619 0.000046 0.000138 +0.000236 0.000205 0.002589 0.000046 0.000137 +0.000262 0.000205 0.002607 0.000045 0.000142 +0.000239 0.000198 0.002754 0.000044 0.000185 +0.000238 0.000198 0.002593 0.000057 0.000160 +0.000242 0.000221 0.003784 0.000122 0.000174 +0.000242 0.000201 0.002625 0.000054 0.000148 +0.000296 0.000225 0.002934 0.000044 0.000134 +0.000239 0.000245 0.003428 0.000046 0.000158 +0.000261 0.000251 0.002569 0.000046 0.000139 +0.000260 0.000230 0.002603 0.000045 0.000145 +0.000302 0.000212 0.002580 0.000045 0.000176 +0.000794 0.000197 0.002856 0.000046 0.000141 +0.000273 0.000209 0.003173 0.000045 0.000217 +0.000240 0.000201 0.002844 0.000043 0.000167 +0.000389 0.000175 0.004315 0.000055 0.000091 +0.000275 0.000534 0.004991 0.000053 0.000092 +0.000229 0.000215 0.004084 0.000045 0.000074 +0.000172 0.000474 0.002611 0.000043 0.000069 +0.000201 0.000174 0.002485 0.000043 0.000069 +0.000173 0.000220 0.002541 0.000045 0.000068 +0.000167 0.000161 0.002827 0.000043 0.000071 +0.000168 0.000160 0.003512 0.000068 0.000075 +0.000211 0.000167 0.002530 0.000044 0.000069 +0.000193 0.000230 0.003664 0.000046 0.000074 +0.000171 0.000161 0.002575 0.000076 0.000075 +0.000169 0.000161 0.002595 0.000044 0.000076 +0.000981 0.000174 0.002556 0.000045 0.000072 +0.000168 0.000163 0.002568 0.000043 0.000072 +0.000163 0.000158 0.002579 0.000043 0.000386 +0.000168 0.000160 0.002579 0.000059 0.000088 +0.000176 0.000163 0.002559 0.000044 0.000075 +0.000167 0.000161 0.002558 0.000043 0.000075 +0.000169 0.000161 0.002599 0.000043 0.000095 +0.000174 0.000163 0.002633 0.000046 0.000076 +0.000170 0.000165 0.002576 0.000858 0.000079 +0.000169 0.000162 0.002611 0.000044 0.000075 +0.000170 0.000199 0.002621 0.000043 0.000074 +0.000170 0.000167 0.003611 0.000043 0.000073 +0.000171 0.000159 0.002764 0.000046 0.000076 +0.000171 0.000165 0.002639 0.000044 0.000073 +0.000168 0.000162 0.003131 0.000046 0.000075 +0.000170 0.000162 0.002858 0.000044 0.000074 +0.000171 0.000164 0.002841 0.000043 0.000075 +0.000167 0.000161 0.002971 0.000043 0.000074 +0.000170 0.000226 0.002842 0.000044 0.000074 +0.000171 0.000165 0.002822 0.000044 0.000075 +0.000173 0.000160 0.002895 0.000045 0.000073 +0.000167 0.000217 0.002697 0.000044 0.000076 +0.000170 0.000197 0.002699 0.000044 0.000075 +0.000171 0.000163 0.003230 0.000045 0.000097 +0.000170 0.000164 0.003167 0.000046 0.000082 +0.000172 0.000196 0.002559 0.000043 0.000075 +0.000168 0.000165 0.003006 0.000045 0.000075 +0.000176 0.000160 0.002567 0.000043 0.000075 +0.000167 0.000163 0.002757 0.000045 0.000076 +0.000171 0.000162 0.002802 0.000045 0.000076 +0.000169 0.000162 0.003102 0.000043 0.000072 +0.000167 0.000162 0.002624 0.000043 0.000075 +0.000170 0.000161 0.002589 0.000043 0.000072 +0.000222 0.000253 0.002657 0.000045 0.000075 +0.000172 0.000162 0.002586 0.000044 0.000084 +0.000172 0.000165 0.002933 0.000044 0.000075 +0.000169 0.000192 0.002609 0.000044 0.000074 +0.000194 0.000162 0.003020 0.000045 0.000081 +0.000170 0.000164 0.002908 0.000045 0.000076 +0.000169 0.000163 0.002567 0.000042 0.000073 +0.000167 0.000159 0.003071 0.000042 0.000074 +0.000222 0.000163 0.003175 0.000043 0.000076 +0.000167 0.000160 0.002641 0.000046 0.000099 +0.000171 0.000168 0.002586 0.000057 0.000075 +0.000170 0.000168 0.003148 0.000046 0.000075 +0.000171 0.000159 0.002770 0.000041 0.000074 +0.000173 0.000158 0.002643 0.000055 0.000077 +0.000313 0.000174 0.002920 0.000045 0.000075 +0.000170 0.000163 0.002551 0.000044 0.000072 +0.000173 0.000161 0.002599 0.000045 0.000073 +0.000167 0.000160 0.003505 0.000046 0.000075 +0.000171 0.000161 0.002894 0.000045 0.000074 +0.000171 0.000166 0.002572 0.000042 0.000073 +0.000166 0.000160 0.004099 0.000044 0.000102 +0.000181 0.000160 0.002499 0.000046 0.000071 +0.000174 0.000175 0.002560 0.000043 0.000068 +0.000165 0.000168 0.003083 0.000044 0.000070 +0.000210 0.000163 0.002535 0.000040 0.000068 +0.000164 0.000177 0.002906 0.000044 0.000075 +0.000175 0.000227 0.002971 0.000043 0.000073 +0.000167 0.000175 0.003409 0.000046 0.000078 +0.000172 0.000166 0.002640 0.000046 0.000074 +0.000177 0.000164 0.002574 0.000046 0.000076 +0.000170 0.000163 0.002631 0.000046 0.000075 +0.000216 0.000168 0.002596 0.000046 0.000076 +0.000170 0.000163 0.002659 0.000045 0.000074 +0.000172 0.000162 0.002677 0.000046 0.000075 +0.000170 0.000159 0.002604 0.000044 0.000081 +0.000171 0.000161 0.003163 0.000046 0.000076 +0.000171 0.000162 0.002574 0.000313 0.000075 +0.000170 0.000186 0.002988 0.000046 0.000074 +0.000171 0.000162 0.002596 0.000043 0.000077 +0.000168 0.000160 0.002640 0.000055 0.000074 +0.000169 0.000161 0.002567 0.000043 0.000371 +0.000170 0.000162 0.002704 0.000057 0.000078 +0.000255 0.000185 0.002453 0.000293 0.000066 +0.000148 0.000143 0.002169 0.000037 0.000066 +0.000173 0.000141 0.002238 0.000039 0.000085 +0.000154 0.000174 0.002679 0.000041 0.000065 +0.000149 0.000144 0.002187 0.000037 0.000065 +0.000146 0.000140 0.002760 0.000039 0.000071 +0.000147 0.000151 0.002193 0.000039 0.000065 +0.000150 0.000172 0.002207 0.000039 0.000067 +0.000147 0.000141 0.002126 0.000037 0.000060 +0.000191 0.000141 0.002119 0.000036 0.000086 +0.000149 0.000144 0.002440 0.000039 0.000065 +0.000148 0.000143 0.003287 0.000041 0.000068 +0.000152 0.000149 0.002555 0.000040 0.000069 +0.000148 0.000141 0.002203 0.000038 0.000065 +0.000147 0.000139 0.002371 0.000052 0.000075 +0.000148 0.000143 0.002201 0.000037 0.000066 +0.000149 0.000140 0.002186 0.000038 0.000062 +0.000152 0.000154 0.002215 0.000038 0.000062 +0.000149 0.000144 0.002505 0.000039 0.000067 +0.000148 0.000140 0.002216 0.000038 0.000101 +0.000160 0.000144 0.002574 0.000039 0.000067 +0.000150 0.000144 0.002266 0.000040 0.000068 +0.000151 0.000142 0.003640 0.000040 0.000068 +0.000150 0.000142 0.002207 0.000038 0.000066 +0.000148 0.000140 0.002337 0.000041 0.000068 +0.000151 0.000144 0.002138 0.000038 0.000063 +0.000146 0.000178 0.002369 0.000039 0.000060 +0.000150 0.000141 0.002290 0.000039 0.000067 +0.000149 0.000143 0.002569 0.000050 0.000070 +0.000149 0.000143 0.002797 0.000040 0.000068 +0.000149 0.000143 0.002720 0.000039 0.000066 +0.000273 0.000154 0.002255 0.000039 0.000066 +0.000147 0.000141 0.002180 0.000037 0.000065 +0.000884 0.000142 0.002164 0.000036 0.000060 +0.000188 0.000143 0.002248 0.000039 0.000062 +0.000148 0.000142 0.002178 0.000038 0.000064 +0.000151 0.000140 0.002705 0.000038 0.000063 +0.000145 0.000144 0.002588 0.000039 0.000064 +0.000147 0.000142 0.002196 0.000037 0.000064 +0.000147 0.000139 0.002169 0.000035 0.000060 +0.000151 0.000894 0.002267 0.000039 0.000061 +0.000152 0.000145 0.002178 0.000038 0.000061 +0.000185 0.000142 0.002148 0.000036 0.000062 +0.000147 0.000141 0.002845 0.000040 0.000065 +0.000159 0.000178 0.002193 0.000039 0.000063 +0.000145 0.000141 0.002571 0.000039 0.000066 +0.000149 0.000141 0.003380 0.000038 0.000065 +0.000200 0.000149 0.002439 0.000039 0.000066 +0.000152 0.000140 0.002193 0.000037 0.000065 +0.000147 0.000139 0.002239 0.000037 0.000066 +0.000200 0.000143 0.002190 0.000039 0.000066 +0.000147 0.000139 0.002243 0.000038 0.000062 +0.000421 0.000144 0.002229 0.000038 0.000062 +0.000147 0.000149 0.002715 0.000038 0.000063 +0.000151 0.000176 0.002144 0.000036 0.000060 +0.000145 0.000138 0.002184 0.000038 0.000064 +0.000146 0.000207 0.002526 0.000040 0.000067 +0.000163 0.000142 0.002366 0.000038 0.000070 +0.000149 0.000143 0.002143 0.000038 0.000065 +0.000150 0.000142 0.002146 0.000035 0.000059 +0.000162 0.000147 0.002736 0.000038 0.000067 +0.000149 0.000146 0.002383 0.000040 0.000071 +0.000147 0.000139 0.002485 0.000038 0.000065 +0.000147 0.000143 0.002811 0.000039 0.000098 +0.000181 0.000142 0.002503 0.000039 0.000066 +0.000150 0.000143 0.002227 0.000039 0.000065 +0.000149 0.000143 0.002182 0.000036 0.000061 +0.000148 0.000387 0.002159 0.000036 0.000059 +0.000147 0.000173 0.002267 0.000039 0.000063 +0.000147 0.000143 0.002729 0.000039 0.000066 +0.000149 0.000142 0.002574 0.000040 0.000069 +0.000149 0.000143 0.002560 0.000040 0.000068 +0.000152 0.000141 0.002203 0.000038 0.000066 +0.000151 0.000139 0.002234 0.000038 0.000087 +0.000148 0.000140 0.002152 0.000036 0.000060 +0.000185 0.000140 0.002274 0.000039 0.000063 +0.000148 0.000144 0.002211 0.000038 0.000066 +0.000149 0.000141 0.002692 0.000039 0.000066 +0.000148 0.000145 0.002519 0.000039 0.000066 +0.000147 0.000143 0.002188 0.000038 0.000066 +0.000149 0.000171 0.002171 0.000038 0.000093 +0.000150 0.000182 0.002185 0.000038 0.000068 +0.000191 0.000154 0.002172 0.000037 0.000061 +0.000145 0.000140 0.002253 0.000043 0.000065 +0.000147 0.000139 0.002673 0.000038 0.000066 +0.000191 0.000144 0.002740 0.000038 0.000066 +0.000147 0.000142 0.002187 0.000038 0.000064 +0.000146 0.000181 0.002180 0.000038 0.000066 +0.000176 0.000142 0.002152 0.000039 0.000061 +0.000149 0.000142 0.002164 0.000037 0.000064 +0.000245 0.000150 0.002771 0.000055 0.000084 +0.000149 0.000145 0.003006 0.000040 0.000069 +0.000153 0.000144 0.002701 0.000040 0.000067 +0.000149 0.000144 0.002192 0.000038 0.000065 +0.000148 0.000143 0.002220 0.000038 0.000063 +0.000146 0.000140 0.002210 0.000038 0.000062 +0.000157 0.000144 0.002174 0.000038 0.000060 +0.000148 0.000171 0.002208 0.000039 0.000061 +0.000146 0.000141 0.002685 0.000039 0.000064 +0.000146 0.000139 0.002811 0.000038 0.000064 +0.000147 0.000140 0.002234 0.000037 0.000063 +0.000143 0.000143 0.002209 0.000040 0.000066 +0.000149 0.000144 0.002162 0.000037 0.000091 +0.000408 0.000141 0.002140 0.000036 0.000060 +0.000142 0.000149 0.002208 0.000132 0.000061 +0.000148 0.000142 0.002706 0.000040 0.000066 +0.000148 0.000142 0.002502 0.000039 0.000065 +0.000176 0.000144 0.002265 0.000039 0.000066 +0.000150 0.000142 0.002199 0.000039 0.000065 +0.000147 0.000154 0.002201 0.000040 0.000067 +0.000150 0.000142 0.002164 0.000036 0.000094 +0.000183 0.000177 0.002253 0.000039 0.000063 +0.000189 0.000143 0.002480 0.000039 0.000066 +0.000148 0.000141 0.002212 0.000037 0.000064 +0.000150 0.000137 0.002192 0.000037 0.000065 +0.000144 0.000140 0.002271 0.000039 0.000062 +0.000190 0.000171 0.002145 0.000037 0.000061 +0.000146 0.000141 0.005865 0.000099 0.000083 +0.000178 0.000165 0.002792 0.000040 0.000066 +0.000148 0.000233 0.002742 0.000039 0.000079 +0.000157 0.000151 0.002225 0.000039 0.000066 +0.000149 0.000142 0.002215 0.000039 0.000081 +0.000165 0.000141 0.002239 0.000039 0.000081 +0.000150 0.000154 0.002154 0.000036 0.000060 +0.000152 0.000151 0.002216 0.000039 0.000075 +0.000172 0.000141 0.004471 0.000060 0.000092 +0.000250 0.000210 0.002881 0.000040 0.000066 +0.000176 0.000152 0.002262 0.000038 0.000337 +0.000164 0.000154 0.002485 0.000039 0.000074 +0.000149 0.000180 0.002148 0.000039 0.000078 +0.000194 0.000145 0.002345 0.000044 0.000064 +0.000164 0.000201 0.002483 0.000040 0.000062 +0.000148 0.000140 0.002249 0.000038 0.000076 +0.000155 0.000144 0.002504 0.000039 0.000067 +0.000166 0.000150 0.002780 0.000040 0.000079 +0.000150 0.000142 0.002194 0.000038 0.000086 +0.000178 0.000153 0.002360 0.000039 0.000079 +0.000160 0.000154 0.002159 0.000036 0.000079 +0.000195 0.000445 0.002203 0.000038 0.000074 +0.000171 0.000161 0.002220 0.000038 0.000087 +0.000165 0.000151 0.002231 0.000038 0.000088 +0.000149 0.000141 0.003445 0.000040 0.000068 +0.000148 0.000143 0.002465 0.000039 0.000081 +0.000165 0.000150 0.002228 0.000038 0.000067 +0.000160 0.000142 0.003231 0.000039 0.000066 +0.000149 0.000141 0.002215 0.000038 0.000078 +0.000146 0.000152 0.002152 0.000038 0.000077 +0.000168 0.000140 0.002258 0.000040 0.000076 +0.000193 0.000142 0.002266 0.000039 0.000085 +0.000261 0.000164 0.002160 0.000037 0.000061 +0.000151 0.000419 0.002217 0.000037 0.000073 +0.000163 0.000148 0.002856 0.000038 0.000106 +0.000258 0.000204 0.002267 0.000040 0.000075 +0.000178 0.000159 0.002266 0.000038 0.000070 +0.000158 0.000149 0.002665 0.000039 0.000085 +0.000164 0.000154 0.002478 0.000039 0.000077 +0.000148 0.000140 0.002459 0.000038 0.000066 +0.000161 0.000142 0.002206 0.000038 0.000074 +0.000155 0.000151 0.002230 0.000039 0.000083 +0.000161 0.000142 0.002225 0.000037 0.000072 +0.000161 0.000187 0.002450 0.000038 0.000063 +0.000145 0.000155 0.002438 0.000039 0.000079 +0.000166 0.000138 0.002296 0.000039 0.000076 +0.000170 0.000156 0.002446 0.000038 0.000078 +0.000160 0.000159 0.002211 0.000038 0.000078 +0.000159 0.000142 0.002190 0.000036 0.000110 +0.000157 0.000150 0.002336 0.000039 0.000073 +0.000165 0.000182 0.002132 0.000038 0.000072 +0.000160 0.000140 0.002641 0.000066 0.000066 +0.000147 0.000153 0.002153 0.000039 0.000080 +0.000148 0.000156 0.002165 0.000037 0.000077 +0.000147 0.000151 0.002201 0.000038 0.000067 +0.000162 0.000143 0.002216 0.000040 0.000080 +0.000165 0.000148 0.002223 0.000055 0.000080 +0.000193 0.000143 0.002155 0.000037 0.000078 +0.000165 0.000143 0.003005 0.000040 0.000067 +0.000151 0.000145 0.002511 0.000039 0.000070 +0.000149 0.000173 0.002246 0.000039 0.000066 +0.000148 0.000143 0.002808 0.000040 0.000067 +0.000148 0.000142 0.002513 0.000038 0.000066 +0.000148 0.000143 0.002203 0.000037 0.000065 +0.000146 0.000138 0.002123 0.000038 0.000061 +0.000170 0.000149 0.002165 0.000036 0.000062 +0.000144 0.000145 0.002186 0.000037 0.000059 +0.000144 0.000139 0.002520 0.000037 0.000065 +0.000146 0.000139 0.002559 0.000038 0.000066 +0.000153 0.000142 0.002537 0.000038 0.000067 +0.000168 0.000144 0.002217 0.000048 0.000066 +0.000147 0.000141 0.002120 0.000037 0.000063 +0.000188 0.001725 0.002541 0.000040 0.000067 +0.000149 0.000143 0.002229 0.000038 0.000076 +0.000147 0.000143 0.002233 0.000037 0.000062 +0.000182 0.000142 0.002150 0.000037 0.000061 +0.000148 0.000140 0.002196 0.000037 0.000065 +0.000145 0.000140 0.002473 0.000037 0.000065 +0.000147 0.000139 0.002725 0.000040 0.000067 +0.000149 0.000142 0.002217 0.000039 0.000065 +0.000146 0.000140 0.002167 0.000037 0.000061 +0.000176 0.000144 0.002415 0.000039 0.000064 +0.000171 0.000144 0.002925 0.000040 0.000068 +0.000152 0.000167 0.002190 0.000039 0.000066 +0.000149 0.000142 0.002530 0.000039 0.000067 +0.000150 0.000142 0.003059 0.000040 0.000068 +0.000149 0.000142 0.002417 0.000038 0.000072 +0.000149 0.000143 0.002569 0.000038 0.000068 +0.000148 0.000141 0.002262 0.000040 0.000068 +0.000152 0.000144 0.002253 0.000038 0.000066 +0.000149 0.000142 0.002134 0.000037 0.000061 +0.000277 0.000427 0.002186 0.000036 0.000060 +0.000145 0.000139 0.002791 0.000039 0.000065 +0.000149 0.000144 0.002238 0.000039 0.000066 +0.000147 0.000144 0.002514 0.000039 0.000066 +0.000148 0.000143 0.002683 0.000038 0.000063 +0.000147 0.000139 0.002214 0.000037 0.000068 +0.000145 0.000139 0.002149 0.000036 0.000059 +0.000185 0.000139 0.002214 0.000037 0.000060 +0.000145 0.000140 0.003549 0.000039 0.000066 +0.000187 0.000142 0.002160 0.000037 0.000059 +0.000147 0.000158 0.002212 0.000038 0.000065 +0.000148 0.000140 0.002483 0.000039 0.000067 +0.000147 0.000142 0.003034 0.000039 0.000066 +0.000148 0.000142 0.002228 0.000039 0.000066 +0.000145 0.000151 0.002225 0.000040 0.000067 +0.000149 0.000142 0.002858 0.000048 0.000083 +0.000203 0.000185 0.004022 0.000049 0.000086 +0.000212 0.000188 0.005086 0.000056 0.000093 +0.000220 0.000203 0.004209 0.000051 0.000085 +0.000208 0.000247 0.009261 0.000098 0.000089 +0.000211 0.000262 0.002546 0.000041 0.000066 +0.000198 0.000150 0.002534 0.000039 0.000079 +0.000159 0.000143 0.002207 0.000038 0.000094 +0.000157 0.000143 0.002173 0.000038 0.000062 +0.000198 0.000505 0.002157 0.000039 0.000079 +0.000164 0.000143 0.002172 0.000038 0.000076 +0.000156 0.000148 0.002259 0.000039 0.000080 +0.000161 0.000142 0.002219 0.000039 0.000076 +0.000161 0.000143 0.002266 0.000039 0.000085 +0.000161 0.000141 0.002150 0.000036 0.000077 +0.000179 0.000140 0.002140 0.000036 0.000071 +0.000157 0.000151 0.002316 0.000040 0.000079 +0.000149 0.000143 0.002269 0.000039 0.000066 +0.000161 0.000142 0.002206 0.000040 0.000091 +0.000172 0.000143 0.002244 0.000039 0.000067 +0.000168 0.000142 0.002189 0.000039 0.000083 +0.000163 0.000188 0.002156 0.000037 0.000077 +0.000168 0.000143 0.002266 0.000039 0.000084 +0.000166 0.000147 0.002205 0.000325 0.000078 +0.000175 0.000140 0.002173 0.000037 0.000106 +0.000170 0.000153 0.002158 0.000036 0.000083 +0.000168 0.000147 0.002825 0.000039 0.000108 +0.000172 0.000151 0.002483 0.000038 0.000085 +0.000160 0.000143 0.002163 0.000038 0.000066 +0.000161 0.000154 0.002493 0.000039 0.000084 +0.000167 0.000153 0.002564 0.000040 0.000082 +0.000159 0.000151 0.002185 0.000046 0.000088 +0.000157 0.000156 0.002175 0.000039 0.000076 +0.000150 0.000144 0.002151 0.000038 0.000063 +0.000160 0.000140 0.002429 0.000038 0.000064 +0.000160 0.000154 0.002184 0.000048 0.000077 +0.000168 0.000142 0.002686 0.000040 0.000119 +0.000164 0.000152 0.002279 0.000039 0.000075 +0.000161 0.000143 0.002192 0.000068 0.000067 +0.000161 0.000154 0.002190 0.000040 0.000092 +0.000246 0.000146 0.003064 0.000038 0.000072 +0.000163 0.000158 0.002171 0.000037 0.000073 +0.000216 0.000144 0.002209 0.000039 0.000115 +0.000159 0.000141 0.003338 0.000039 0.000079 +0.000277 0.000158 0.002464 0.000039 0.000082 +0.000168 0.000150 0.002227 0.000037 0.000079 +0.000168 0.000146 0.002775 0.000038 0.000077 +0.000146 0.000147 0.002694 0.000042 0.000084 +0.000160 0.000145 0.002807 0.000039 0.000066 +0.000162 0.000177 0.002187 0.000063 0.000066 +0.000147 0.000141 0.002220 0.000038 0.000085 +0.000160 0.000142 0.002216 0.000037 0.000077 +0.000166 0.000159 0.002224 0.000039 0.000108 +0.000147 0.000141 0.002746 0.000039 0.000078 +0.000159 0.000141 0.002194 0.000037 0.000063 +0.000164 0.000143 0.002164 0.000039 0.000067 +0.000169 0.000152 0.002278 0.000074 0.000088 +0.000157 0.000157 0.002155 0.000068 0.000076 +0.000159 0.000140 0.002170 0.000035 0.000078 +0.000156 0.000141 0.002299 0.000040 0.000066 +0.000192 0.000160 0.002241 0.000039 0.000082 +0.000149 0.000143 0.002288 0.000039 0.000079 +0.000161 0.000142 0.002185 0.000049 0.000077 +0.000147 0.000149 0.002284 0.000039 0.000063 +0.000456 0.000144 0.002203 0.000046 0.000064 +0.000187 0.000144 0.002147 0.000037 0.000061 +0.000147 0.000140 0.002238 0.000040 0.000067 +0.000147 0.000140 0.003077 0.000041 0.000068 +0.000151 0.000142 0.002226 0.000038 0.000065 +0.000146 0.000142 0.002188 0.000039 0.000065 +0.000145 0.000141 0.002156 0.000036 0.000061 +0.000143 0.000172 0.002379 0.000037 0.000060 +0.000152 0.000231 0.002172 0.000038 0.000065 +0.000153 0.000142 0.002181 0.000039 0.000065 +0.000148 0.000142 0.002567 0.000039 0.000067 +0.000150 0.000142 0.002177 0.000038 0.000072 +0.000147 0.000146 0.002328 0.000038 0.000063 +0.000146 0.000150 0.002211 0.000038 0.000063 +0.000149 0.000143 0.002222 0.000040 0.000072 +0.000150 0.000144 0.002455 0.000039 0.000065 +0.000147 0.000144 0.002206 0.000039 0.000066 +0.000145 0.000141 0.002153 0.000055 0.000070 +0.000443 0.000144 0.002139 0.000036 0.000069 +0.000147 0.000182 0.002188 0.000037 0.000061 +0.000146 0.000138 0.002248 0.000038 0.000067 +0.000147 0.000142 0.002817 0.000039 0.000067 +0.000148 0.000144 0.002230 0.000038 0.000066 +0.000148 0.000142 0.002239 0.000039 0.000067 +0.000149 0.000142 0.002197 0.000038 0.000063 +0.000181 0.000674 0.002170 0.000038 0.000061 +0.000146 0.000195 0.002204 0.000037 0.000061 +0.000146 0.000141 0.002260 0.000039 0.000067 +0.000150 0.000142 0.002193 0.000045 0.000065 +0.000147 0.000140 0.002229 0.000036 0.000066 +0.000146 0.000137 0.002197 0.000037 0.000062 +0.000152 0.000159 0.002187 0.000036 0.000060 +0.000145 0.000139 0.002224 0.000037 0.000064 +0.000149 0.000144 0.002175 0.000038 0.000066 +0.000150 0.000143 0.002187 0.000038 0.000066 +0.000148 0.000141 0.002152 0.000036 0.000061 +0.000185 0.000141 0.002176 0.000036 0.000064 +0.000169 0.000145 0.002483 0.000038 0.000067 +0.000149 0.000141 0.002225 0.000036 0.000064 +0.000244 0.000149 0.002538 0.000038 0.000065 +0.000156 0.000143 0.002317 0.000039 0.000297 +0.000228 0.000172 0.002222 0.000039 0.000300 +0.000149 0.000145 0.002173 0.000040 0.000066 +0.000154 0.000145 0.002155 0.000038 0.000093 +0.000161 0.000145 0.002178 0.000039 0.000063 +0.000147 0.000170 0.002299 0.000039 0.000066 +0.000149 0.000142 0.003494 0.000040 0.000066 +0.000149 0.000178 0.002237 0.000038 0.000062 +0.000148 0.000143 0.002150 0.000037 0.000064 +0.000146 0.000139 0.002315 0.000038 0.000065 +0.000147 0.000141 0.002269 0.000039 0.000067 +0.000173 0.000145 0.002191 0.000037 0.000065 +0.000166 0.000144 0.002247 0.000038 0.000061 +0.000146 0.000140 0.002551 0.000038 0.000065 +0.000148 0.000175 0.002202 0.000037 0.000064 +0.000145 0.000141 0.002217 0.000038 0.000063 +0.000146 0.000138 0.002164 0.000132 0.000547 +0.000148 0.000144 0.008140 0.000160 0.000893 +0.000311 0.000221 0.004526 0.000058 0.000109 +0.000238 0.000225 0.003475 0.000044 0.000094 +0.000178 0.000177 0.002537 0.000041 0.000087 +0.000172 0.000161 0.002194 0.000048 0.000084 +0.000172 0.000163 0.002177 0.000040 0.000084 +0.001177 0.000156 0.002351 0.000041 0.000325 +0.000167 0.000163 0.002273 0.000040 0.000088 +0.000170 0.000151 0.002245 0.000040 0.000077 +0.000172 0.000896 0.002181 0.000038 0.000080 +0.000202 0.000164 0.002449 0.000038 0.000076 +0.000162 0.000161 0.002188 0.000037 0.000078 +0.000165 0.000154 0.002440 0.000074 0.000091 +0.000167 0.000149 0.002185 0.000039 0.000081 +0.000176 0.000154 0.002427 0.000040 0.000093 +0.000168 0.000154 0.002304 0.000038 0.000105 +0.000672 0.000160 0.002260 0.000038 0.000088 +0.000686 0.000159 0.002207 0.000038 0.000084 +0.000163 0.000154 0.002186 0.000037 0.000077 +0.000173 0.000153 0.002399 0.000038 0.000082 +0.000166 0.000157 0.002709 0.000039 0.000077 +0.000155 0.000149 0.002143 0.000038 0.000097 +0.000166 0.000154 0.003454 0.000051 0.000106 +0.000166 0.000160 0.002539 0.000039 0.000128 +0.000169 0.000149 0.002307 0.000039 0.000085 +0.000170 0.000158 0.002225 0.000040 0.000088 +0.000170 0.000180 0.002165 0.000036 0.000103 +0.000203 0.000160 0.002345 0.000039 0.000075 +0.000173 0.000191 0.002160 0.000038 0.000074 +0.000165 0.000156 0.002243 0.000039 0.000085 +0.000172 0.000154 0.002260 0.000040 0.000090 +0.000163 0.000164 0.002258 0.000040 0.000085 +0.000168 0.000143 0.002755 0.000039 0.000086 +0.000178 0.000155 0.002202 0.000039 0.000075 +0.000164 0.000153 0.002267 0.000038 0.000081 +0.000161 0.000154 0.002158 0.000036 0.000090 +0.000169 0.000158 0.002454 0.000037 0.000061 +0.000162 0.000154 0.002543 0.000038 0.000091 +0.000170 0.000154 0.002168 0.000037 0.000085 +0.000166 0.000151 0.002852 0.000038 0.000087 +0.000167 0.000165 0.002484 0.000039 0.000089 +0.000374 0.000197 0.002217 0.000038 0.000082 +0.000156 0.000150 0.002213 0.000038 0.000112 +0.000683 0.000155 0.002131 0.000038 0.000077 +0.000162 0.000164 0.002199 0.000038 0.000076 +0.000176 0.000154 0.002345 0.000038 0.000089 +0.000175 0.000150 0.002928 0.000039 0.000082 +0.000161 0.000140 0.002528 0.000039 0.000066 +0.000159 0.000151 0.002256 0.000039 0.000075 +0.000155 0.000156 0.002233 0.000040 0.000066 +0.000171 0.000156 0.002149 0.000066 0.000084 +0.000182 0.000154 0.002233 0.000037 0.000117 +0.000166 0.000160 0.002460 0.000037 0.000088 +0.000159 0.000165 0.002891 0.000043 0.000075 +0.000169 0.000143 0.002383 0.000038 0.000084 +0.000162 0.000149 0.002313 0.000039 0.000078 +0.000166 0.000161 0.003837 0.000041 0.000092 +0.000166 0.000144 0.002389 0.000038 0.000078 +0.000185 0.000153 0.002548 0.000040 0.000090 +0.000166 0.000152 0.002943 0.000037 0.000063 +0.000147 0.000140 0.002284 0.000038 0.000066 +0.000145 0.000141 0.002555 0.000038 0.000071 +0.000189 0.000143 0.002235 0.000038 0.000359 +0.000149 0.000140 0.002779 0.000053 0.000089 +0.000211 0.000206 0.002744 0.000040 0.000067 +0.000150 0.000144 0.002471 0.000039 0.000065 +0.000151 0.000140 0.002563 0.000040 0.000064 +0.000148 0.000138 0.002305 0.000039 0.000066 +0.000148 0.000141 0.002162 0.000036 0.000060 +0.000182 0.000145 0.002403 0.000042 0.000063 +0.000152 0.000141 0.002311 0.000039 0.000065 +0.000148 0.000180 0.002192 0.000038 0.000065 +0.000149 0.000141 0.002516 0.000039 0.000066 +0.000147 0.000142 0.002193 0.000040 0.000064 +0.000146 0.000138 0.002194 0.000036 0.000060 +0.000197 0.000142 0.002291 0.000038 0.000063 +0.000148 0.000142 0.002440 0.000039 0.000066 +0.000148 0.000143 0.002228 0.000039 0.000066 +0.000149 0.000140 0.002216 0.000038 0.000067 +0.000148 0.000145 0.002196 0.000038 0.000066 +0.000148 0.000141 0.002157 0.000036 0.000061 +0.000144 0.000175 0.002491 0.000039 0.000063 +0.000147 0.000141 0.002290 0.000039 0.000066 +0.000149 0.000143 0.002508 0.000039 0.000067 +0.000149 0.000142 0.002536 0.000039 0.000067 +0.000150 0.000141 0.003132 0.000046 0.000070 +0.000153 0.000145 0.002202 0.000039 0.000067 +0.000149 0.000143 0.002102 0.000037 0.000067 +0.000989 0.000142 0.002188 0.000063 0.000068 +0.000151 0.000142 0.002229 0.000038 0.000068 +0.001481 0.000141 0.002238 0.000039 0.000070 +0.000148 0.000142 0.002204 0.000037 0.000093 +0.000160 0.000141 0.002138 0.000038 0.000062 +0.000145 0.000141 0.002708 0.000039 0.000065 +0.000147 0.000142 0.002218 0.000039 0.000067 +0.000148 0.000140 0.002759 0.000038 0.000066 +0.000148 0.000139 0.003156 0.000037 0.000067 +0.000185 0.000141 0.002259 0.000040 0.000066 +0.000148 0.000142 0.002226 0.000047 0.000068 +0.000148 0.000142 0.002305 0.000040 0.000090 +0.001000 0.000155 0.002217 0.000064 0.000068 +0.000154 0.000144 0.002554 0.000038 0.000065 +0.000148 0.000141 0.002151 0.000038 0.000066 +0.000146 0.000181 0.003031 0.000039 0.000062 +0.000146 0.000180 0.002254 0.000039 0.000061 +0.000147 0.000143 0.002188 0.000039 0.000065 +0.000147 0.000140 0.002259 0.000039 0.000063 +0.000146 0.000141 0.002238 0.000038 0.000076 +0.000148 0.000141 0.002163 0.000038 0.000061 +0.000153 0.000143 0.002195 0.000043 0.000072 +0.000149 0.000177 0.003291 0.000039 0.000063 +0.000258 0.000153 0.002150 0.000039 0.000066 +0.000157 0.000144 0.002155 0.000037 0.000060 +0.000160 0.001194 0.002269 0.000040 0.000100 +0.000164 0.000151 0.002162 0.000038 0.000078 +0.000163 0.000424 0.002178 0.000036 0.000069 +0.001333 0.000389 0.002249 0.000039 0.000066 +0.000175 0.000142 0.002208 0.000037 0.000102 +0.000443 0.000156 0.002249 0.000040 0.000062 +0.000244 0.001562 0.003049 0.000041 0.000083 +0.000208 0.000183 0.002483 0.000040 0.000068 +0.000164 0.000156 0.002220 0.000040 0.000078 +0.000169 0.000142 0.002694 0.000040 0.000083 +0.000162 0.000152 0.002453 0.000038 0.000077 +0.000157 0.000189 0.002306 0.000040 0.000077 +0.000162 0.000151 0.002200 0.000039 0.000325 +0.000150 0.000142 0.002251 0.000039 0.000066 +0.000172 0.000157 0.002184 0.000039 0.000073 +0.000160 0.000150 0.002678 0.000038 0.000326 +0.000165 0.000151 0.002292 0.000038 0.000094 +0.000162 0.000156 0.002203 0.000037 0.000083 +0.000170 0.000141 0.002175 0.000037 0.000074 +0.000149 0.000166 0.002235 0.000039 0.000071 +0.000161 0.000143 0.002423 0.000036 0.000180 +0.000164 0.000152 0.003095 0.000039 0.000076 +0.000172 0.000153 0.002466 0.000039 0.000115 +0.000151 0.000153 0.002274 0.000039 0.000066 +0.000150 0.000142 0.003179 0.000040 0.000080 +0.000172 0.000159 0.002421 0.000039 0.000083 +0.000159 0.000142 0.002165 0.000037 0.000068 +0.000155 0.000150 0.002233 0.000041 0.000123 +0.000153 0.000158 0.002253 0.000039 0.000571 +0.000203 0.000145 0.002269 0.000041 0.000077 +0.000164 0.000158 0.002176 0.000038 0.000086 +0.000197 0.000144 0.002220 0.000041 0.000080 +0.000174 0.000403 0.002224 0.000039 0.000063 +0.000218 0.000144 0.002150 0.000036 0.000069 +0.000149 0.000141 0.002479 0.000040 0.000079 +0.000163 0.000145 0.002664 0.000039 0.000082 +0.000150 0.000152 0.002446 0.000040 0.000069 +0.000203 0.000154 0.002205 0.000043 0.000077 +0.000160 0.000143 0.002210 0.000039 0.000087 +0.000194 0.000145 0.002167 0.000038 0.000069 +0.000151 0.000154 0.002137 0.000036 0.000079 +0.000162 0.000140 0.002697 0.000037 0.000085 +0.000162 0.000143 0.002233 0.000039 0.000076 +0.000148 0.000144 0.002210 0.000039 0.000065 +0.000151 0.000152 0.003015 0.000041 0.000084 +0.000158 0.000156 0.002730 0.000039 0.000079 +0.000312 0.000165 0.002207 0.000038 0.000076 +0.000167 0.000139 0.002297 0.000040 0.000065 +0.000172 0.000154 0.002205 0.000037 0.000080 +0.000146 0.000149 0.002286 0.000039 0.000076 +0.000164 0.000151 0.002214 0.000038 0.000073 +0.000162 0.000169 0.003110 0.000038 0.000067 +0.000293 0.000144 0.002182 0.000038 0.000060 +0.000157 0.000153 0.003778 0.000049 0.000095 +0.001735 0.000210 0.004360 0.000050 0.000083 +0.000297 0.000198 0.002532 0.000039 0.000072 +0.000185 0.000163 0.002173 0.000039 0.000070 +0.000183 0.000142 0.002122 0.000038 0.000062 +0.000147 0.000145 0.002443 0.000039 0.000066 +0.000149 0.000144 0.002473 0.000040 0.000066 +0.000147 0.000139 0.002949 0.000038 0.000063 +0.000147 0.000139 0.002737 0.000039 0.000066 +0.000199 0.000142 0.002927 0.000038 0.000066 +0.000149 0.000141 0.002188 0.000038 0.000065 +0.000147 0.000144 0.002203 0.000038 0.000066 +0.000149 0.000141 0.002154 0.000037 0.000062 +0.000144 0.000137 0.003526 0.000037 0.000066 +0.000151 0.000153 0.002150 0.000036 0.000060 +0.000145 0.000138 0.002202 0.000037 0.000065 +0.000272 0.000187 0.002477 0.000038 0.000306 +0.000148 0.000141 0.002421 0.000038 0.000067 +0.000147 0.000141 0.002252 0.000039 0.000065 +0.000150 0.000140 0.002144 0.000037 0.000061 +0.000191 0.000144 0.002229 0.000038 0.000060 +0.000145 0.000145 0.002202 0.000038 0.000061 +0.000146 0.000142 0.002418 0.000038 0.000065 +0.000189 0.000171 0.002568 0.000040 0.000066 +0.000150 0.000141 0.002300 0.000039 0.000067 +0.000151 0.000141 0.002199 0.000038 0.000347 +0.000147 0.000140 0.002165 0.000035 0.000061 +0.000151 0.000646 0.002310 0.000040 0.000062 +0.000161 0.000410 0.002195 0.000038 0.000061 +0.000147 0.000141 0.002466 0.000039 0.000066 +0.000147 0.000141 0.003026 0.000038 0.000066 +0.000148 0.000142 0.002223 0.000038 0.000065 +0.000147 0.000142 0.002196 0.000038 0.000067 +0.000147 0.000141 0.002155 0.000044 0.000064 +0.000146 0.000140 0.002354 0.000039 0.000067 +0.000149 0.000143 0.002186 0.000037 0.000062 +0.000150 0.000144 0.002498 0.000040 0.000063 +0.000178 0.000212 0.002453 0.000039 0.000062 +0.000149 0.000177 0.002463 0.000038 0.000063 +0.000147 0.000142 0.002507 0.000038 0.000067 +0.000149 0.000142 0.002717 0.000038 0.000066 +0.000148 0.000141 0.002452 0.000037 0.000065 +0.000147 0.000140 0.002266 0.000039 0.000066 +0.000149 0.000141 0.002183 0.000037 0.000066 +0.000153 0.000142 0.002203 0.000039 0.000067 +0.000152 0.000419 0.002245 0.000040 0.000062 +0.000149 0.000181 0.002181 0.000038 0.000063 +0.000147 0.000142 0.002224 0.000039 0.000066 +0.000147 0.000142 0.002204 0.000038 0.000066 +0.000146 0.000141 0.002250 0.000038 0.000065 +0.000148 0.000141 0.002142 0.000038 0.000063 +0.000156 0.000139 0.002176 0.000036 0.000060 +0.000243 0.000148 0.002768 0.000039 0.000069 +0.000146 0.000204 0.002194 0.000037 0.000065 +0.000147 0.000143 0.003071 0.000039 0.000066 +0.000148 0.000144 0.003489 0.000042 0.000073 +0.000151 0.000151 0.002173 0.000039 0.000064 +0.000146 0.000140 0.003509 0.000038 0.000067 +0.000148 0.000142 0.002191 0.000038 0.000064 +0.000146 0.000139 0.002441 0.000039 0.000117 +0.000174 0.000141 0.002133 0.000038 0.000065 +0.000151 0.000142 0.002257 0.000039 0.000073 +0.000163 0.000147 0.002187 0.000038 0.000061 +0.000146 0.000222 0.002193 0.000038 0.000062 +0.000145 0.000143 0.002434 0.000037 0.000064 +0.000145 0.000139 0.002933 0.000041 0.000066 +0.000146 0.000140 0.002680 0.000037 0.000065 +0.000143 0.000139 0.002217 0.001029 0.000065 +0.000145 0.000139 0.002361 0.000039 0.000067 +0.000150 0.000143 0.002186 0.000068 0.000066 +0.000148 0.000142 0.002149 0.000037 0.000061 +0.000147 0.000181 0.002183 0.000037 0.000061 +0.000146 0.000455 0.002305 0.000038 0.000074 +0.000148 0.000143 0.002223 0.000038 0.000066 +0.000148 0.000141 0.002547 0.000038 0.000066 +0.000148 0.000143 0.002180 0.000038 0.000336 +0.000146 0.000141 0.002102 0.000037 0.000063 +0.000150 0.000145 0.002170 0.000037 0.000067 +0.000152 0.000138 0.002982 0.000038 0.000067 +0.000149 0.000143 0.002419 0.000037 0.000064 +0.000145 0.000195 0.002228 0.000040 0.000067 +0.000148 0.000143 0.002193 0.000038 0.000064 +0.000155 0.000141 0.002166 0.000067 0.000066 +0.000454 0.000176 0.002193 0.000038 0.000063 +0.000186 0.000142 0.002165 0.000035 0.000066 +0.000144 0.000138 0.002542 0.000038 0.000066 +0.000148 0.000143 0.002733 0.000039 0.000066 +0.000147 0.000141 0.002227 0.000038 0.000067 +0.000145 0.000142 0.002764 0.000037 0.000064 +0.000144 0.000138 0.002207 0.000037 0.000065 +0.000147 0.000185 0.002262 0.000038 0.000062 +0.000154 0.000160 0.002163 0.000038 0.000063 +0.000150 0.000145 0.002719 0.000038 0.000065 +0.000145 0.000139 0.002226 0.000037 0.000074 +0.000148 0.000140 0.002517 0.000038 0.000067 +0.000148 0.000142 0.003734 0.000039 0.000067 +0.000147 0.000143 0.002508 0.000039 0.000067 +0.000146 0.000143 0.002288 0.000038 0.000067 +0.000149 0.000143 0.002899 0.000039 0.000067 +0.000150 0.000145 0.002232 0.000037 0.000065 +0.000148 0.000142 0.002169 0.000039 0.000067 +0.000161 0.000141 0.002196 0.000036 0.000060 +0.000145 0.000137 0.002467 0.000040 0.000064 +0.000147 0.000141 0.002168 0.000037 0.000063 +0.000147 0.000139 0.002165 0.000037 0.000064 +0.000146 0.000138 0.002167 0.000036 0.000060 +0.000150 0.000141 0.002326 0.000039 0.000063 +0.000149 0.000179 0.002197 0.000039 0.000063 +0.000148 0.000142 0.002538 0.000039 0.000067 +0.000148 0.000148 0.002555 0.000039 0.000067 +0.000150 0.000144 0.002180 0.000038 0.000066 +0.000245 0.000152 0.002203 0.000038 0.000065 +0.000146 0.000142 0.002118 0.000036 0.000091 +0.000648 0.000141 0.002173 0.000035 0.000058 +0.000142 0.000149 0.002137 0.000037 0.000059 +0.000144 0.000138 0.002191 0.000037 0.000063 +0.000143 0.000137 0.002795 0.000039 0.000065 +0.000147 0.000256 0.002250 0.000038 0.000064 +0.000148 0.000142 0.002231 0.000040 0.000075 +0.000149 0.000143 0.002174 0.000038 0.000061 +0.000182 0.000708 0.002255 0.000038 0.000061 +0.000181 0.000170 0.002222 0.000038 0.000060 +0.000148 0.000141 0.002177 0.000038 0.000065 +0.000147 0.000141 0.002478 0.000039 0.000065 +0.000148 0.000141 0.002191 0.000039 0.000065 +0.000146 0.000139 0.002161 0.000067 0.000063 +0.000157 0.000138 0.002174 0.000036 0.000059 +0.000143 0.000165 0.002396 0.000040 0.000067 +0.000148 0.000141 0.002302 0.000044 0.000067 +0.000148 0.000142 0.002226 0.000043 0.000065 +0.000149 0.000142 0.002198 0.000038 0.000087 +0.000147 0.000143 0.002221 0.000039 0.000066 +0.000146 0.000142 0.002376 0.000065 0.000063 +0.000152 0.000154 0.002201 0.000038 0.000062 +0.000150 0.000142 0.002705 0.000039 0.000067 +0.000149 0.000142 0.002267 0.000039 0.000067 +0.000194 0.000149 0.002347 0.000039 0.000066 +0.000155 0.000141 0.002594 0.000038 0.000066 +0.000148 0.000141 0.002189 0.000038 0.000064 +0.000202 0.000142 0.002155 0.000039 0.000062 +0.000182 0.000146 0.002204 0.000037 0.000061 +0.000146 0.000139 0.002466 0.000037 0.000065 +0.000146 0.000140 0.002463 0.000036 0.000065 +0.000146 0.000139 0.002209 0.000037 0.000063 +0.000145 0.000138 0.002146 0.000036 0.000060 +0.000181 0.000142 0.003356 0.000038 0.000068 +0.000161 0.000142 0.002169 0.000038 0.000062 +0.000146 0.000175 0.002538 0.000039 0.000061 +0.000148 0.000141 0.002482 0.000039 0.000067 +0.000148 0.000144 0.002450 0.000040 0.000066 +0.000149 0.000143 0.002466 0.000043 0.000068 +0.000148 0.000144 0.003551 0.000038 0.000068 +0.000149 0.000142 0.002482 0.000039 0.000066 +0.000149 0.000142 0.002220 0.000039 0.000066 +0.000151 0.000140 0.002199 0.000038 0.000064 +0.000148 0.000184 0.002185 0.000038 0.000066 +0.000145 0.000140 0.002158 0.000036 0.000092 +0.000158 0.000140 0.002262 0.000038 0.000062 +0.000148 0.000143 0.002674 0.000039 0.000066 +0.000148 0.000140 0.002421 0.000039 0.000066 +0.000149 0.000149 0.002433 0.000038 0.000065 +0.000146 0.000172 0.002187 0.000038 0.000065 +0.000146 0.000140 0.002311 0.000039 0.000323 +0.000149 0.000142 0.002180 0.000038 0.000091 +0.000420 0.000143 0.002483 0.000038 0.000063 +0.000685 0.000145 0.002136 0.000035 0.000064 +0.000146 0.000145 0.002433 0.000038 0.000062 +0.000146 0.000139 0.002496 0.000039 0.000066 +0.000149 0.000139 0.003626 0.000041 0.000068 +0.000153 0.000147 0.002272 0.000042 0.000067 +0.000248 0.000155 0.002208 0.000038 0.000063 +0.000146 0.000138 0.002524 0.000038 0.000068 +0.000147 0.000140 0.002176 0.000210 0.000065 +0.000147 0.000140 0.002166 0.000036 0.000060 +0.000144 0.000146 0.002169 0.000036 0.000057 +0.000144 0.000138 0.002207 0.000037 0.000063 +0.000145 0.000138 0.002183 0.000037 0.000062 +0.000145 0.000137 0.002167 0.000036 0.000059 +0.000148 0.000453 0.002310 0.000038 0.000061 +0.000183 0.000855 0.002326 0.000037 0.000061 +0.000146 0.000175 0.002672 0.000036 0.000060 +0.000143 0.000140 0.002238 0.000039 0.000065 +0.000146 0.000139 0.002473 0.000037 0.000064 +0.000146 0.000139 0.002196 0.000039 0.000065 +0.000145 0.000139 0.002141 0.000036 0.000061 +0.000174 0.000397 0.002175 0.000036 0.000059 +0.000143 0.000139 0.002647 0.000037 0.000065 +0.000147 0.000138 0.002196 0.000037 0.000064 +0.000146 0.000138 0.002199 0.000037 0.000063 +0.000146 0.000138 0.002167 0.000036 0.000066 +0.000169 0.000141 0.002156 0.000036 0.000060 +0.000143 0.000139 0.002180 0.000037 0.000065 +0.000144 0.000136 0.002756 0.000039 0.000066 +0.000150 0.000141 0.002919 0.000039 0.000066 +0.000147 0.000140 0.002184 0.000036 0.000065 +0.000145 0.000138 0.002168 0.000036 0.000091 +0.000156 0.000139 0.002169 0.000036 0.000059 +0.000143 0.000139 0.002741 0.000038 0.000065 +0.000147 0.000140 0.002429 0.000037 0.000063 +0.000145 0.000139 0.002226 0.000037 0.000064 +0.000145 0.000139 0.003381 0.000040 0.000066 +0.000153 0.000141 0.002262 0.000038 0.000064 +0.000145 0.000140 0.002137 0.000036 0.000062 +0.000154 0.000650 0.002217 0.000038 0.000063 +0.000184 0.000143 0.002209 0.000038 0.000062 +0.000153 0.000142 0.002907 0.000039 0.000066 +0.000147 0.000142 0.002158 0.000038 0.000064 +0.000146 0.000140 0.002953 0.000039 0.000068 +0.000148 0.000143 0.002208 0.000039 0.000065 +0.000149 0.000139 0.002187 0.000036 0.000065 +0.000144 0.000139 0.002157 0.000036 0.000061 +0.000154 0.000926 0.002139 0.000036 0.000059 +0.000183 0.000140 0.002526 0.000038 0.000062 +0.000148 0.000142 0.002207 0.000038 0.000066 +0.000147 0.000139 0.002790 0.000039 0.000069 +0.000149 0.000144 0.002251 0.000038 0.000066 +0.000151 0.000140 0.002220 0.000039 0.000066 +0.000148 0.000142 0.002523 0.000038 0.000064 +0.000151 0.000138 0.002151 0.000037 0.000065 +0.000147 0.000140 0.002251 0.000037 0.000062 +0.000149 0.000139 0.002607 0.000037 0.000065 +0.000147 0.000141 0.003380 0.000037 0.000066 +0.000147 0.000139 0.002285 0.000069 0.000066 +0.000149 0.000142 0.002566 0.000038 0.000067 +0.000147 0.000142 0.002523 0.000038 0.000067 +0.000152 0.000143 0.002215 0.000038 0.000067 +0.000150 0.000144 0.002243 0.000038 0.000075 +0.000149 0.000141 0.002148 0.000036 0.000063 +0.000182 0.000144 0.002167 0.000036 0.000062 +0.000278 0.000155 0.002631 0.000036 0.000061 +0.000149 0.000139 0.003175 0.000040 0.000066 +0.000156 0.000140 0.002660 0.000038 0.000065 +0.000148 0.000141 0.006171 0.000067 0.000069 +0.000164 0.000142 0.002713 0.000038 0.000064 +0.000161 0.000150 0.002270 0.000038 0.000081 +0.000160 0.000283 0.002276 0.000038 0.000083 +0.000168 0.000150 0.002207 0.000037 0.000072 +0.000151 0.000669 0.002160 0.000038 0.000062 +0.000196 0.000156 0.002363 0.000036 0.000061 +0.000162 0.000141 0.002160 0.000037 0.000077 +0.000147 0.000141 0.002676 0.000038 0.000096 +0.000162 0.000143 0.002263 0.000037 0.000065 +0.000162 0.000141 0.002206 0.000036 0.000080 +0.000146 0.000139 0.002149 0.000036 0.000060 +0.000169 0.000884 0.002163 0.000036 0.000076 +0.000187 0.000140 0.002222 0.000036 0.000061 +0.000145 0.000140 0.002192 0.000037 0.000084 +0.000145 0.000138 0.002619 0.000039 0.000116 +0.000158 0.000149 0.002213 0.000038 0.000089 +0.000145 0.000183 0.002154 0.000038 0.000089 +0.000162 0.000142 0.002142 0.000037 0.000061 +0.000146 0.000178 0.002401 0.000038 0.000062 +0.000145 0.000150 0.002741 0.000037 0.000081 +0.000147 0.000139 0.002360 0.000040 0.000067 +0.000151 0.000153 0.002459 0.000039 0.000075 +0.000148 0.000155 0.002459 0.000037 0.000091 +0.000153 0.000152 0.002174 0.000036 0.000064 +0.000424 0.000149 0.002116 0.000036 0.000068 +0.000166 0.000168 0.002625 0.000038 0.000076 +0.000146 0.000141 0.002957 0.000038 0.000067 +0.000160 0.000142 0.002501 0.000039 0.000079 +0.000147 0.000143 0.002219 0.000038 0.000066 +0.000160 0.000143 0.002771 0.000040 0.000079 +0.000148 0.000150 0.002426 0.000037 0.000082 +0.000146 0.000138 0.002134 0.000036 0.000103 +0.000659 0.000143 0.002197 0.000036 0.000073 +0.000179 0.000153 0.002301 0.000038 0.000074 +0.000147 0.000142 0.002258 0.000038 0.000066 +0.000146 0.000141 0.002210 0.000038 0.000066 +0.000161 0.000141 0.002235 0.000038 0.000084 +0.000145 0.000138 0.002131 0.000036 0.000060 +0.000151 0.000211 0.002265 0.000038 0.000062 +0.000147 0.000142 0.002254 0.000038 0.000067 +0.000148 0.000143 0.002217 0.000038 0.000079 +0.000160 0.000155 0.002229 0.000038 0.000066 +0.000145 0.000142 0.002129 0.000038 0.000065 +0.000165 0.000140 0.002140 0.000036 0.000076 +0.000162 0.000142 0.002452 0.000039 0.000079 +0.000148 0.000143 0.002253 0.000059 0.000068 +0.000164 0.000142 0.003378 0.000039 0.000096 +0.000150 0.000194 0.002192 0.000039 0.000067 +0.000161 0.000152 0.002202 0.000037 0.000077 +0.000160 0.000141 0.002258 0.000039 0.000067 +0.000167 0.000143 0.002706 0.000039 0.000067 +0.000149 0.000155 0.002280 0.000037 0.000100 +0.000174 0.000144 0.002134 0.000037 0.000090 +0.001167 0.000154 0.002224 0.000038 0.000067 +0.000162 0.000155 0.002181 0.000035 0.000065 +0.000773 0.000153 0.002145 0.000036 0.000060 +0.000149 0.000161 0.002160 0.000036 0.000071 +0.000208 0.000144 0.002164 0.000035 0.000060 +0.000143 0.000138 0.002156 0.000036 0.000064 +0.000143 0.000138 0.002225 0.000055 0.000066 +0.000147 0.000141 0.002734 0.000038 0.000065 +0.000145 0.000147 0.002173 0.000037 0.000064 +0.000146 0.000139 0.002112 0.000037 0.000060 +0.000144 0.000137 0.002708 0.000038 0.000064 +0.000144 0.000139 0.002421 0.000037 0.000064 +0.000145 0.000140 0.002449 0.000037 0.000063 +0.000143 0.000138 0.002278 0.000038 0.000064 +0.000145 0.000140 0.002427 0.000040 0.000130 +0.000151 0.000142 0.002155 0.000036 0.000064 +0.000181 0.000139 0.002435 0.000036 0.000060 +0.000145 0.000138 0.003527 0.000038 0.000065 +0.000146 0.000178 0.002178 0.000036 0.000060 +0.000145 0.000138 0.002139 0.000037 0.000065 +0.000145 0.000137 0.003006 0.000037 0.000064 +0.000146 0.000139 0.002204 0.000037 0.000065 +0.000145 0.000139 0.002211 0.000038 0.000062 +0.000182 0.000140 0.002221 0.000036 0.000061 +0.000145 0.000139 0.003169 0.000038 0.000068 +0.000149 0.000174 0.002414 0.000038 0.000066 +0.000147 0.000142 0.002234 0.000038 0.000066 +0.000149 0.000143 0.002678 0.000038 0.000065 +0.000148 0.000141 0.002886 0.000038 0.000066 +0.000145 0.000140 0.002250 0.000038 0.000065 +0.000148 0.000139 0.002181 0.000035 0.000065 +0.000718 0.000142 0.002141 0.000035 0.000059 +0.000189 0.000140 0.002383 0.000036 0.000059 +0.000145 0.000139 0.002206 0.000039 0.000065 +0.000154 0.000186 0.002457 0.000038 0.000066 +0.000190 0.000141 0.002224 0.000038 0.000066 +0.000149 0.000141 0.002151 0.000037 0.000066 +0.000215 0.000143 0.002151 0.000035 0.000061 +0.000144 0.000138 0.002822 0.000039 0.000065 +0.000147 0.000139 0.002275 0.000038 0.000065 +0.000148 0.000141 0.002211 0.000036 0.000064 +0.000146 0.000138 0.002201 0.000037 0.000066 +0.000148 0.000140 0.002273 0.000038 0.000068 +0.000150 0.000144 0.002188 0.000037 0.000063 +0.000152 0.000151 0.002190 0.000037 0.000062 +0.000146 0.000142 0.003145 0.000039 0.000067 +0.000151 0.000139 0.002218 0.000037 0.000065 +0.000145 0.000138 0.002264 0.000037 0.000066 +0.000148 0.000142 0.003011 0.000039 0.000067 +0.000149 0.000141 0.002196 0.000038 0.000065 +0.000146 0.000141 0.002188 0.000036 0.000060 +0.000149 0.000140 0.002190 0.000035 0.000060 +0.000144 0.000137 0.002641 0.000038 0.000064 +0.000146 0.000138 0.002182 0.000043 0.000065 +0.000146 0.000141 0.002216 0.000036 0.000064 +0.000147 0.000139 0.002294 0.000039 0.000068 +0.000657 0.000145 0.002143 0.000037 0.000062 +0.000154 0.000415 0.002237 0.000040 0.000084 diff --git a/public/gce-aws-docker-and-why-i-choose-classic-vms-and-digitalocean.html b/public/gce-aws-docker-and-why-i-choose-classic-vms-and-digitalocean.html new file mode 100644 index 0000000..db12c53 --- /dev/null +++ b/public/gce-aws-docker-and-why-i-choose-classic-vms-and-digitalocean.html @@ -0,0 +1,27 @@ +GCE, AWS, Docker and why I choose classic VM’s and DigitalOcean f

GCE, AWS, Docker and why I choose classic VM’s and DigitalOcean f

Published on by Mitja Felicijan

Table of contents

  1. Docker tools and complexity that comes with it
  2. Lack of real life examples of Docker in action
  3. Ease of deployment
  4. Where to go from here

I have been developing a product for the past few months and one of product’s requirement is the ability to automatically scale quickly on system’s demand.

As most of you probably know system design is much more important then actual code that will drive the product. And this was my main concern when developing this product. I have read anything I could get my hands on about Docker as it was hyped so much in media for the past two years. At a first glance Docker was ideal fit for this platform. But then as I started to seriously experiment with it and developing around it several problems occurred. Well, it would be unfair to call them problems but lets say drawbacks when developing rapidly.

To put it in perspective: this project is basically MVP that needs to automatically scale when new customers signs up. These customers are sending metrics to my system that is later visualized and analyzed. There were some basic requirements that I needed the answer before I choose technology.

  • Pricing involving hardware and infrastructure.
  • Ease of implementation/deployment and scaling.
  • How much will this cost me per customer?

The way I envisioned the architecture was straight forward → simple nodes in cluster that take care of x number of customers (1 node ~ 10 customers). I found that pricing in GCE and AWS is very hard to predict → what the cost will be when system would scale. And this was necessary for me to know in order to make financial projection of costs. This is the most important thing for me at this time as I am deciding on prices we should charge future customers and establish healthy revenue model and subsequently business model. I want this product to organically scale and fuel its future development with money made by product itself → very little startup capital (10 nodes for six months & capital for company expanses). I have made many simulations but could not figure out with at least some certainty what will that cost be. Based on this both of the providers are currently not suited for me. So I choose DigitalOcean. They have really straight forward pricing model and this allowed me to make pretty accurate cost matrix for my infrastructure.

I love hard metrics. By this I mean metrics I can test now and have trust they will hold in the future. This was the reason I found Docker too volatile as containers are spawned and halted and there is really no way in predicting this numbers. I have no problem with spawning multiple VMs and not using them but having basically limited control over that is at this time unacceptable for me.

Docker tools and complexity that comes with it

Probably some of you will correct me on this one, but I find all this management tools like Kubernetes, Swarm etc a bit overkill for a startup project. All this tools are able to scale really massively but they all require extensive knowledge of DevOps. When you are a one man band trying to push a product out, there just is no time to learn these tools and concepts in depth in order to really take advantage of their features. It is much easier to use internal metrics of your app (uwsgi stats server, golang middleware stats) and simply fetch them to one server and visualize them. That task alone took me couple of hours and I had simple metrics system in place that with collaboration of DigitalOcean API enabled me to auto spawn new VMs on demand when users reached max number of users supported by current number of nodes. There is something to say about simplicity of this solution. And I love simple solutions.

Lack of real life examples of Docker in action

I found many HelloWorld examples and tutorials showing how to spawn containers and deploying simple python apps but I haven’t found really clear example of showing how to battle permanent storage with containers, load balancing, disk management, ip & port management.

This is not Docker’s nor community’s fault to be absolutely clear. It just shows that it is not that simple to deploy real-world application with Docker. Maybe my software architecture is not designed with Docker in mind.

Ease of deployment

What I really love about Docker is ease of deployment of your application code via container. Multilayered architecture of Docker images also adds to pros list. And the fact that containers sit on top of host OS makes it very intriguing. But if you use container engine from Google you basically spawn VM’s and run containers in this machines and this takes bare-metal approach out of the equation. So at the end you still use hypervisior. I guess if I had my own hardware servers I would be able to fully take advantage of containers.

Because most of my code in nodes is written in Golang and C++ deployment becomes pretty easy. All I have to do is replace binaries on node and that’s that. To avoid downtime I have two instances of one node and I load balance between them. So when I am updating software I first update on node1.A and then node1.B if first one is successful.

Where to go from here

Docker is amazing technology. But the weird pricing model and steep learning curve for deployment of real live application at this time is too much of a hassle for me. I am sure I could lower costs with Docker approach but it would just took too much time at this stage to implement it properly.

I am currently trying to adapt my project to fit Docker and I believe this would be an interesting solution. Idea is to use one container for one customer. I would just need to find the solution for auto-spawning containers on demand for a specific customer. I would then need a flexible load balancer to correctly forward traffic to container designated for this customer. The problem I have is that I need very flexible storage solution because the amount of data that will be aggregated will scale exponentially and I need to permanently store this on disk. And VM approach is allowing me to precisely calculate per customer per VM how much disk I need. Maybe one of you may have a better solution.

\ No newline at end of file diff --git a/public/golang-profiling-simplified.html b/public/golang-profiling-simplified.html new file mode 100644 index 0000000..d8780c0 --- /dev/null +++ b/public/golang-profiling-simplified.html @@ -0,0 +1,94 @@ +Golang profiling simplified

Golang profiling simplified

Published on by Mitja Felicijan

Table of contents

  1. Where are my pprof files?
  2. Why is my cpu profile empty?
  3. Profiling
    1. Memory profiling
    2. CPU profiling
    3. Generating profiling reports

Many posts have been written regarding profiling in Golang and I haven’t found proper tutorial regarding this. Almost all of them are missing some part of important information and it gets pretty frustrating when you have a deadline and are not finding simple distilled solution.

Nevertheless, after searching and experimenting I have found a solution that works for me and probably should also for you.

Where are my pprof files?

By default pprof files are generated in /tmp/ folder. You can override folder where this files are generated programmatically in your golang code as we will see below in example.

Why is my cpu profile empty?

I have found out that sometimes CPU profile is empty because program was not executing long enough. Programs, that execute too quickly don’t produce pprof file in my cases. Well, file is generated but only contains 4KB of information.

Profiling

As you can see from examples we are executing dummy_benchmark functions to ensure some sort of execution. Memory profiling can be done without such a “complex” function. But CPU profiling needs it.

Both memory and CPU profiling examples are almost the same. Only parameters in main function when calling profile.Start are different. When we set profile.ProfilePath(“.”) we tell profiler to store pprof files in the same folder as our program.

Memory profiling

package main
+
+import (
+  "fmt"
+  "time"
+  "github.com/pkg/profile"
+)
+
+func dummy_benchmark() {
+
+  fmt.Println("first set ...")
+  for i := 0; i < 918231333; i++ {
+    i *= 2
+    i /= 2
+  }
+
+  <-time.After(time.Second*3)
+
+  fmt.Println("sencond set ...")
+  for i := 0; i < 9182312232; i++ {
+    i *= 2
+    i /= 2
+  }
+}
+
+func main() {
+  defer profile.Start(profile.MemProfile, profile.ProfilePath("."), profile.NoShutdownHook).Stop()
+  dummy_benchmark()
+}
+

CPU profiling

package main
+
+import (
+  "fmt"
+  "time"
+  "github.com/pkg/profile"
+)
+
+func dummy_benchmark() {
+
+  fmt.Println("first set ...")
+  for i := 0; i < 918231333; i++ {
+    i *= 2
+    i /= 2
+  }
+
+  <-time.After(time.Second*3)
+
+  fmt.Println("sencond set ...")
+  for i := 0; i < 9182312232; i++ {
+    i *= 2
+    i /= 2
+  }
+}
+
+func main() {
+  defer profile.Start(profile.CPUProfile, profile.ProfilePath("."), profile.NoShutdownHook).Stop()
+  dummy_benchmark()
+}
+

Generating profiling reports

# memory profiling
+go build mem.go
+./mem
+go tool pprof -pdf ./mem mem.pprof > mem.pdf
+
+# cpu profiling
+go build cpu.go
+./cpu
+go tool pprof -pdf ./cpu cpu.pprof > cpu.pdf
+

This will generate PDF document with visualized profile.

\ No newline at end of file diff --git a/public/how-we-successfully-destroyed-the-joy-of-product-development.html b/public/how-we-successfully-destroyed-the-joy-of-product-development.html new file mode 100644 index 0000000..63f0001 --- /dev/null +++ b/public/how-we-successfully-destroyed-the-joy-of-product-development.html @@ -0,0 +1,27 @@ +How we successfully destroyed the joy of product development

How we successfully destroyed the joy of product development

Published on by Mitja Felicijan

No matter how hard we try to reinvent processes in software development we still haven’t found perfect solution for this. And to dismiss SDLC just because it’s something old is as ridiculous as the concept of designers being user experience gurus. As I have written couple of times before designers have their place and is not in the UX community. Most of them probably never heard of Jakob Nielsen and this proves a lot. Don’t get me wrong. There are designers out there that are absolutely amazing in what they do, but most of them are not. Good design has little to do with how things look in my opinion. But it has very much to do with how product behaves. And to take a chance on design look only is scary to me.

I have this huge beef with so called UX “experts”. I really do. From the bottom of my heart. I almost hate them. Well, not the pure ghetto ones. There are many of them out there I am sure of. But I have not had the pleasure to work with such person.

Good UX expert should have programming background and an eye for design. Being UX expert requires you to be analytical and precise. Not really qualities of designers. Design is much more about the feeling and emotional perception. And this two don’t dance well together.

Natural progression of project focused on user should be:

  • detailed requirements and fantastic prototypes/wireframes with detailed user journey diagrams,
  • design focused and restricted to serve requirements,
  • code written just to fulfill design and requirements → nothing more and nothing less → no additional dead code should be allowed,
  • testing should be done on all targeted devices → avoid bugs and you will avoid brand failure.

Designer should never be allowed to have blank canvas. Good software is written because there are many restrictions either in requirements or real world. And most importantly → good software is solving only one problem at the time. I don’t see why this shouldn’t apply to design as well.

Yes yes we get it, but we don’t have the time or the money to do project development like that. Well, you better find it or you will slowly decline into abyss of mediocre companies that have nothing to show for. Clients are not dumb and are in need of quality products and services. It is not enough anymore for a product that just works. It has to be technically precise and functionally on the spot.

When developers and designers are forced to think and work from the scratch many new doors open. New ideas are born how to solve problems that were previously not possible because they were living in a box of limited thought and patterns. If you solve problems always only with your knowledge nothing new can be invented. When there is no room for experimentation there is no room for improvement. You want your developers and designers to be this fountain of innovation and you don’t really let them innovate, you are just slowly closing front doors of your company. Good developers and designers are hard to find and even easier to loose.

Being agile does not mean to be a slave of constant changes. It does not mean that project managers can constantly change requirements at their will. And it sure does not mean that clear vision on product direction should be something we said goodby to. We have perverted initial intention of Manifesto for Agile Software Development as we always do. We have taken it so far and we have all become slaves of advertisement by consulting companies trying to cash in on this “new - but old” concept.

Manifesto for Agile Software Development states:

  • individuals and interactions over processes and tools,
  • working software over comprehensive documentation,
  • customer collaboration over contract negotiation,
  • responding to change over following a plan.

This was written in times when software was developed very differently than how we do it now. We have eliminated many of the problems from old age just by listening to reason and not trendy hyped words that are just tools of marketing strategist to avoid the real issues. Being flat, being agile, being stupid is what I say.

Development and design should be about improving yourself and consequently product you are working on. When this becomes a chore you should probably start thinking about changing companies. People make products not management.

\ No newline at end of file diff --git a/public/index.html b/public/index.html new file mode 100644 index 0000000..ba01d3a --- /dev/null +++ b/public/index.html @@ -0,0 +1,27 @@ +Mitja Felicijan - Embedded systems developer
\ No newline at end of file diff --git a/public/profiling-python-web-applications-with-visual-tools.html b/public/profiling-python-web-applications-with-visual-tools.html new file mode 100644 index 0000000..8198388 --- /dev/null +++ b/public/profiling-python-web-applications-with-visual-tools.html @@ -0,0 +1,139 @@ +Profiling Python web applications with visual tools

Profiling Python web applications with visual tools

Published on by Mitja Felicijan

Table of contents

  1. Simple web-service
  2. Visualize profile
  3. Update 2017-04-22

I have been profiling my software with KCachegrind for a long time now and I was missing this option when I am developing API’s or other web services. I always knew that this is possible but never really took the time and dive into it.

Before we begin there are some requirements. We will need to:

If you are using MacOS you should check out Profiling Viewer or MacCallGrind.

KCachegrind

We will be dividing this post into two main categories:

  • writing simple web-service,
  • visualize profile of this web-service.

Simple web-service

Let’s use virtualenv so we won’t pollute our base system. If you don’t have virtualenv installed on your system you can install it with pip command.

# let's install virtualenv globally
+$ sudo pip install virtualenv
+
+# let's also install pyprof2calltree globally
+$ sudo pip install pyprof2calltree
+
+# now we create project
+$ mkdir demo-project
+$ cd demo-project/
+
+# now let's create folder where we will store profiles
+$ mkdir prof
+
+# now we create empty virtualenv in venv/ folder
+$ virtualenv --no-site-packages venv
+
+# we now need to activate virtualenv
+$ source venv/bin/activate
+
+# you can check if virtualenv was correctly initialized by
+# checking where your python interpreter is located
+# if command bellow points to your created directory and not some
+# system dir like /usr/bin/python then everything is fine
+$ which python
+
+# we can check now if all is good ➜ if ok couple of
+# lines will be displayed
+$ pip freeze
+# appdirs==1.4.3
+# packaging==16.8
+# pyparsing==2.2.0
+# six==1.10.0
+
+# now we are ready to install bottlepy ➜ web micro-framework
+$ pip install bottle
+
+# you can deactivate virtualenv but you will then go
+# under system domain ➜ for now don't deactivate
+$ deactivate
+

We are now ready to write simple web service. Let’s create file app.py and paste code bellow in this newly created file.

# -*- coding: utf-8 -*-
+
+import bottle
+import random
+import cProfile
+
+app = bottle.Bottle()
+
+# this function is a decorator and encapsulates function
+# and performs profiling and then saves it to subfolder
+# prof/function-name.prof
+# in our example only awesome_random_number function will
+# be profiled because it has do_cprofile defined
+def do_cprofile(func):
+  def profiled_func(*args, **kwargs):
+    profile = cProfile.Profile()
+    try:
+      profile.enable()
+      result = func(*args, **kwargs)
+      profile.disable()
+      return result
+    finally:
+      profile.dump_stats("prof/" + str(func.__name__) + ".prof")
+  return profiled_func
+
+
+# we use profiling over specific function with including
+# @do_cprofile above function declaration
+@app.route("/")
+@do_cprofile
+def awesome_random_number():
+  awesome_random_number = random.randint(0, 100)
+  return "awesome random number is " + str(awesome_random_number)
+
+@app.route("/test")
+def test():
+  return "dummy test"
+
+if __name__ == '__main__':
+  bottle.run(
+    app = app,
+    host = "0.0.0.0",
+    port = 4000
+  )
+
+# run with 'python app.py'
+# open browser 'http://0.0.0.0:4000'
+

When browser hits awesome_random_number() function profile is created in prof/ subfolder.

Visualize profile

Now let’s create callgrind format from this cProfile output.

$ cd prof/
+$ pyprof2calltree -i awesome_random_number.prof
+# this creates 'awesome_random_number.prof.log' file in the same folder
+

This file can be opened with visualizing tools listed above. In this case we will be using Profilling Viewer under MacOS. You can open image in new tab. As you can see from this example there is hierarchy of execution order of your code.

Profilling Viewer

Make sure you convert output of the cProfile output every time you want to refresh and take a look at your possible optimizations because cProfile updates .prof file every time browser hits the function.

This is just a simple example but when you are developing real-life applications this can be very illuminating, especially to see which parts of your code are bottlenecks and need to be optimized.

Update 2017-04-22

Reddit user mvt also recommended this awesome web based profile visualizer SnakeViz that directly takes output from cProfile module.

Comment from discussion Profiling Python web applications with visual tools.
# let's install it globally as well
+$ sudo pip install snakeviz
+
+# now let's visualize
+$ cd prof/
+$ snakeviz awesome_random_number.prof
+# this automatically opens browser window and
+# shows visualized profile
+

SnakeViz

Reddit user ccharles suggested a better way for installing pip software by targeting user level instead of using sudo.

Comment from discussion Profiling Python web applications with visual tools.
# now we need to add this path to our $PATH variable
+# we do this my adding this line at the end of your
+# ~/.bashrc file
+PATH=$PATH:$HOME/.local/bin/
+
+# in order to use this new configuration you can close
+# and reopen terminal or reload .bashrc file
+$ source ~/.bashrc
+
+# now let's test if new directory is present in $PATH
+$ echo $PATH
+
+# now we can install on user level by adding --user
+# without use of sudo
+$ pip install snakeviz --user
+

Or as suggested by mvt you can use pipsi.

\ No newline at end of file diff --git a/public/robots.txt b/public/robots.txt new file mode 100644 index 0000000..eb05362 --- /dev/null +++ b/public/robots.txt @@ -0,0 +1,2 @@ +User-agent: * +Disallow: diff --git a/public/simple-iot-application.html b/public/simple-iot-application.html new file mode 100644 index 0000000..c24e970 --- /dev/null +++ b/public/simple-iot-application.html @@ -0,0 +1,340 @@ +Simple IOT application supported by real-time monitoring and data

Simple IOT application supported by real-time monitoring and data

Published on by Mitja Felicijan

Table of contents

  1. Initial thoughts
  2. Simple Python API
    1. Basic web application
    2. Web application security
    3. Simple API for writing data-points
  3. Sending data to API with Arduino MKR1000
  4. Data visualization
  5. Conclusion

Initial thoughts

I have been developing these kind of application for the better part of my last 5 years and people keep asking me how to approach developing such application and I will give a try explaining it here.

IOT applications are really no different than any other kind of applications. We have data that needs to be collected and visualized in some form of tables or charts. The main difference here is that most of the times these data is collected by some kind of device foreign to developer that mainly operates in web domain. But fear not, it’s not that different than writing some JavaScript.

There are many devices able to transmit data via wireless or wired network by default but for the sake of example we will be using commonly known Arduino with wireless module already on the board → Arduino MKR1000.

In order to make this little project as accessible to others as possible I will try to make it as inexpensive as possible. And by this I mean that I will avoid using hosted virtual servers and will be using my own laptop as a server. But you must buy Arduino MKR1000 to follow steps below. But if you would want to deploy this software I would suggest using DigitalOcean → smallest VPS is only per month making this one of the most affordable option out there. Please notice that this software will not run on stock web hosting that only supports LAMP (Linux, Apache, MySQL, and PHP).

But before we begin please take notice that this is strictly experimental code and not well optimized and there are much better ways in handling some aspects of the application but that requires much deeper knowledge of technology that is not needed for an example like this.

Development steps

  1. Simple Python API that will receive and store incoming data.
  2. Prototype C++ code that will read “sensor data” and transmit it to API.
  3. Data visualization with charts → extends Python web application.

Step 1. and 3. will share the same web application. One route will be dedicated to API and another to serving HTML with chart.

Schema below represents what we will try to achieve and how different parts correlates to each other.

Overview

Simple Python API

I have always been a fan of simplicity so we will be using Bottle: Python Web Framework. It is a single file web framework that seriously simplifies working with routes, templating and has built-in web server that satisfies our need in this case.

First we need to install bottle package. This can be done by downloading bottle.py and placing it in the root of your application or by using pip software pip install bottle --user.

If you are using Linux or MacOS then Python is already installed. If you will try to test this on Windows please install Python for Windows. There may be some problems with path when you will try to launch python webapp.py so please take care of this before you continue.

Basic web application

Most basic bottle application is quite simple. Paste code below in webapp.py file and save.

# -*- coding: utf-8 -*-
+
+import bottle
+
+# initializing bottle app
+app = bottle.Bottle()
+
+# triggered when / is accessed from browser
+# only accepts GET → no POST allowed
+@app.route("/", method=["GET"])
+def route_default():
+  return "howdy from python"
+
+# starting server on http://0.0.0.0:5000
+if __name__ == "__main__":
+  bottle.run(
+    app = app,
+    host = "0.0.0.0",
+    port = 5000,
+    debug = True,
+    reloader = True,
+    catchall = True,
+  )
+

To run this simple application you should open command prompt or terminal on your machine and go to the folder containing your file and type python webapp.py. If everything goes ok then open your web browser and point it to http://0.0.0.0:5000.

If you would like change the port of your application (like port 80) and not use root to run your app this will present a problem. The TCP/IP port numbers below 1024 are privileged ports → this is a security feature. So in order of simplicity and security use a port number above 1024 like I have used port 5000.

If this fails at any time please fix it before you continue, because nothing below will work otherwise.

We use 0.0.0.0 as default host so that this app is available over your local network. If you find your local ip ifconfig and try accessing this site with your phone (if on same network/router as your machine) this should work as well (example of such ip http://192.168.1.15:5000). This is a must have because Arduino will be accessing this application to send it’s data.

Web application security

There is a lot to be said about security and is a topic of many books. Of course all this can not be written here but to just establish some basic security → you should always use SSL with your application. Some fantastic free certificates are available by Let’s Encrypt - Free SSL/TLS Certificates. With SSL certificate installed you should then make use of HTTP headers and send your “API key” via a header. If your key is send via header then this key is encrypted by SSL and send encrypted over the network. Never send your api keys by GET parameter like http://example.com/?api_key=somekeyvalue. The problem that this kind of sending presents is that this key is visible in logs and by network sniffers.

There is a fantastic article describing some aspects about security: 11 Web Application Security Best Practices. Please check it out.

Simple API for writing data-points

We will now be using boilerplate code from example above and extend it to be able to write data received by API to local storage. For example use I will use SQLite3 because it plays well with Python and can store quite large amount of data. I have been using it to collect gigabytes of data in a single database without any corruption or problems → your experience may vary.

To avoid learning SQLite I will be using Dataset: databases for lazy people. This package abstracts SQL and simplifies writing and reading data from database. You should install this package with pip software pip install dataset --user.

Because API will use POST method I will be testing if code works correctly by using Restlet Client for Google Chrome. This software also allows you to set headers → for basic security with API_KEY.

To quickly generate passwords or API keys I usually use this nifty website RandomKeygen.

Copy and paste code below over your previous code in file webapp.py.

# -*- coding: utf-8 -*-
+
+import time
+import bottle
+import random
+import dataset
+
+# initializing bottle app
+app = bottle.Bottle()
+
+# connects to sqlite database
+# check_same_thread=False allows using it in multi-threaded mode
+app.config["dsn"] = dataset.connect("sqlite:///data.db?check_same_thread=False")
+
+# api key that will be used in Arduino code
+app.config["api_key"] = "JtF2aUE5SGHfVJBCG5SH"
+
+# triggered when /api is accessed from browser
+# only accepts POST → no GET allowed
+@app.route("/api", method=["POST"])
+def route_default():
+  status = 400
+  ts = int(time.time()) # current timestamp
+  value = bottle.request.body.read() # data from device
+  api_key = bottle.request.get_header("Api_Key") # api key from header
+
+  # outputs to console received data for debug reason
+  print ">>> {} :: {}".format(value, api_key)
+
+  # if api_key is correct and value is present
+  # then writes attribute to point table
+  if api_key == app.config["api_key"] and value:
+    app.config["dsn"]["point"].insert(dict(ts=ts, value=value))
+    status = 200
+
+  # we only need to return status
+  return bottle.HTTPResponse(status=status, body="")
+
+# starting server on http://0.0.0.0:5000
+if __name__ == "__main__":
+  bottle.run(
+    app = app,
+    host = "0.0.0.0",
+    port = 5000,
+    debug = True,
+    reloader = True,
+    catchall = True,
+  )
+

To run this simply go to folder containing python file and run python webapp.py from terminal. If everything goes ok you should have simple API available via POST method on /api route.

After testing the service with Restlet Client you should be able to view your data in a database file data.db.

REST settings example

You can also check the contents of new database file by using desktop client for SQLite → DB Browser for SQLite.

SQLite database example

Table structure is as simple as it can be. We have ts (timestamp) and value (value from Arduino). As you can see timestamp is generated on API side. If you would happen to have atomic clock on Arduino it would be then better to generate and send timestamp with the value. This would be particularity useful if we would be collecting sensor data at a higher frequency and then sending this data in bulk to API.

If you will deploy this app with uWSGI and multi-threaded, use DSN (Data Source Name) url with ?check_same_thread=False.

Ok, now that we have some sort of a working API with some basic security so unwanted people can not post data to your database can we proceed further and try to program Arduino to send data to API.

Sending data to API with Arduino MKR1000

First of all you should have MKR1000 module and microUSB cable to proceed. If you have ever done any work with Arduino you should know that you also need Arduino IDE. On provided link you should be able to download and install IDE. Once that task is completed and you have successfully run blink example you should proceed to the next step.

In order to use wireless capabilities of MKR1000 you need to first install WiFi101 library in Arduino IDE. Please check before you install, you may already have it installed.

Code below is a working example that sends data to API. Before you try to test your code make sure you have run Python web application. Then change settings for wifi, api endpoint and api_key. If by some reason code bellow doesn’t work for you please leave a comment and I’ll try to help.

Once you have opened IDE and copied this code try to compile and upload it. Then open “Serial monitor” to see if any output is presented by Arduino.

#include <WiFi101.h>
+
+// wifi settings
+char ssid[] = "ssid-name";
+char pass[] = "ssid-password";
+
+// api server enpoint
+char server[] = "192.168.6.22";
+int port = 5000;
+
+// api key that must be the same as the one in Python code
+String api_key = "JtF2aUE5SGHfVJBCG5SH";
+
+// frequency data is sent in ms - every 5 seconds
+int timeout = 1000 * 5;
+
+int status = WL_IDLE_STATUS;
+
+void setup() {
+
+  // initialize serial and wait for port to open:
+  Serial.begin(9600);
+  delay(1000);
+
+  // check for the presence of the shield
+  if (WiFi.status() == WL_NO_SHIELD) {
+    Serial.println("WiFi shield not present");
+    while (true);
+  }
+
+  // attempt to connect to wifi network
+  while (status != WL_CONNECTED) {
+    Serial.print("Attempting to connect to SSID: ");
+    Serial.println(ssid);
+    status = WiFi.begin(ssid, pass);
+    // wait 10 seconds for connection
+    delay(10000);
+  }
+
+  // output wifi status to serial monitor
+  Serial.print("SSID: ");
+  Serial.println(WiFi.SSID());
+
+  IPAddress ip = WiFi.localIP();
+  Serial.print("IP Address: ");
+  Serial.println(ip);
+
+  long rssi = WiFi.RSSI();
+  Serial.print("signal strength (RSSI):");
+  Serial.print(rssi);
+  Serial.println(" dBm");
+}
+
+void loop() {
+
+  WiFiClient client;
+
+  if (client.connect(server, port)) {
+
+    // I use random number generator for this example
+    // but you can use analog or digital inputs from arduino
+    String content = String(random(1000));
+
+    client.println("POST /api HTTP/1.1");
+    client.println("Connection: close");
+    client.println("Api-Key: " + api_key);
+    client.println("Content-Length: " + String(content.length()));
+    client.println();
+    client.println(content);
+
+    delay(100);
+    client.stop();
+    Serial.println("Data sent successfully ...");
+
+  } else {
+    Serial.println("Problem sending data ...");
+  }
+
+  // waits for x seconds and continue looping
+  delay(timeout);
+
+}
+

As seen from example you can notice that Arduino is generating random integer between [ 0 … 1000 ]. You can easily replace this with a temperature sensor or any other kind of sensor.

Now that we have API under the hood and Arduino is sending demo data we can now focus on data visualization.

Data visualization

Before we continue we should examine our project folder structure. Currently we only have two files in our project:

simple-iot-app/

  • webapp.py
  • data.db

We will now add HTML template that will contain CSS and JavaScript code inline for the simplicity reason. And for the bottle framework to be able to scan root application folder for templates we will add bottle.TEMPLATE_PATH.insert(0, "./") in webapp.py. By default bottle framework uses views/ subfolder to store templates. This is not the ideal situation and if you will use bottle to develop web applications you should use native behavior and store templates in it’s predefined folder. But for the sake of example we will over-ride this. Be careful to fully replace your code with new code that is provided below. Avoid partially replacing code in file :) Also new code for reading data-points is provided in Python example below.

First we add new route to our web application. It should be trigger when browser hits root of application http://0.0.0.0:5000/. This route will do nothing more than render frontend.html template. This is done by return bottle.template("frontend.html"). Check code below to further examine how exactly this is done.

Now we will expand /api route and use different methods to write or read data-points. For writing data-point we will use POST method and for reading points we will use GET method. GET method will return JSON object with latest readings and historical data.

There is a fantastic JavaScript library for plotting time-series charts called MetricsGraphics.js that is based on D3.js library for visualizing data.

Data schema required by MetricsGraphics.js → to achieve this we need to transform data from database into this format:

[
+  {
+    "date": "2017-08-11 01:07:20",
+    "value": 933
+  },
+  {
+    "date": "2017-08-11 01:07:30",
+    "value": 743
+  }
+]
+

Web application is now complete and we only need frontend.html that we will develop now. If you would try to start web app now and go to root app this will return error because we don’t have frontend.html yet.

# -*- coding: utf-8 -*-
+
+import time
+import bottle
+import json
+import datetime
+import random
+import dataset
+
+# initializing bottle app
+app = bottle.Bottle()
+
+# adds root directory as template folder
+bottle.TEMPLATE_PATH.insert(0, "./")
+
+# connects to sqlite database
+# check_same_thread=False allows using it in multi-threaded mode
+app.config["db"] = dataset.connect("sqlite:///data.db?check_same_thread=False")
+
+# api key that will be used in Arduino code
+app.config["api_key"] = "JtF2aUE5SGHfVJBCG5SH"
+
+# triggered when / is accessed from browser
+# only accepts GET → no POST allowed
+@app.route("/", method=["GET"])
+def route_default():
+  return bottle.template("frontend.html")
+
+# triggered when /api is accessed from browser
+# accepts POST and GET
+@app.route("/api", method=["GET", "POST"])
+def route_default():
+
+  # if method is POST then we write datapoint
+  if bottle.request.method == "POST":
+    status = 400
+    ts = int(time.time()) # current timestamp
+    value = bottle.request.body.read() # data from device
+    api_key = bottle.request.get_header("Api-Key") # api key from header
+
+    # outputs to console recieved data for debug reason
+    print ">>> {} :: {}".format(value, api_key)
+
+    # if api_key is correct and value is present
+    # then writes attribute to point table
+    if api_key == app.config["api_key"] and value:
+      app.config["db"]["point"].insert(dict(ts=ts, value=value))
+      status = 200
+
+      # we only need to return status
+      return bottle.HTTPResponse(status=status, body="")
+
+  # if method is GET then we read datapoint
+  else:
+    response = []
+    datapoints = app.config["db"]["point"].all()
+
+    for point in datapoints:
+      response.append({
+        "date": datetime.datetime.fromtimestamp(int(point["ts"])).strftime("%Y-%m-%d %H:%M:%S"),
+        "value": point["value"]
+      })
+
+    bottle.response.content_type = "application/json"
+    return json.dumps(response)
+
+# starting server on http://0.0.0.0:5000
+if __name__ == "__main__":
+  bottle.run(
+    app = app,
+    host = "0.0.0.0",
+    port = 5000,
+    debug = True,
+    reloader = True,
+    catchall = True,
+  )
+

And now finally we can implement frontend.html. Create file with this name and copy code below. When you are done you can start web application. Steps for this part are listed below the code.

<!DOCTYPE html>
+<html>
+
+  <head>
+    <meta charset="utf-8">
+    <title>Simple IOT application</title>
+  </head>
+
+  <body>
+
+    <h1>Simple IOT application</h1>
+
+    <div class="chart-placeholder">
+      <div id="chart"></div>
+    </div>
+
+    <!-- application main script -->
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/d3/4.10.0/d3.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/metrics-graphics/2.11.0/metricsgraphics.min.js"></script>
+    <script>
+      function fetch_and_render() {
+        d3.json("/api", function(data) {
+          data = MG.convert.date(data, "date", "%Y-%m-%d %H:%M:%S");
+          MG.data_graphic({
+            data: data,
+            chart_type: "line",
+            full_width: true,
+            height: 270,
+            target: document.getElementById("chart"),
+            x_accessor: "date",
+            y_accessor: "value"
+          });
+        });
+      }
+      window.onload = function() {
+        // initial call for rendering
+        fetch_and_render();
+
+        // updates chart every 5 seconds
+        setInterval(function() {
+          fetch_and_render();
+        }, 5000);
+      }
+    </script>
+
+    <!-- application styles -->
+    <style>
+      body {
+        font: 13px sans-serif;
+        padding: 20px 50px;
+      }
+      .chart-placeholder {
+        border: 2px solid #ccc;
+        width: 100%;
+        user-select: none;
+      }
+      /* chart styles */
+      .mg-line1-color {
+        stroke: red;
+        stroke-width: 2;
+      }
+      .mg-main-area, .mg-main-line {
+        fill: #fff;
+      }
+      .mg-x-axis line, .mg-y-axis line {
+        stroke: #b3b2b2;
+        stroke-width: 1px;
+      }
+    </style>
+
+  </body>
+
+</html>
+

Now the folder structure should look like:

simple-iot-app/

  • webapp.py
  • data.db
  • frontend.html

Ok, lets now start application and start feeding it data.

  1. python webapp.py
  2. connect Arduino MKR1000 to power source
  3. open browser and go to http://0.0.0.0:5000

If everything goes well you should be seeing new data-points rendered on chart every 5 seconds.

If you navigate to http://0.0.0.0:5000 you should see rendered chart as shown on picture below.

Application output

Complete application with all the code is available for download.

Conclusion

I hope this clarifies some aspects of IOT application development. Of course this is a minimal example and is far from what can be done in real life with some further dive into other technologies.

If you would like to continue exploring IOT world here are some interesting resources for you to examine:

Any comment or additional ideas are welcomed in comments below.)

\ No newline at end of file diff --git a/public/software-development-pitfalls.html b/public/software-development-pitfalls.html new file mode 100644 index 0000000..9a4ad83 --- /dev/null +++ b/public/software-development-pitfalls.html @@ -0,0 +1,27 @@ +Software development and my favorite pitfalls

Software development and my favorite pitfalls

Published on by Mitja Felicijan

Table of contents

  1. Initial thoughts
  2. Ping emails
  3. Everybody is a project manager
  4. We are never wrong
  5. Micromanaging
  6. Human contact, no need for it!
  7. MVP is killing innovation
  8. Pressure wasteland
  9. Conclusion

Initial thoughts

Over the years I had privilege to work on some very excited projects both in software development field and also in electronics field and every experience taught me some invaluable lessons about how NOT TO approach development. And through this post I will try to point out some of the absurd outdated techniques I find the most annoying and damaging during a development cycle. There will be swearing because this topic really gets on my nerves and I never coherently tried to explain them in writing. So if I get heated up please bare with me :)

As new methods of project management are emerging, underlaying processes still stay old and outdated. This is mainly because we as people are unable to completely shift away from this approaches.

I was always struggling with communication, and many times that cost me a relationship or two because I was not on the ball all the time. Through every experience I became more convinced that I am the problem and never ever doubted that the problem may be that communication never evolved a single step from emails. And if you think for a second, not many thing have changed around this topic. We just have different representations of email (message boards, chats, project management tools). And I believe this is the real problem we are facing now.

There are many articles written about hyper connectivity and the effects that are a direct result of it. But mainstream does nothing towards it. We are just putting out fires and we do nothing to prevent it. I am certain this will be a major source of grief in coming years. And what we all can do to avoid this is to change our mindset and experiment on our communication skills, development approaches. We need to maximize possible output that a person can give. And to achieve this we need to listen to them, encourage them. I know that not everybody is a naturally born leader, but everybody has an opinion.

There are many talks right now about methodologies such as Scrum, Kanban, Cleanroom and they all fucking piss me of :). These are all boxes that imprison people and take away their freedom of thought. This is a straight forward mindfuck / amputation of creativity.

Let me list a couple of things that I find really destructive and bad for a project and in a long run company.

Ping emails

Ping emails are emails you have to write as soon as you receive an email. It’s sole purpose is to inform sender that you received their email and you are working on it. It’s result is only to calm down the sender that their task is being dealt with. It’s intent basically is, I did my job by sending you this email so I am on clear ground. I categorize this email as fuck you email. This is one of the most irritating types of emails I need to write. This is the ultimate control freak show you can experience and it gives sender false feeling of control. Newsflash: We do not live in 1982 where there was a possibility that email never reached destination. I really fucking hate this from the bottom of my heart :)

They should be like: “Yes, I am fucking alive and I am at your service my leash!”. I guess if I would reply like this, I wouldn’t have to write any more of this kind of messages :)

Everybody is a project manager

Well, this is a tough one. I noticed that as soon as you let people to give their suggestions you are basically fucked. There is a truth in saying: “Give low expectations and deliver little bit more you promised.”.

People tend to take a role of a manager as soon as they are presented with an opportunity. And by getting angry at them you only provoke yourself. They are not at fault. You just need to tell them they are only giving suggestions and not tasks at the beginning and everything will be alright. But if you give them a feeling that they are in control you will have immense problems explaining why their features are not in current release.

Project mission must be always on the top of project requirements and any deviation from it will result in major project butchering. And by this I mean that project will get it’s own path and you will be left with half done software that helps nobody. Clear mission goal and clean execution will allow you to develop software will clear intent.

We are never wrong

I find this type of arrogance the worst. We must always conduct ourselves that we are infallible and cannot make mistakes. As soon as procedure or process is established there is no room for changes or improvements. This is the most idiotic thing someone can say of think. If think that processes need to involve and change over time. This is imperative and need to have in your organization if you want to improve and develop company. We all need to grow balls and change everything in order to adapt to current situations. Being a prisoner of predefined processes kills creativity.

I am constantly trying new software for project managing and communication. I believe every team has it’s own dynamic and it needs to be discovered organically and naturally through many experiments. By putting team in a box you are amputating their creativity and therefore minimizing their potential. But if you talk to an executive you will mainly find archetypical thinking and a strong need to compartmentalize everything from business processes to resource management. And this type of management that often displays micro management technique only works on short periods (couple of years) and then employees either leave company or become basically retarded drones on auto pilot.

Micromanaging

This basically implies that everybody on the team is a fucking idiot that needs to have a todo list that they can not write themselves. How about spoon feeding the team at launch because besides the team leader everybody must be a retarded idiot at best.

I prefer milestones as they give developers much more freedom and creativity developing and not waste their time checking some bizarre todo list that was not even thought through. Project always changes through development cycle and all you are left at the end is a list of unchecked tasks and the wrath of management why they are not completed. Best WTF moment!

Human contact, no need for it!

We are vigorously trying to eliminate physical contact by replacing short meetings with software with no regards that we are not machines. Many times a simple 5min meeting at morning can solve most of the problems. In rapid development short bursts of man to man communication is possibly the best way to go.

We now have all this software available and all what we get out of it is a huge clusterfuck. An obstacle and not a solution. So why we still use them? Because we strive to better ourselves.

MVP is killing innovation

Many will disagree with me on this one but I stand strong by this statement. What I noticed in my experience that all this buzz words surrounding us only mislead and capture you in a circle of solving a problem that already has a solution but we are unable to see it without using some fancy word for it. The toughest this to do for a developer is to minimize requirements. Well this is though only for bad developers. Yes, I said is. There are many types of developers out there. And those unable to minimize feature scope are the ones you don’t need on your team. Their only goal is to solve problems that exist only in their fucking heads. And than you have to argue with them and waste energy on them instead of developing your awesome product. They are a cancer and I suggest you cut them off.

MVP as an idea is great but sadly people don’t understand underlaying philosophy and they spent too much time focusing and fixating on something that every sane person with normal IQ will understand without some made up acronym. And the result is a lot of talking and barely no execution.

Well MVP is not directly killing innovation but stupid people do when they try to understand it.

Pressure wasteland

You must never allow to be pressured into confirming a deadline if you are not sure. We often feel a need that we are in service of others which is true to some extent. But it is also true that others are in service to us to some extent. And we forget this. We are all pressured all the time to make decisions just to calm other people down. And when they leave your office you experience WTF moment :) How the hell did they manage to fuck me up again :)

People need to realize that more pressure you put on somebody less they will be able todo. So 5 min update email requests will only resolve in mental breakdown and inability to work that day. Constant poking is probably the only thing I loose my mind instantly. For all you that are doing this: “We are not fucking idiots and stop bothering us with your own insecurities and let us do our job. We will do it quicker and better without you moron breathing on our necks.”

If this happens to me I end up with no energy at the end. Don’t you get it? You will get much more from and out of me if you ask me like a human person and not your personal butler. On a long run you are destroying your relationships and nobody would want to work with you. Your schizophrenic approach will damage only you in a long run. Nobody is anybody’s property.

Conclusion

I am guilty of many things described in this post. And I find it hard sometimes to acknowledge this. And I lie to myself and try vigorously to find some explanation why I do this things. There is always space for growth. And maybe you will also find some of yourself in this post and realize what needs to change in order to evolve.

\ No newline at end of file diff --git a/public/the-bullshit-web-developments-pov.html b/public/the-bullshit-web-developments-pov.html new file mode 100644 index 0000000..a67ba14 --- /dev/null +++ b/public/the-bullshit-web-developments-pov.html @@ -0,0 +1,27 @@ +The Bullshit Web - Development's Point of View

The Bullshit Web - Development's Point of View

Published on by Mitja Felicijan

Table of contents

  1. Initial thoughts
  2. Front-end frameworks
  3. Obsolescence to the rescue
  4. Unnecessary complexity
  5. Speed of development trumps code quality
  6. Load times of most popular websites

Initial thoughts

I have recently read an amazing essay by Nick Heer on the web called The Bullshit Web and it got me thinking about the future of the web as it is today.

The average internet connection in the United States is about six times as fast as it was just ten years ago, but instead of making it faster to browse the same types of websites, we’re simply occupying that extra bandwidth with more stuff.

– Nick Heer

I really try to stray away from frond-end development as much as possible. The reason is nowhere close to me having any bad opinions but having to work with clients on visual stuff drains me to the point of sheer horror.

I have observed silently the progress that was made in this field because I thought things will get better with time. I was so wrong. So wrong. Not only that things got extremely complicated to work with, the whole stack became so massive even simple pages have insanely large footprint.

The Bullshit Web essay concentrates mostly on page sizes and AMP but I would like to address tooling and technologies for development in this post.

Currently we have two types of websites:

  • informational websites,
  • web applications.

The problem that occurs is that more and more websites are treathed as web application where simple web page would suffice. And this in my opinion adds insult to the injury.

We talk about progressive web applications, AMP, and other technologies that are solving the problems of bandwidth, usability and in general making web faster but in reality this rarely gets applied in real life scenarios. Most of the time this are just demos on conferences.

Front-end frameworks

I am not of those purists that denies usage of JavaScript frameworks or SASS but there are limits to where this obsession should go. In order to use these technologies properly one should ask himself where exactly they are needed and not use them like hammer for nails.

Whenever I need to do front-end UI I usually check specification before embarking on journey of coding. And most of the times I really don’t need frameworks. Most of the code I need to write in JavaScript is done in couple of hundred lines of code and does exactly what specification requires. And developer that will be working on this code after me doesn’t need to learn new framework, tooling, etc. Just pure vanilla JavaScript. In all of my years as a developer I can count on fingers on my one hand when I used some sort of a framework. And even in this exceptions we later rewrote code to vanilla JavaScript because maintaining complex code was just to time consuming.

There is an argument to be made for using frameworks in cases where multiple people are working a project and code must be easily transferable and on-boarding process must be swift. But in reality this is just another bullshit excuse to stick with what is “cool”. I stand by Function over Form. And this also conflicts with the notion that frameworks never change. Frameworks evolve and adapt to market needs and most of the times get massive and hard to maintain. And we get stuck with massive codebase that is developed with many hacks and workarounds, because framework didn’t support some feature at the time of development. I personally hate workarounds and being a smart-ass that intentionally makes code harder to read. I find frameworks similar to the story about Cain and Abel. Either you get murdered or framework gets. Most of the times framework dies and leaves legacy nobody would want.

Huge strives have been made to address this problem and many fantastic frameworks emerged and some of theme are absolutely amazing. But there needs to be a strong case for using them in a project. We should never blindly use them regardless of the problem we are trying to solve.

I must admit that tooling around front-end is getting better and better and we are slowly getting there but there still is a long road ahead.

Obsolescence to the rescue

We can all agree that frameworks or libraries usually are there to fill the gap what currently is widely supported by the standard. Most of this so called frameworks are just libraries that unifies browser compatibility. The prime example of this is jQuery. There was a time almost everybody was using jQuery. But through time HTML5 specs were updated to include ideas from jQuery and this filled the browser compatibility gap. There is this awesome article The Rise and Fall of jQuery.

Don’t get me wrong. Yes, I dislike jQuery but I find it indispensable and without it our web would be very different. For the worst in my opinion. It was a huge stepping stone for front-end development. But there comes a time where technologies get obsolete and standards catch up with the requirements of the field.

And because libraries and frameworks have short lifespan I try to stay away from them and if possible use vanilla code. There is a wonderfull article about The Brutal Lifecycle of JavaScript Frameworks that explains how quick they popup and become obsolete.

JavaScript UI frameworks and libraries work in cycles. Every six months or so, a new one pops up, claiming that it has revolutionized UI development. Thousands of developers adopt it into their new projects, blog posts are written, Stack Overflow questions are asked and answered, and then a newer (and even more revolutionary) framework pops up to usurp the throne.

– Ian Allen

Unnecessary complexity

Libraries have a tendency to speed up development which is ok but there are a huge drawbacks in the future. Most of the times we work on simple projects. Not everybody is working on Facebook, Google or that kind of mamuth apps and by using libraries provided to us by these companies we introduce complexity these companies need in order to make their apps. And usually these libraries include edge case functionalities that only apply to them and by providing simpler way to use libraries very complex approaches get implemented.

Another reason for me to not use frameworks and libraries is that there usually is a team behind a project and by working on a feature by your own it takes too much time to read through the documentation and properly understand what the reasoning was behind a feature in a library. Most of the stuff (dashboarding, tables, widgets) that I work on are done much faster by pure using JS. Codebase footprint is smaller and doesn’t require other developers to learn a completly new framework.

This freameworks are heavily opinionated. No question about it. And by using them you accept their dogma. And by doing so you put yourself in a wierd position when new “disruptive” framework comes to life. If we think about it these frameworks should rather be called “approaches”.

Just to be completely honest

There are use-cases for such frameworks. And there are situations where they are indispensable. I am not saying that they don’t make sense. All I am saying that in my line of work I noticed that not every project is fit for a framework and it’s better to not use them in such cases.

An awesome talk about Learning from JavaScript Libraries by Trevor Landau.

Speed of development trumps code quality

I have found out that most of these frameworks or libraries have become very difficult to undestand in a matter of hours. In the past this was diifferent somehow. You could learn jQuery in a matter of hours and use it the next day like you were a pro. I know that it’s not fair to compare framework and library but for our case this is acceptable.

Every developer should have the knowledge and experience when selecting or not selection framework. I always stay true to Occam’s razor. And when prototyping I always use as barebone setup as I can. I see no problem with completly dumping a block of code and replacing it with something more complex if this makes sense. But there needs to be a huge reason behind this decision.

Workarounds are one of the nessesary evils perticulary when dealing with frameworks. Either because the lack of time of just plain reason that framework doesn’t support something. And this is the my main problem with them. In real life we don’t have the time to properly implement ideas behind a framework. And when shit hits the fan we butcher up the code and mix different ideas just to catch a deadline. And this is in contadiction with the whole idea of using a framework.

The impact that this has on quality and readability of code is massive. And threating this just as a symptom is probably the worst thing you can do. Through time these hacked-up code becomes legacy and additional code is molded to the code that already is in the codebase. And by doing this our code becomes more and more foregin of the initial concept.

Code quality and readability should come first regardless of frameworks and libraries. Code should be as close to bare-metal as possible so when frameworks change our code is still usable and can be refreshed by any developer with the basic knowledge of desired programming language.

All this directly impacts performanse. Terabytes of bandwidth wasted because there was a decision made early in the development cycle. Laggy performance, slow loading, bad experience just because development team was not cautious enough.

\ No newline at end of file diff --git a/public/using-digitalocean-spaces-object-storage-with-fuse.html b/public/using-digitalocean-spaces-object-storage-with-fuse.html new file mode 100644 index 0000000..18fb543 --- /dev/null +++ b/public/using-digitalocean-spaces-object-storage-with-fuse.html @@ -0,0 +1,190 @@ +Using DigitalOcean Spaces Object Storage with FUSE

Using DigitalOcean Spaces Object Storage with FUSE

Published on by Mitja Felicijan

Table of contents

  1. Is it possible to use them as a mounted drive with FUSE?
  2. Will the performance degrade over time and over different sizes of objects?
    1. Measurement experiment 1: File copy
    2. Measurement experiment 2: SQLite performanse
  3. Can storage be mounted on multiple machines at the same time and be writable?
  4. Observations and conslusion

Couple of months ago DigitalOcean introduced new product called Spaces which is Object Storage very similar to Amazon’s S3. This really peaked my interest, because this was something I was missing and even the thought of going over the internet for such functionality was in no interest to me. Also in fashion with their previous pricing this also is very cheap and pricing page is a no-brainer compared to AWS or GCE. Prices are clearly and precisely defined and outlined. You must love them for that :)

Initial requirements

  • Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)
  • Will the performance degrade over time and over different sizes of objects? (tl;dr NO&YES)
  • Can storage be mounted on multiple machines at the same time and be writable? (tl;dr YES)

Let me be clear. This scripts I use are made just for benchmarking and are not intended to be used in real-life situations. Besides that, I am looking into using this approaches but adding caching service in front of it and then dumping everything as an object to storage. This could potentially be some interesting post of itself. But in case you would need real-time data without eventual consistency please take this scripts as they are: not usable in such situations.

Is it possible to use them as a mounted drive with FUSE?

Well, actually they can be used in such manor. Because they are similar to AWS S3 many tools are available and you can find many articles and Stackoverflow items.

To make this work you will need DigitalOcean account. If you don’t have one you will not be able to test this code. But if you have an account then you go and create new Droplet. If you click on this link you will already have preselected Debian 9 with smallest VM option.

  • Please be sure to add you SSH key, because we will login to this machine remotely.
  • If you change your region please remember which one you choose because we will need this information when we try to mount space to our machine.

Instuctions on how to use SSH keys and how to setup them are available in article How To Use SSH Keys with DigitalOcean Droplets.

DigitalOcean Droplets

After we created Droplet it’s time to create new Space. This is done by clicking on a button Create (right top corner) and selecting Spaces. Choose pronounceable Unique name because we will use it in examples below. You can either choose Private or Public, it doesn’t matter in our case. And you can always change that in the future.

When you have created new Space we should generate Access key. This link will guide to the page when you can generate this key. After you create new one, please save provided Key and Secret because Secret will not be shown again.

DigitalOcean Spaces

Now that we have new Space and Access key we should SSH into our machine.

# replace IP with the ip of your newly created droplet
+ssh root@IP
+
+# this will install utilities for mounting storage objects as FUSE
+apt install s3fs
+
+# we now need to provide credentials (access key we created earlier)
+# replace KEY and SECRET with your own credentials but leave the colon between them
+# we also need to set proper permissions
+echo "KEY:SECRET" > .passwd-s3fs
+chmod 600 .passwd-s3fs
+
+# now we mount space to our machine
+# replace UNIQUE-NAME with the name you choose earlier
+# if you choose different region for your space be careful about -ourl option (ams3)
+s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp
+
+# now we try to create a file
+# once you mount it may take a couple of seconds to retrieve data
+echo "Hello cruel world" > /mnt/hello.txt
+

After all this you can return to your browser and go to DigitalOcean Spaces and click on your created space. If file hello.txt is present you have successfully mounted space to your machine and wrote data to it.

I choose the same region for my Droplet and my Space but you don’t have to. You can have different regions. What this actually does to performance I don’t know.

Additional information on FUSE:

Will the performance degrade over time and over different sizes of objects?

For this task I didn’t want to just read and write text files or uploading images. I actually wanted to figure out if using something like SQlite is viable in this case.

Measurement experiment 1: File copy

# first we create some dummy files at different sizes
+dd if=/dev/zero of=10KB.dat bs=1024 count=10 #10KB
+dd if=/dev/zero of=100KB.dat bs=1024 count=100 #100KB
+dd if=/dev/zero of=1MB.dat bs=1024 count=1024 #1MB
+dd if=/dev/zero of=10MB.dat bs=1024 count=10240 #10MB
+
+# now we set time command to only return real
+TIMEFORMAT=%R
+
+# now lets test it
+(time cp 10KB.dat /mnt/) |& tee -a 10KB.results.txt
+
+# and now we automate
+# this will perform the same operation 100 times
+# this will output results into separated files based on objecty size
+n=0; while (( n++ < 100 )); do (time cp 10KB.dat /mnt/10KB.$n.dat) |& tee -a 10KB.results.txt; done
+n=0; while (( n++ < 100 )); do (time cp 100KB.dat /mnt/100KB.$n.dat) |& tee -a 100KB.results.txt; done
+n=0; while (( n++ < 100 )); do (time cp 1MB.dat /mnt/1MB.$n.dat) |& tee -a 1MB.results.txt; done
+n=0; while (( n++ < 100 )); do (time cp 10MB.dat /mnt/10MB.$n.dat) |& tee -a 10MB.results.txt; done
+

Files of size 100MB were not successfully transferred and ended up displaying error (cp: failed to close ‘/mnt/100MB.1.dat’: Operation not permitted).

As I suspected, object size is not really that important. Sadly I don’t have the time to test performance over periods of time. But if some of you would do it please send me your data. I would be interested in seeing results.

Here are plotted results

You can download raw result here. Measurements are in seconds.

As far as these tests show, performance is quite stable and can be predicted which is fantastic. But this is a small test and spans only over couple of hours. So you should not completely trust them.

Measurement experiment 2: SQLite performanse

I was unable to use database file directly from mounted drive so this is a no-go as I suspected. So I executed code below on a local disk just to get some benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT for 1000 times to generate statistics. As you can see performance of SQLite is quite amazing. You could then potentially just copy file to mounted drive and be done with it.

import time
+import sqlite3
+import sys
+
+if len(sys.argv) < 3:
+  print("usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT")
+  exit()
+
+def data_iter(x):
+  for i in range(x):
+    yield "m" + str(i), "f" + str(i*i)
+
+header_line = "%s\t%s\t%s\t%s\t%s\n" % ("DROPTABLE", "CREATETABLE", "INSERTMANY", "FETCHALL", "COMMIT")
+with open("sqlite-benchmarks.tsv", "w") as fp:
+  fp.write(header_line)
+
+start_time = time.time()
+conn = sqlite3.connect(sys.argv[1])
+c = conn.cursor()
+end_time = time.time()
+result_time = CONNECT = end_time - start_time
+print("CONNECT: %g seconds" % (result_time))
+
+start_time = time.time()
+c.execute("PRAGMA journal_mode=WAL")
+c.execute("PRAGMA temp_store=MEMORY")
+c.execute("PRAGMA synchronous=OFF")
+result_time = PRAGMA = end_time - start_time
+print("PRAGMA: %g seconds" % (result_time))
+
+for i in range(int(sys.argv[3])):
+  print("#%i" % (i))
+
+  start_time = time.time()
+  c.execute("drop table if exists test")
+  end_time = time.time()
+  result_time = DROPTABLE = end_time - start_time
+  print("DROPTABLE: %g seconds" % (result_time))
+
+  start_time = time.time()
+  c.execute("create table if not exists test(a,b)")
+  end_time = time.time()
+  result_time = CREATETABLE = end_time - start_time
+  print("CREATETABLE: %g seconds" % (result_time))
+
+  start_time = time.time()
+  c.executemany("INSERT INTO test VALUES (?, ?)", data_iter(int(sys.argv[2])))
+  end_time = time.time()
+  result_time = INSERTMANY = end_time - start_time
+  print("INSERTMANY: %g seconds" % (result_time))
+
+  start_time = time.time()
+  c.execute("select count(*) from test")
+  res = c.fetchall()
+  end_time = time.time()
+  result_time = FETCHALL = end_time - start_time
+  print("FETCHALL: %g seconds" % (result_time))
+
+  start_time = time.time()
+  conn.commit()
+  end_time = time.time()
+  result_time = COMMIT = end_time - start_time
+  print("COMMIT: %g seconds" % (result_time))
+
+  print
+  log_line = "%f\t%f\t%f\t%f\t%f\n" % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT)
+  with open("sqlite-benchmarks.tsv", "a") as fp:
+    fp.write(log_line)
+
+start_time = time.time()
+conn.close()
+end_time = time.time()
+result_time = CLOSE = end_time - start_time
+print("CLOSE: %g seconds" % (result_time))
+

You can download raw result here. And again, these results are done on a local block storage and do not represent capabilities of object storage. With my current approach and state of the test code these can not be done. I would need to make Python code much more robust and check locking etc.

Can storage be mounted on multiple machines at the same time and be writable?

Well, this one didn’t take long to test. And the answer is YES. I mounted space on both machines and measured same performance on both machines. But because file is downloaded before write and then uploaded on complete there could potentially be problems is another process is trying to access the same file.

Observations and conslusion

Using Spaces in this way makes it easier to access and manage files. But besides that you would need to write additional code to make this one play nice with you applications.

Nevertheless, this was extremely simple to setup and use and this is just another excellent product in DigitalOcean product line. I found this exercise very valuable and am thinking about implementing some sort of mechanism for SQLite, so data can be stored on Spaces and accessed by many VM’s. For a project where data doesn’t need to be accessible in real-time and can have couple of minutes old data this would be very interesting. If any of you find this proposal interesting please write in a comment box below or shoot me an email and I will keep you posted.

\ No newline at end of file diff --git a/public/what-i-ve-learned-developing-ad-server.html b/public/what-i-ve-learned-developing-ad-server.html new file mode 100644 index 0000000..fb9e87f --- /dev/null +++ b/public/what-i-ve-learned-developing-ad-server.html @@ -0,0 +1,78 @@ +What I've learned developing ad server

What I've learned developing ad server

Published on by Mitja Felicijan

Table of contents

  1. Aggregate everything
  2. Measure everything
  3. Cache control is your friend
  4. Learn NGINX
  5. Use Redis/Memcached
  6. Conclusion

For the past year and half I have been developing native advertising server that contextually matches ads and displays them in different template forms on variety of websites. This project grew from serving thousands of ads per day to millions.

The system is made from couple of core components:

  • API for serving ads,
  • Utils - cronjobs and queue management tools,
  • Dashboard UI.

Initial release was using MongoDB for full-text search but was later replaced by Elasticsearch for better CPU utilization and better search performance. This provided us with many amazing functionalities of Elasticsearch. You should check it out if you do any search related operations.

Because the premise of the server is to provide native ad experience, they are rendered on the client side via simple templating engine. This ensures that ads can be displayed number of different ways based on the visual style of the page. And this makes Javascript client library quite complex.

So now that you know basic information about the product lets get into the lessons we learned.

Aggregate everything

After beta version was released everything (impressions, clicks, etc) was written in nanosecond resolution in the database. At that time we were using PostgreSQL and database quickly grew way above 200GB in disk space. And that was problematic. Statistics took disturbingly long time to aggregate. Also using indexes on stats table in database was no help after we reached 500 million datapoints.

There is a marketing product information and there is real life experience. And the tend to be quite the opposite.

This was the reason that now everything is aggregated on daily basis and this data is then fed to Elastic in form of daily summary. With this we achieved we can now track many more dimensions such as zone, channel and platform information. And with this information we can now adapt occurrences of ads on specific places more precisely.

We have also adapted Redis as a full-time citizen in our stack. Because Redis also stores information on a local disk we have some sort of backup if server would accidentally suffer some failure.

All the real-time statistics for ad serving and redirecting is presented as counters in Redis instance and daily extracted and pushed to Elastic.

Measure everything

The thing about software is that we really don’t know how well it is performing under load until such load is presented. When testing locally everything is fine but when on production things tend to fall apart.

As a solution for this we are measuring everything we can. Function execution time (by encapsulating functions with timers), server performance (cpu, memory, disk, etc), Nginx and uWSGI performance. We sacrifice a bit of performance for the sake of this information. And we store all this information for later analysis.

Example of function execution time

{
+  "get_final_filtered_ads": {
+    "counter": 1931250,
+    "avg": 0.0066143431,
+    "elapsed": 12773.9500310003
+  },
+  "store_keywords_statistics": {
+    "counter": 1931011,
+    "avg": 0.0004605267,
+    "elapsed": 889.2821669996
+  },
+  "match_by_context": {
+    "counter": 1931011,
+    "avg": 0.0055960716,
+    "elapsed": 10806.0758889999
+  },
+  "match_by_high_performance": {
+    "counter": 262,
+    "avg": 0.0152770229,
+    "elapsed": 4.00258
+  },
+  "store_impression_stats": {
+    "counter": 1931250,
+    "avg": 0.0006189991,
+    "elapsed": 1195.4419869999
+  }
+}
+

We have also started profiling with cProfile and then visualizing with KCachegrind. This provides much more detailed look into code execution.

Cache control is your friend

Because we use Javascript library for rendering ads we rely on this script extensively and when in need we need to be able to change behavior of the script quickly.

In our case we can not simply replace javascript url in html code. It usually takes a day or two for the guys who maintain sites to change code or add ?ver=xxx attribute. And this makes rapid deployment and testing very difficult and time consuming. There is a limitation of how much you can test locally.

We are now in the process of integrating Google Tag Manager but couple of websites are developed on ASP.net platform that have some problems with tag manager. With a solution below we are certain that we are serving latest version of the script.

And it only takes one mistake and users have the script cached and in case of caching it for 1 year you probably know where the problem is.

# nginx ➜ /etc/nginx/sites-available/default
+location /static/ {
+  alias /path-to-static-content/;
+  autoindex off;
+  charset utf-8;
+  gzip on;
+  gzip_types text/plain application/javascript application/x-javascript text/javascript text/xml text/css;
+  location ~* \.(ico|gif|jpeg|jpg|png|woff|ttf|otf|svg|woff2|eot)$ {
+    expires 1y;
+    add_header Pragma public;
+    add_header Cache-Control "public";
+  }
+  location ~* \.(css|js|txt)$ {
+    expires 3600s;
+    add_header Pragma public;
+    add_header Cache-Control "public, must-revalidate";
+  }
+}
+

Also be careful when redirecting to url in your python code. We noticed that if we didn’t precisely setup cache control and expire headers in response we didn’t get the request on the server and therefore couldn’t measure clicks. So when redirecting do as follows and there will be no problems.

# python ➜ bottlepy web micro-framework
+response = bottle.HTTPResponse(status=302)
+response.set_header("Cache-Control", "no-store, no-cache, must-revalidate")
+response.set_header("Expires", "Thu, 01 Jan 1970 00:00:00 GMT")
+response.set_header("Location", url)
+return response
+

Cache control in browsers is quite aggressive and you need to be precise to avoid future problems. We learned that lesson the hard way.

Learn NGINX

When deciding on a web server we went with Nginx as a reverse proxy for our applications. We adapted micro-service oriented architecture early in the project to ensure when we scale we can easily add additional servers to our cluster. And Nginx was crucial to perform load balancing and static content delivery.

At first our config file was quite simple and later grew larger. After patching and adding new settings I sat down and learned more about the guts of Nginx. This proved to be very useful and we were able to squeeze much more out of our setup. So I advise you to take your time and read through the documentation. This saved us a lot of headache. Googling for solutions only goes so far.

Use Redis/Memcached

As explained above we are using caching basically for everything. It is the corner stone of our services. At first we were very careful about the quantity of things we stored in Redis. But we later found out that the memory footprint is very low even when storing large amount of data in it.

So we gradually increased our usage to caching whole HTML outputs of dashboard. This improved our performance in order of magnitude. And by using native TTL support this goes hand in hand with our needs.

The reason why we choose Redis over Memcached was the nature of scalability of Redis out of the box. But all this can be achieved with Memcached.

Conclusion

There are a lot more details that could have been written and every single topic in here deserves it’s own post but you probably got the idea about the problems we faced.

\ No newline at end of file diff --git a/public/what-its-like-to-be-a-software-developer.html b/public/what-its-like-to-be-a-software-developer.html new file mode 100644 index 0000000..d0b5341 --- /dev/null +++ b/public/what-its-like-to-be-a-software-developer.html @@ -0,0 +1,27 @@ +What it's like to be a software developer

What it's like to be a software developer

Published on by Mitja Felicijan

I get asked a lot what the hell I actually do. I find it funny but I guess it is my fault in most cases. I try not to be the kind of a man that is always talking about his work. I live in a small village and most of my neighbours probably have no idea what I actually do. And I am ok with that. I prefer this. But on some occasions I find it disturbing how people judge other people just because they don’t understand what they are all about. Many of them probably think I am some strange kind of a looser that is awake all the time and works from home. He probably plays games and type on a computer :) What kind of a job is that? That is no job at all! :) You work for eight hours, then you go home and drink a beer and go work in your workshop. This is what real men do!

Well, you know. It’s just the way it is. And it takes time for people to understand. Being home after many years in living elsewhere really grounded me in some cases. Coming back to the place where you grew up brings some sort of a humility back in your life. And this is ok. Nobody want’s to be Icarus anyways.

What I am meaning to say is if you are in a similar situation as me it will take time for people to start understanding you. Don’t get discouraged by this. Take it as it is. People judge what they don’t understand.

I have this saying that sleeping is for pussies and we will sleep when we die. I am 32 years old now and I haven’t slowed down regarding my work hours. I have steped up the pace. I usually work for about 16-18 hours a day every day. It doesn’t matter if it’s Monday or Saturday. Work needs to be done.

I know that there are other ways. But if you want to be good there really is no other way. There are no shortcuts. There is no easier way to get to the point where you really know what the hell you are doing. Myth about this genius programmer truly is one huge bullshit. Without putting in the hours nothing can be achieved. There is no success without dedication.

My friends and coworkers often ask me how the hell did I learn so much stuff. Where do I find the time to go through all this material. And I have a simple response for them: “When you go to sleep I begin reading and prototyping. When you go on a trip I make prototype projects just for the sake of learning. When you take your time for fucking around I read articles and books hunting that single small piece of information that will help me one day.” And often they don’t believe me. They think I am just that smart and everything is easy for me. They have this misguided belief that I just had all this knowledge implanted in me at birth. And this is not the case. I have read so much in my lifetime and most of this information was useful to me later in my life. But that didn’t stop me even though I had no immediate use of it. This probably is the main difference between me and my friends. I don’t learn because I need to but because I am piecing together this huge puzzle and I threat is like a game. This amazing game of enlightenment.

I had many burn-downs in my career. Most of them come around new years. I guess around this time things slow down a bit and right then when you relax for a minute or two things get real :). They say when you enter your retirment you should never ever park your ass on a couch. You will die there :) When my burndown happens I fall into this huge depression and I start questioning my sanity. I question my decisions. I question my progress in life. I question everything. I try to understand if all this is worth it?! And every time this happens I struggle with this kind of questions. And by the time all this is over I come to the same conclusion every single time. Yes it fucking is worth it. And through the years I have noticed that this is some sort of a reset for me. This helps me maintain my sanity in the long run :) I love it when things get tough. It gets me to the next level. This teaches me progress is life.

I don’t even count anymore how many programming languages I have learned. I even stop noticing projects. They just fly by. It’s like I am hunting this revelation that is set for me. And this drives me. This helps me every day to step up my game. Every single problem I solve I come little closer to my goal. My never reaching goal. And it’s ok with me if I never reach this goal.

The only problem I have now is time. There just ain’t enough time to learn everything day has to offer. It’s like I am on a quest to become this mini search machine :).

This obsession with learning has come to the point where I stopped watching TV and news all together. I find this as noise that clutters your mind. The whole point about news is to frighten you and put your mind into a dangerous loop where you thinks that nothing matters anyways → world is going to shit. And the truth is so far away from this. We are living in this times where all this amazing possibilities are at hand. We just need to take control of our mindset and everything starts to look possible again.

What else can say after more than 10 years in this space? What else can be said anyways? I still love what I do as much as I did 10 years ago. I love it even more. And if I would have a single suggestion for all of you is to stop worrying about immediate benefits and focus on the long run. Learn, prototype, experiment and have fun. We all get frustrated at times but that doesn’t mean we should stop. Doing this kind of work is a privilege. We are making and creating. In the most pure sense we are creators. And there really is no better way to live your life.

A life without challenge, a life without hardship, a life without purpose, seems pale and pointless. With challenge come perseverance and gumption. With hardship come resilience and resolve. With purpose come strength and understanding.

— Terry Fallis, The High Road

\ No newline at end of file -- cgit v1.2.3