aboutsummaryrefslogtreecommitdiff
path: root/public/using-digitalocean-spaces-object-storage-with-fuse.html
diff options
context:
space:
mode:
Diffstat (limited to 'public/using-digitalocean-spaces-object-storage-with-fuse.html')
-rwxr-xr-xpublic/using-digitalocean-spaces-object-storage-with-fuse.html280
1 files changed, 0 insertions, 280 deletions
diff --git a/public/using-digitalocean-spaces-object-storage-with-fuse.html b/public/using-digitalocean-spaces-object-storage-with-fuse.html
deleted file mode 100755
index 4f700e0..0000000
--- a/public/using-digitalocean-spaces-object-storage-with-fuse.html
+++ /dev/null
@@ -1,280 +0,0 @@
1<!doctype html><html lang=en-us><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><meta name=generator content="JBMAFP - github.com/mitjafelicijan/jbmafp"><link href="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL69vf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAv76+/8LBwQkAAAAAAAAAAAAAAAC+vb3/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL+9vf/Bv78JAAAAAAAAAAAAAAAAu7q6/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC7ubr/vr29CAAAAAAAAAAAy8nJAZ6foP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnqGj/6GipAoAAAAAHLjU/xcXHf/BwsL/I8XY/yPK3v8XGiD/IbjL/yPF2f8XGiD/Fxkf/yLF2f8gnK3/Fxog/62ztv8fwNf/FRcd/x271v8mz93/GRsi/xkXHf8p097/GiIp/xobIv8p0t3/KdPe/xocIv8fYmr/KNPe/xoZH/8aHCL/J87c/xy81/8VFxz/IsPZ/8zS0/8XGiD/Ir/R/yPH2/8XGiD/Fxkf/yPH2/8dd4T/GBog/yPJ3f8jyNr/uru9/xcUGv8cudb/EhITDKi5vRKlvMP/RUpOERwcHRAdOj4QHTk8EBwdHRAdNTgQHTo/EBwcHRAcHB0QSGduEKW4vf+koqQfHzg+EBqz0ewSFRv7EyMr/xq51vsTERb7ExUb+xq41fsau9j7ExUb+xiPp/sZudb7ExUb+xMVG/sZuNX/GKvI/BIUGfMdvdn/IrfL/xcaIP8n1eb/J9Dh/xkcIf8ZGR7/J8/f/xxCSv8ZGyH/J9Dg/ybQ4P8ZHCL/FSQs/yPK3/8UExj/GE1b/ybS5P8ZGB7/Ghwj/ynW5P8p2Ob/Ghwi/yWrtv8p1eH/Ghwi/xocIv8p1uT/J8XT/xkcIv8m1un/Hb7d/xUYH/8hzOr/HtHu/xcaIf8XGB//I8vi/xgxOv8XGSD/I8rg/yPK4P8XGiD/GUFL/yPP6f8SERj/Fhkh/x3A4f8AAAAAJ2f9/ydr//8mZPH/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlYu38J2v//ydo/f8AAAAAAAAAAAd8/fkFqf//Iob8sAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMY39awWr//8FfP3/AAAAAAAAAAAFm/7/SfD//wR+/f8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOB/f9B7v//BaX+/wAAAAAAAAAAQ878SAyZ/v9n1v4KAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADu9v8DDJb+/z3N/XgAAAAA3/sAAN/7AADf+wAA3/sAAAAAAAAAAAAAAAAAAN/7AAAAAAAAAAAAAAAAAAAAAAAAj/EAAI/5AACP8QAA3/sAAA==" rel=icon type=image/x-icon><title>Using DigitalOcean Spaces Object Storage with FUSE</title><meta name=description content="Couple of months ago DigitalOcean introduced newproduct calledSpaces whichis Object Storage very similar to Amazon&amp;#39;s S3."><meta name=author content="Mitja Felicijan"><link rel=alternate type=application/rss+xml title="Mitja Felicijan's posts" href=https://mitjafelicijan.com/index.xml><link rel=alternate type=application/rss+xml title="Mitja Felicijan's notes" href=https://mitjafelicijan.com/notes.xml><style>:root{--border-color:gainsboro;--border-size:2px;--link-color:blue;--bg-color:#eee}*::selection{background:var(--link-color);color:#fff}*::-moz-selection{background:var(--link-color);color:#fff}*::-webkit-selection{background:var(--link-color);color:#fff}body{padding:2.5rem;max-width:1900px;background:#fff;font-family:sans-serif;line-height:1.35rem;font-size:16px}hr{border:0;border-bottom:var(--border-size)solid var(--border-color);margin-block-start:1.5rem}a{color:var(--link-color);text-decoration:none}h1,h2,h3{line-height:initial}h1{font-size:xx-large}footer{margin-block-start:2rem}cap{text-transform:capitalize}blockquote{font-style:italic}table{max-width:100%;border:var(--border-size)solid var(--border-color);border-collapse:separate;border-spacing:0}table thead tr th{border-bottom:var(--border-size)solid var(--border-color);text-align:left}table th,table td{padding:.5em .8em}ul.list li{padding:.2em 0}ul{line-height:1.35em}pre{text-wrap:nowrap;overflow-x:auto;padding:0 1em;border:var(--border-size)solid var(--border-color)}code{padding:0 3px;font-size:14px;border:0;background:var(--bg-color)}pre code{line-height:1.3em;background:#fff}pre,code,pre *,code *{font-family:monospace}figure{margin-inline-start:0;margin-inline-end:0}figcaption{width:800px;max-width:100%;text-align:center}figcaption p{margin:.3em 0 1.5em;font-style:italic}img,video,audio{width:800px;max-width:100%;border:var(--border-size)solid var(--border-color);padding:.5em}header nav{display:flex;gap:.9rem}article iframe{margin:0!important}audio::-webkit-media-controls-enclosure{border-radius:0}@media only screen and (max-width:600px){body{padding:.5em;word-wrap:break-word}header nav{gap:.7rem}header nav .hob{display:none}a{word-wrap:break-word}img,video,audio{padding:0}}</style><header><nav class=main itemscope itemtype=http://schema.org/SiteNavigationElement role=navigation aria-label="Main navigation"><a href=/>Home</a>
2<a href=/#posts>Posts</a>
3<a href=/#notes>Notes</a>
4<a href=/#sideprojects class=hob>Side Projects</a>
5<a href=/vault.html>Vault</a>
6<a href=https://github.com/mitjafelicijan target=_blank>Code</a>
7<a href=/mitjafelicijan.pgp.pub.txt target=_blank class=hob>PGP</a>
8<a href=/curriculum-vitae.html>CV</a>
9<a href=/index.xml target=_blank class=hob>RSS</a></nav></header><main role=main><article itemtype=http://schema.org/Article><h1 itemtype=headline>Using DigitalOcean Spaces Object Storage with FUSE</h1><p><cap>post</cap>, Jan 16, 2018 on <a href=https://mitjafelicijan.com>Mitja Felicijan's blog</a><div><p>Couple of months ago <a href=https://www.digitalocean.com>DigitalOcean</a> introduced new
10product called
11<a href=https://blog.digitalocean.com/introducing-spaces-object-storage/>Spaces</a> which
12is Object Storage very similar to Amazon's S3. This really peaked my interest,
13because this was something I was missing and even the thought of going over the
14internet for such functionality was in no interest to me. Also in fashion with
15their previous pricing this also is very cheap and pricing page is a no-brainer
16compared to AWS or GCE. <a href=https://www.digitalocean.com/pricing/>Prices are clearly and precisely defined and
17outlined</a>. You must love them for that
18:)<h2 id=initial-requirements>Initial requirements</h2><ul><li>Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)<li>Will the performance degrade over time and over different sizes of objects?
19(tl;dr NO&amp;YES)<li>Can storage be mounted on multiple machines at the same time and be writable?
20(tl;dr YES)</ul><blockquote><p>Let me be clear. This scripts I use are made just for benchmarking and are not
21intended to be used in real-life situations. Besides that, I am looking into
22using this approaches but adding caching service in front of it and then
23dumping everything as an object to storage. This could potentially be some
24interesting post of itself. But in case you would need real-time data without
25eventual consistency please take this scripts as they are: not usable in such
26situations.</blockquote><h2 id=is-it-possible-to-use-them-as-a-mounted-drive-with-fuse>Is it possible to use them as a mounted drive with FUSE?</h2><p>Well, actually they can be used in such manor. Because they are similar to <a href=https://aws.amazon.com/s3/>AWS
27S3</a> many tools are available and you can find many
28articles and <a href="https://stackoverflow.com/search?q=s3+fuse">Stackoverflow items</a>.<p>To make this work you will need DigitalOcean account. If you don't have one you
29will not be able to test this code. But if you have an account then you go and
30<a href="https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&amp;region=ams3&amp;distro=debian&amp;distroImage=debian-9-x64&amp;options=private_networking,install_agent">create new
31Droplet</a>.
32If you click on this link you will already have preselected Debian 9 with
33smallest VM option.<ul><li>Please be sure to add you SSH key, because we will login to this machine
34remotely.<li>If you change your region please remember which one you choose because we will
35need this information when we try to mount space to our machine.</ul><p>Instuctions on how to use SSH keys and how to setup them are available in
36article <a href=https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets>How To Use SSH Keys with DigitalOcean
37Droplets</a>.<figure><img src=/posts/do-fuse/fuse-droplets.png alt="DigitalOcean Droplets"></figure><p>After we created Droplet it's time to create new Space. This is done by clicking
38on a button <a href=https://cloud.digitalocean.com/spaces/new>Create</a> (right top
39corner) and selecting Spaces. Choose pronounceable <code>Unique name</code> because we
40will use it in examples below. You can either choose Private or Public, it
41doesn't matter in our case. And you can always change that in the future.<p>When you have created new Space we should <a href=https://cloud.digitalocean.com/settings/api/tokens>generate Access
42key</a>. This link will guide
43to the page when you can generate this key. After you create new one, please
44save provided Key and Secret because Secret will not be shown again.<figure><img src=/posts/do-fuse/fuse-spaces.png alt="DigitalOcean Spaces"></figure><p>Now that we have new Space and Access key we should SSH into our machine.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># replace IP with the ip of your newly created droplet</span>
45</span></span><span style=display:flex><span>ssh root@IP
46</span></span><span style=display:flex><span>
47</span></span><span style=display:flex><span><span style=color:green># this will install utilities for mounting storage objects as FUSE</span>
48</span></span><span style=display:flex><span>apt install s3fs
49</span></span><span style=display:flex><span>
50</span></span><span style=display:flex><span><span style=color:green># we now need to provide credentials (access key we created earlier)</span>
51</span></span><span style=display:flex><span><span style=color:green># replace KEY and SECRET with your own credentials but leave the colon between them</span>
52</span></span><span style=display:flex><span><span style=color:green># we also need to set proper permissions</span>
53</span></span><span style=display:flex><span>echo <span style=color:#a31515>&#34;KEY:SECRET&#34;</span> &gt; .passwd-s3fs
54</span></span><span style=display:flex><span>chmod 600 .passwd-s3fs
55</span></span><span style=display:flex><span>
56</span></span><span style=display:flex><span><span style=color:green># now we mount space to our machine</span>
57</span></span><span style=display:flex><span><span style=color:green># replace UNIQUE-NAME with the name you choose earlier</span>
58</span></span><span style=display:flex><span><span style=color:green># if you choose different region for your space be careful about -ourl option (ams3)</span>
59</span></span><span style=display:flex><span>s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp
60</span></span><span style=display:flex><span>
61</span></span><span style=display:flex><span><span style=color:green># now we try to create a file</span>
62</span></span><span style=display:flex><span><span style=color:green># once you mount it may take a couple of seconds to retrieve data</span>
63</span></span><span style=display:flex><span>echo <span style=color:#a31515>&#34;Hello cruel world&#34;</span> &gt; /mnt/hello.txt
64</span></span></code></pre><p>After all this you can return to your browser and go to <a href=https://cloud.digitalocean.com/spaces>DigitalOcean
65Spaces</a> and click on your created
66space. If file hello.txt is present you have successfully mounted space to your
67machine and wrote data to it.<p>I choose the same region for my Droplet and my Space but you don't have to. You
68can have different regions. What this actually does to performance I don't know.<p>Additional information on FUSE:<ul><li><a href=https://github.com/s3fs-fuse/s3fs-fuse>Github project page for s3fs</a><li><a href=https://en.wikipedia.org/wiki/Filesystem_in_Userspace>FUSE - Filesystem in Userspace</a></ul><h2 id=will-the-performance-degrade-over-time-and-over-different-sizes-of-objects>Will the performance degrade over time and over different sizes of objects?</h2><p>For this task I didn't want to just read and write text files or uploading
69images. I actually wanted to figure out if using something like SQlite is viable
70in this case.<h3 id=measurement-experiment-1-file-copy>Measurement experiment 1: File copy</h3><pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># first we create some dummy files at different sizes</span>
71</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=10KB.dat bs=1024 count=10 <span style=color:green>#10KB</span>
72</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=100KB.dat bs=1024 count=100 <span style=color:green>#100KB</span>
73</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=1MB.dat bs=1024 count=1024 <span style=color:green>#1MB</span>
74</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=10MB.dat bs=1024 count=10240 <span style=color:green>#10MB</span>
75</span></span><span style=display:flex><span>
76</span></span><span style=display:flex><span><span style=color:green># now we set time command to only return real</span>
77</span></span><span style=display:flex><span>TIMEFORMAT=%R
78</span></span><span style=display:flex><span>
79</span></span><span style=display:flex><span><span style=color:green># now lets test it</span>
80</span></span><span style=display:flex><span>(time cp 10KB.dat /mnt/) |&amp; tee -a 10KB.results.txt
81</span></span><span style=display:flex><span>
82</span></span><span style=display:flex><span><span style=color:green># and now we automate</span>
83</span></span><span style=display:flex><span><span style=color:green># this will perform the same operation 100 times</span>
84</span></span><span style=display:flex><span><span style=color:green># this will output results into separated files based on objecty size</span>
85</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 10KB.dat /mnt/10KB.$n.dat) |&amp; tee -a 10KB.results.txt; <span style=color:#00f>done</span>
86</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 100KB.dat /mnt/100KB.$n.dat) |&amp; tee -a 100KB.results.txt; <span style=color:#00f>done</span>
87</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 1MB.dat /mnt/1MB.$n.dat) |&amp; tee -a 1MB.results.txt; <span style=color:#00f>done</span>
88</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 10MB.dat /mnt/10MB.$n.dat) |&amp; tee -a 10MB.results.txt; <span style=color:#00f>done</span>
89</span></span></code></pre><p>Files of size 100MB were not successfully transferred and ended up displaying
90error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted).<p>As I suspected, object size is not really that important. Sadly I don't have the
91time to test performance over periods of time. But if some of you would do it
92please send me your data. I would be interested in seeing results.<p><strong>Here are plotted results</strong><p>You can download <a href=/posts/do-fuse/copy-benchmarks.tsv>raw result here</a>.
93Measurements are in seconds.</p><script src=//cdn.plot.ly/plotly-latest.min.js></script><div id=copy-benchmarks></div><script>
94(function(){
95 var request = new XMLHttpRequest();
96 request.open("GET", "/posts/do-fuse/copy-benchmarks.tsv", true);
97 request.onload = function() {
98 if (request.status >= 200 && request.status < 400) {
99 var payload = request.responseText.trim();
100 var tsv = payload.split("\n");
101 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
102 var traces = [];
103 var headers = tsv[0];
104 tsv.shift();
105 Array.prototype.forEach.call(headers, function(el, idx) {
106 var x = [];
107 var y = [];
108 for (var j=0; j<tsv.length; j++) {
109 x.push(j);
110 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
111 }
112 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
113 });
114 var copy = Plotly.newPlot("copy-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 40, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } }, xaxis: { title: "fn(i)", titlefont: { size: 12 } } });
115 } else { }
116 };
117 request.onerror = function() { };
118 request.send(null);
119})();
120</script><p>As far as these tests show, performance is quite stable and can be predicted
121which is fantastic. But this is a small test and spans only over couple of
122hours. So you should not completely trust them.<h3 id=measurement-experiment-2-sqlite-performanse>Measurement experiment 2: SQLite performanse</h3><p>I was unable to use database file directly from mounted drive so this is a no-go
123as I suspected. So I executed code below on a local disk just to get some
124benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY,
125FETCHALL, COMMIT for 1000 times to generate statistics. As you can see
126performance of SQLite is quite amazing. You could then potentially just copy
127file to mounted drive and be done with it.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:#00f>import</span> time
128</span></span><span style=display:flex><span><span style=color:#00f>import</span> sqlite3
129</span></span><span style=display:flex><span><span style=color:#00f>import</span> sys
130</span></span><span style=display:flex><span>
131</span></span><span style=display:flex><span><span style=color:#00f>if</span> len(sys.argv) &lt; 3:
132</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT&#34;</span>)
133</span></span><span style=display:flex><span> exit()
134</span></span><span style=display:flex><span>
135</span></span><span style=display:flex><span><span style=color:#00f>def</span> data_iter(x):
136</span></span><span style=display:flex><span> <span style=color:#00f>for</span> i <span style=color:#00f>in</span> range(x):
137</span></span><span style=display:flex><span> <span style=color:#00f>yield</span> <span style=color:#a31515>&#34;m&#34;</span> + str(i), <span style=color:#a31515>&#34;f&#34;</span> + str(i*i)
138</span></span><span style=display:flex><span>
139</span></span><span style=display:flex><span>header_line = <span style=color:#a31515>&#34;</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\n</span><span style=color:#a31515>&#34;</span> % (<span style=color:#a31515>&#34;DROPTABLE&#34;</span>, <span style=color:#a31515>&#34;CREATETABLE&#34;</span>, <span style=color:#a31515>&#34;INSERTMANY&#34;</span>, <span style=color:#a31515>&#34;FETCHALL&#34;</span>, <span style=color:#a31515>&#34;COMMIT&#34;</span>)
140</span></span><span style=display:flex><span><span style=color:#00f>with</span> open(<span style=color:#a31515>&#34;sqlite-benchmarks.tsv&#34;</span>, <span style=color:#a31515>&#34;w&#34;</span>) <span style=color:#00f>as</span> fp:
141</span></span><span style=display:flex><span> fp.write(header_line)
142</span></span><span style=display:flex><span>
143</span></span><span style=display:flex><span>start_time = time.time()
144</span></span><span style=display:flex><span>conn = sqlite3.connect(sys.argv[1])
145</span></span><span style=display:flex><span>c = conn.cursor()
146</span></span><span style=display:flex><span>end_time = time.time()
147</span></span><span style=display:flex><span>result_time = CONNECT = end_time - start_time
148</span></span><span style=display:flex><span>print(<span style=color:#a31515>&#34;CONNECT: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
149</span></span><span style=display:flex><span>
150</span></span><span style=display:flex><span>start_time = time.time()
151</span></span><span style=display:flex><span>c.execute(<span style=color:#a31515>&#34;PRAGMA journal_mode=WAL&#34;</span>)
152</span></span><span style=display:flex><span>c.execute(<span style=color:#a31515>&#34;PRAGMA temp_store=MEMORY&#34;</span>)
153</span></span><span style=display:flex><span>c.execute(<span style=color:#a31515>&#34;PRAGMA synchronous=OFF&#34;</span>)
154</span></span><span style=display:flex><span>result_time = PRAGMA = end_time - start_time
155</span></span><span style=display:flex><span>print(<span style=color:#a31515>&#34;PRAGMA: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
156</span></span><span style=display:flex><span>
157</span></span><span style=display:flex><span><span style=color:#00f>for</span> i <span style=color:#00f>in</span> range(int(sys.argv[3])):
158</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;#</span><span style=color:#a31515>%i</span><span style=color:#a31515>&#34;</span> % (i))
159</span></span><span style=display:flex><span>
160</span></span><span style=display:flex><span> start_time = time.time()
161</span></span><span style=display:flex><span> c.execute(<span style=color:#a31515>&#34;drop table if exists test&#34;</span>)
162</span></span><span style=display:flex><span> end_time = time.time()
163</span></span><span style=display:flex><span> result_time = DROPTABLE = end_time - start_time
164</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;DROPTABLE: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
165</span></span><span style=display:flex><span>
166</span></span><span style=display:flex><span> start_time = time.time()
167</span></span><span style=display:flex><span> c.execute(<span style=color:#a31515>&#34;create table if not exists test(a,b)&#34;</span>)
168</span></span><span style=display:flex><span> end_time = time.time()
169</span></span><span style=display:flex><span> result_time = CREATETABLE = end_time - start_time
170</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;CREATETABLE: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
171</span></span><span style=display:flex><span>
172</span></span><span style=display:flex><span> start_time = time.time()
173</span></span><span style=display:flex><span> c.executemany(<span style=color:#a31515>&#34;INSERT INTO test VALUES (?, ?)&#34;</span>, data_iter(int(sys.argv[2])))
174</span></span><span style=display:flex><span> end_time = time.time()
175</span></span><span style=display:flex><span> result_time = INSERTMANY = end_time - start_time
176</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;INSERTMANY: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
177</span></span><span style=display:flex><span>
178</span></span><span style=display:flex><span> start_time = time.time()
179</span></span><span style=display:flex><span> c.execute(<span style=color:#a31515>&#34;select count(*) from test&#34;</span>)
180</span></span><span style=display:flex><span> res = c.fetchall()
181</span></span><span style=display:flex><span> end_time = time.time()
182</span></span><span style=display:flex><span> result_time = FETCHALL = end_time - start_time
183</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;FETCHALL: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
184</span></span><span style=display:flex><span>
185</span></span><span style=display:flex><span> start_time = time.time()
186</span></span><span style=display:flex><span> conn.commit()
187</span></span><span style=display:flex><span> end_time = time.time()
188</span></span><span style=display:flex><span> result_time = COMMIT = end_time - start_time
189</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;COMMIT: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
190</span></span><span style=display:flex><span>
191</span></span><span style=display:flex><span> print
192</span></span><span style=display:flex><span> log_line = <span style=color:#a31515>&#34;</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\n</span><span style=color:#a31515>&#34;</span> % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT)
193</span></span><span style=display:flex><span> <span style=color:#00f>with</span> open(<span style=color:#a31515>&#34;sqlite-benchmarks.tsv&#34;</span>, <span style=color:#a31515>&#34;a&#34;</span>) <span style=color:#00f>as</span> fp:
194</span></span><span style=display:flex><span> fp.write(log_line)
195</span></span><span style=display:flex><span>
196</span></span><span style=display:flex><span>start_time = time.time()
197</span></span><span style=display:flex><span>conn.close()
198</span></span><span style=display:flex><span>end_time = time.time()
199</span></span><span style=display:flex><span>result_time = CLOSE = end_time - start_time
200</span></span><span style=display:flex><span>print(<span style=color:#a31515>&#34;CLOSE: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
201</span></span></code></pre><p>You can download <a href=/posts/do-fuse/sqlite-benchmarks.tsv>raw result here</a>. And
202again, these results are done on a local block storage and do not represent
203capabilities of object storage. With my current approach and state of the test
204code these can not be done. I would need to make Python code much more robust
205and check locking etc.<div id=sqlite-benchmarks></div><script>
206(function(){
207 var request = new XMLHttpRequest();
208 request.open("GET", "/posts/do-fuse/sqlite-benchmarks.tsv", true);
209 request.onload = function() {
210 if (request.status >= 200 && request.status < 400) {
211 var payload = request.responseText.trim();
212 var tsv = payload.split("\n");
213 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
214 var traces = [];
215 var headers = tsv[0];
216 tsv.shift();
217 Array.prototype.forEach.call(headers, function(el, idx) {
218 var x = [];
219 var y = [];
220 for (var j=0; j<tsv.length; j++) {
221 x.push(j);
222 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
223 }
224 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
225 });
226 var sqlite = Plotly.newPlot("sqlite-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 50, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } } });
227 } else { }
228 };
229 request.onerror = function() { };
230 request.send(null);
231})();
232</script><h2 id=can-storage-be-mounted-on-multiple-machines-at-the-same-time-and-be-writable>Can storage be mounted on multiple machines at the same time and be writable?</h2><p>Well, this one didn't take long to test. And the answer is <strong>YES</strong>. I mounted
233space on both machines and measured same performance on both machines. But
234because file is downloaded before write and then uploaded on complete there
235could potentially be problems is another process is trying to access the same
236file.<h2 id=observations-and-conslusion>Observations and conslusion</h2><p>Using Spaces in this way makes it easier to access and manage files. But besides
237that you would need to write additional code to make this one play nice with you
238applications.<p>Nevertheless, this was extremely simple to setup and use and this is just
239another excellent product in DigitalOcean product line. I found this exercise
240very valuable and am thinking about implementing some sort of mechanism for
241SQLite, so data can be stored on Spaces and accessed by many VM's. For a project
242where data doesn't need to be accessible in real-time and can have couple of
243minutes old data this would be very interesting. If any of you find this
244proposal interesting please write in a comment box below or shoot me an email
245and I will keep you posted.</div></article></main><section><hr><h2>Posts from blogs I follow around the net</h2><ul><li><a href=https://utcc.utoronto.ca/~cks/space/blog/linux/NFSv4ServerLockClients target=_blank rel=noopener>Finding which NFSv4 client owns a lock on a Linux NFS(v4) server</a> — <a href=https://utcc.utoronto.ca/~cks/space/blog/>Chris's Wiki :: blog</a><div>A while back I wrote an entry about finding which NFS client owns
246a lock on a Linux NFS server, which turned
247out to be specific to NFS v3 (which I really should have seen coming,
248since it involved NLM and lockd). Finding the NFS v4 client that
249owns a lock is, depending on your perspective, either simpl…<li><a href=http://www.landley.net/notes-2023.html#28-10-2023 target=_blank rel=noopener>October 28, 2023</a> — <a href=http://www.landley.net/notes-2023.html>Rob Landley's Blog Thing for 2023</a><div>Oh good grief, two of my least favorite licensing people, Larry Rosen
250and Bradley Kuhn, are interacting on the OSI's license-discuss
251list where the're doing
252bad computer history and insisting that a guy Larry Rosen
253coincidentally interviewed for a book years ago is clearly the origin of
254somethin…<li><a href="http://offbeatpursuit.com:80/blog/?id=25" target=_blank rel=noopener>A fix by any other name</a> — <a href=http://offbeatpursuit.com:80/blog/>WLOG - blog</a><div>tags:
255i2c, plan9
256Another month, another file system.
257Well, if you can’t fix it in software, fix it in hardware (looking at
258you, bme680, we’re not
259done yet). The show must go on, as they say, and I would like my
260experiments to go on.
261So a “new” addition to the environmental sensor family connected to
262the h…<li><a href=https://mirzapandzo.com/next-image-url-parameter-is-valid-but-upstream-response-is-invalid target=_blank rel=noopener>Next/Image "url" parameter is valid but upstream response is invalid</a> — <a href=https://mirzapandzo.com/>Mirza Pandzo's Blog</a><div>Getting "url" parameter is valid but upstream response is invalid error with Next/Image on WSL2<li><a href=https://drewdevault.com/2023/10/13/Going-off-script.html target=_blank rel=noopener>Going off-script</a> — <a href=https://drewdevault.com>Drew DeVault's blog</a><div>There is a phenomenon in society which I find quite bizarre. Upon our entry to
263this mortal coil, we are endowed with self-awareness, agency, and free will.
264Each of the 8 billion members of this human race represents a unique person, a
265unique worldview, and a unique agency. Yet, many of us have the sam…<li><a href=https://szymonkaliski.com/writing/2023-10-02-building-a-diy-pen-plotter/ target=_blank rel=noopener>Building a DIY Pen Plotter</a> — <a href=http://github.com/dylang/node-rss>Szymon Kaliski</a><div>This article documents my learnings from designing and building a DIY Pen Plotter during the summer of 2023.
266My ultimate goal is to build my…<li><a href=https://neil.computer/notes/chart-of-accounts-for-startups-and-saas-companies/ target=_blank rel=noopener>Chart of Accounts for Startups and SaaS Companies</a> — <a href=https://neil.computer/>Neil Panchal</a><div>Accounting is fundamental to starting a business. You need to have a basic understanding of accounting principles and essential bookkeeping. I had to learn it. There was no choice. For filing taxes, your CPA is going to ask you for an Income Statement (also known as P/L statement). If<li><a href=https://journal.valeriansaliou.name/deploy-a-nomad-cluster-on-alpine-linux-with-vultr/ target=_blank rel=noopener>Deploy a Nomad Cluster on Alpine Linux with Vultr</a> — <a href=https://journal.valeriansaliou.name/>Valerian Saliou</a><div>After spending countless hours trying to understand how to deploy my apps on Kubernetes for the first time to host Mirage, an AI API service that I run, I ended up making myself a promise that the next app I work on would be using a more productive & simpler<li><a href=https://jcs.org/2023/10/25/wifi_da target=_blank rel=noopener>BlueSCSI Wi-Fi Desk Accessory 1.0 Released</a> — <a href=https://jcs.org/>joshua stein</a><div>BlueSCSI Wi-Fi Desk Accessory
2671.0 has been released:
268wifi_da-1.0.sit
269(StuffIt 3 archive)
270SHA256: ccfc9d27dd5da7412d10cef73b81119a1fec3848e4d1d88ff652a07ffdc6a69aSHA1: ff124972f202ceda6d7fa4788110a67ccda6a13a
271This is the initial public release of my BlueSCSI Wi-Fi Desk Accessory for
272classic MacOS.<li><a href=https://michael.stapelberg.ch/posts/2023-10-25-my-all-flash-zfs-network-storage-build/ target=_blank rel=noopener>My 2023 all-flash ZFS NAS (Network Storage) build</a> — <a href=https://michael.stapelberg.ch/>Michael Stapelbergs Website</a><div>For over 10 years now, I run two self-built NAS (Network Storage) devices which serve media (currently via Jellyfin) and run daily backups of all my PCs and servers.
273In this article, I describe my goals, which hardware I picked for my new build (and why) and how I set it up.
274Design Goals
275I use my netw…</ul><p>Generated with <a href=https://git.sr.ht/~sircmpwn/openring target=_blank rel=noopener>openring</a>.</section><footer><hr><p><big><strong>Want to comment or have something to add?</strong></big><p>You can write me an email
276at <a href=mailto:mitja.felicijan@gmail.com>mitja.felicijan@gmail.com</a> or
277catch up with me <a href=https://telegram.me/mitjafelicijan target=_blank>on Telegram</a>.<hr><p>This website does not track you. Content is made available under the <a href=https://creativecommons.org/licenses/by/4.0/ target=_blank rel=noreferrer>CC BY 4.0 license</a> unless
278specified otherwise. Blog is also available as <a href=/index.xml target=_blank>RSS feed</a>.</footer><script>
279 window.va = window.va || function () { (window.vaq = window.vaq || []).push(arguments); };
280 </script><script defer src=/_vercel/insights/script.js></script> \ No newline at end of file
generated by cgit v1.2.3 (git 2.46.0) at 2026-05-01 00:05:04 +0000