aboutsummaryrefslogtreecommitdiff
path: root/public/using-digitalocean-spaces-object-storage-with-fuse.html
diff options
context:
space:
mode:
Diffstat (limited to 'public/using-digitalocean-spaces-object-storage-with-fuse.html')
-rwxr-xr-xpublic/using-digitalocean-spaces-object-storage-with-fuse.html247
1 files changed, 247 insertions, 0 deletions
diff --git a/public/using-digitalocean-spaces-object-storage-with-fuse.html b/public/using-digitalocean-spaces-object-storage-with-fuse.html
new file mode 100755
index 0000000..cfe235f
--- /dev/null
+++ b/public/using-digitalocean-spaces-object-storage-with-fuse.html
@@ -0,0 +1,247 @@
1<!doctype html><html lang=en-us><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><link href="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL69vf8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAv76+/8LBwQkAAAAAAAAAAAAAAAC+vb3/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAL+9vf/Bv78JAAAAAAAAAAAAAAAAu7q6/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC7ubr/vr29CAAAAAAAAAAAy8nJAZ6foP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAnqGj/6GipAoAAAAAHLjU/xcXHf/BwsL/I8XY/yPK3v8XGiD/IbjL/yPF2f8XGiD/Fxkf/yLF2f8gnK3/Fxog/62ztv8fwNf/FRcd/x271v8mz93/GRsi/xkXHf8p097/GiIp/xobIv8p0t3/KdPe/xocIv8fYmr/KNPe/xoZH/8aHCL/J87c/xy81/8VFxz/IsPZ/8zS0/8XGiD/Ir/R/yPH2/8XGiD/Fxkf/yPH2/8dd4T/GBog/yPJ3f8jyNr/uru9/xcUGv8cudb/EhITDKi5vRKlvMP/RUpOERwcHRAdOj4QHTk8EBwdHRAdNTgQHTo/EBwcHRAcHB0QSGduEKW4vf+koqQfHzg+EBqz0ewSFRv7EyMr/xq51vsTERb7ExUb+xq41fsau9j7ExUb+xiPp/sZudb7ExUb+xMVG/sZuNX/GKvI/BIUGfMdvdn/IrfL/xcaIP8n1eb/J9Dh/xkcIf8ZGR7/J8/f/xxCSv8ZGyH/J9Dg/ybQ4P8ZHCL/FSQs/yPK3/8UExj/GE1b/ybS5P8ZGB7/Ghwj/ynW5P8p2Ob/Ghwi/yWrtv8p1eH/Ghwi/xocIv8p1uT/J8XT/xkcIv8m1un/Hb7d/xUYH/8hzOr/HtHu/xcaIf8XGB//I8vi/xgxOv8XGSD/I8rg/yPK4P8XGiD/GUFL/yPP6f8SERj/Fhkh/x3A4f8AAAAAJ2f9/ydr//8mZPH/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlYu38J2v//ydo/f8AAAAAAAAAAAd8/fkFqf//Iob8sAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMY39awWr//8FfP3/AAAAAAAAAAAFm/7/SfD//wR+/f8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAOB/f9B7v//BaX+/wAAAAAAAAAAQ878SAyZ/v9n1v4KAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADu9v8DDJb+/z3N/XgAAAAA3/sAAN/7AADf+wAA3/sAAAAAAAAAAAAAAAAAAN/7AAAAAAAAAAAAAAAAAAAAAAAAj/EAAI/5AACP8QAA3/sAAA==" rel=icon type=image/x-icon><title>Using DigitalOcean Spaces Object Storage with FUSE</title><meta name=description content="Couple of months ago DigitalOcean introduced newproduct calledSpaces whichis Object Storage very similar to Amazon&amp;#39;s S3."><link rel=alternate type=application/rss+xml title="Mitja Felicijan's posts" href=https://mitjafelicijan.com/index.xml><link rel=alternate type=application/rss+xml title="Mitja Felicijan's notes" href=https://mitjafelicijan.com/notes.xml><style>body{padding:1rem;max-width:760px;background:#fff;font-family:times new roman,Times,serif;line-height:1.35rem}hr{margin-block-start:1.5rem}h1,h2,h3{line-height:initial}footer{margin-block-start:3rem}table{max-width:100%;border-collapse:separate;border-spacing:2px;border:1px solid #000;border-left:1px solid #999;border-top:1px solid #999}blockquote{font-style:italic}table thead{background:#eee}td,th{border:1px solid #000;padding:4px;border-right:1px solid #999;border-bottom:1px solid #999;text-align:left}pre{text-wrap:nowrap;overflow-x:auto;margin-block-start:1.5rem;margin-block-end:1.5rem;padding:.5rem 0;border-top:1px solid #000;border-bottom:1px solid #000}pre code{line-height:1.3em}pre,code,pre *,code *{font-family:monospace;font-size:initial!important}img,video,audio{max-width:100%}header{display:flex;flex-direction:row;gap:3rem}nav{display:flex;gap:.75rem}.pstatus-orange{background:gold}.pstatus-green{background:#9acd32}.pstatus-red{background:#cd5c5c}@media only screen and (max-width:600px){header{flex-direction:column;gap:1rem}a{word-wrap:break-word}}</style><header><nav class=main><a href=/>Home</a>
2<a href=https://git.mitjafelicijan.com/ target=_blank>Git</a>
3<a href=https://files.mitjafelicijan.com/ target=_blank>Files</a>
4<a href=/mitjafelicijan.pgp.pub.txt target=_blank>PGP</a>
5<a href=/curriculum-vitae.html>CV</a>
6<a href=/index.xml target=_blank>RSS</a></nav></header><main><div><h1>Using DigitalOcean Spaces Object Storage with FUSE</h1><p>Jan 16, 2018<div><p>Couple of months ago <a href=https://www.digitalocean.com>DigitalOcean</a> introduced new
7product called
8<a href=https://blog.digitalocean.com/introducing-spaces-object-storage/>Spaces</a> which
9is Object Storage very similar to Amazon's S3. This really peaked my interest,
10because this was something I was missing and even the thought of going over the
11internet for such functionality was in no interest to me. Also in fashion with
12their previous pricing this also is very cheap and pricing page is a no-brainer
13compared to AWS or GCE. <a href=https://www.digitalocean.com/pricing/>Prices are clearly and precisely defined and
14outlined</a>. You must love them for that
15:)<h2 id=initial-requirements>Initial requirements</h2><ul><li>Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)<li>Will the performance degrade over time and over different sizes of objects?
16(tl;dr NO&amp;YES)<li>Can storage be mounted on multiple machines at the same time and be writable?
17(tl;dr YES)</ul><blockquote><p>Let me be clear. This scripts I use are made just for benchmarking and are not
18intended to be used in real-life situations. Besides that, I am looking into
19using this approaches but adding caching service in front of it and then
20dumping everything as an object to storage. This could potentially be some
21interesting post of itself. But in case you would need real-time data without
22eventual consistency please take this scripts as they are: not usable in such
23situations.</blockquote><h2 id=is-it-possible-to-use-them-as-a-mounted-drive-with-fuse>Is it possible to use them as a mounted drive with FUSE?</h2><p>Well, actually they can be used in such manor. Because they are similar to <a href=https://aws.amazon.com/s3/>AWS
24S3</a> many tools are available and you can find many
25articles and <a href="https://stackoverflow.com/search?q=s3+fuse">Stackoverflow items</a>.<p>To make this work you will need DigitalOcean account. If you don't have one you
26will not be able to test this code. But if you have an account then you go and
27<a href="https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&amp;region=ams3&amp;distro=debian&amp;distroImage=debian-9-x64&amp;options=private_networking,install_agent">create new
28Droplet</a>.
29If you click on this link you will already have preselected Debian 9 with
30smallest VM option.<ul><li>Please be sure to add you SSH key, because we will login to this machine
31remotely.<li>If you change your region please remember which one you choose because we will
32need this information when we try to mount space to our machine.</ul><p>Instuctions on how to use SSH keys and how to setup them are available in
33article <a href=https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets>How To Use SSH Keys with DigitalOcean
34Droplets</a>.<p><img src=/assets/do-fuse/fuse-droplets.png alt="DigitalOcean Droplets"><p>After we created Droplet it's time to create new Space. This is done by clicking
35on a button <a href=https://cloud.digitalocean.com/spaces/new>Create</a> (right top
36corner) and selecting Spaces. Choose pronounceable <code>Unique name</code> because we
37will use it in examples below. You can either choose Private or Public, it
38doesn't matter in our case. And you can always change that in the future.<p>When you have created new Space we should <a href=https://cloud.digitalocean.com/settings/api/tokens>generate Access
39key</a>. This link will guide
40to the page when you can generate this key. After you create new one, please
41save provided Key and Secret because Secret will not be shown again.<p><img src=/assets/do-fuse/fuse-spaces.png alt="DigitalOcean Spaces"><p>Now that we have new Space and Access key we should SSH into our machine.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># replace IP with the ip of your newly created droplet</span>
42</span></span><span style=display:flex><span>ssh root@IP
43</span></span><span style=display:flex><span>
44</span></span><span style=display:flex><span><span style=color:green># this will install utilities for mounting storage objects as FUSE</span>
45</span></span><span style=display:flex><span>apt install s3fs
46</span></span><span style=display:flex><span>
47</span></span><span style=display:flex><span><span style=color:green># we now need to provide credentials (access key we created earlier)</span>
48</span></span><span style=display:flex><span><span style=color:green># replace KEY and SECRET with your own credentials but leave the colon between them</span>
49</span></span><span style=display:flex><span><span style=color:green># we also need to set proper permissions</span>
50</span></span><span style=display:flex><span>echo <span style=color:#a31515>&#34;KEY:SECRET&#34;</span> &gt; .passwd-s3fs
51</span></span><span style=display:flex><span>chmod 600 .passwd-s3fs
52</span></span><span style=display:flex><span>
53</span></span><span style=display:flex><span><span style=color:green># now we mount space to our machine</span>
54</span></span><span style=display:flex><span><span style=color:green># replace UNIQUE-NAME with the name you choose earlier</span>
55</span></span><span style=display:flex><span><span style=color:green># if you choose different region for your space be careful about -ourl option (ams3)</span>
56</span></span><span style=display:flex><span>s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp
57</span></span><span style=display:flex><span>
58</span></span><span style=display:flex><span><span style=color:green># now we try to create a file</span>
59</span></span><span style=display:flex><span><span style=color:green># once you mount it may take a couple of seconds to retrieve data</span>
60</span></span><span style=display:flex><span>echo <span style=color:#a31515>&#34;Hello cruel world&#34;</span> &gt; /mnt/hello.txt
61</span></span></code></pre><p>After all this you can return to your browser and go to <a href=https://cloud.digitalocean.com/spaces>DigitalOcean
62Spaces</a> and click on your created
63space. If file hello.txt is present you have successfully mounted space to your
64machine and wrote data to it.<p>I choose the same region for my Droplet and my Space but you don't have to. You
65can have different regions. What this actually does to performance I don't know.<p>Additional information on FUSE:<ul><li><a href=https://github.com/s3fs-fuse/s3fs-fuse>Github project page for s3fs</a><li><a href=https://en.wikipedia.org/wiki/Filesystem_in_Userspace>FUSE - Filesystem in Userspace</a></ul><h2 id=will-the-performance-degrade-over-time-and-over-different-sizes-of-objects>Will the performance degrade over time and over different sizes of objects?</h2><p>For this task I didn't want to just read and write text files or uploading
66images. I actually wanted to figure out if using something like SQlite is viable
67in this case.<h3 id=measurement-experiment-1-file-copy>Measurement experiment 1: File copy</h3><pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:green># first we create some dummy files at different sizes</span>
68</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=10KB.dat bs=1024 count=10 <span style=color:green>#10KB</span>
69</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=100KB.dat bs=1024 count=100 <span style=color:green>#100KB</span>
70</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=1MB.dat bs=1024 count=1024 <span style=color:green>#1MB</span>
71</span></span><span style=display:flex><span>dd <span style=color:#00f>if</span>=/dev/zero of=10MB.dat bs=1024 count=10240 <span style=color:green>#10MB</span>
72</span></span><span style=display:flex><span>
73</span></span><span style=display:flex><span><span style=color:green># now we set time command to only return real</span>
74</span></span><span style=display:flex><span>TIMEFORMAT=%R
75</span></span><span style=display:flex><span>
76</span></span><span style=display:flex><span><span style=color:green># now lets test it</span>
77</span></span><span style=display:flex><span>(time cp 10KB.dat /mnt/) |&amp; tee -a 10KB.results.txt
78</span></span><span style=display:flex><span>
79</span></span><span style=display:flex><span><span style=color:green># and now we automate</span>
80</span></span><span style=display:flex><span><span style=color:green># this will perform the same operation 100 times</span>
81</span></span><span style=display:flex><span><span style=color:green># this will output results into separated files based on objecty size</span>
82</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 10KB.dat /mnt/10KB.$n.dat) |&amp; tee -a 10KB.results.txt; <span style=color:#00f>done</span>
83</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 100KB.dat /mnt/100KB.$n.dat) |&amp; tee -a 100KB.results.txt; <span style=color:#00f>done</span>
84</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 1MB.dat /mnt/1MB.$n.dat) |&amp; tee -a 1MB.results.txt; <span style=color:#00f>done</span>
85</span></span><span style=display:flex><span>n=0; <span style=color:#00f>while</span> (( n++ &lt; 100 )); <span style=color:#00f>do</span> (time cp 10MB.dat /mnt/10MB.$n.dat) |&amp; tee -a 10MB.results.txt; <span style=color:#00f>done</span>
86</span></span></code></pre><p>Files of size 100MB were not successfully transferred and ended up displaying
87error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted).<p>As I suspected, object size is not really that important. Sadly I don't have the
88time to test performance over periods of time. But if some of you would do it
89please send me your data. I would be interested in seeing results.<p><strong>Here are plotted results</strong><p>You can download <a href=/assets/do-fuse/copy-benchmarks.tsv>raw result here</a>.
90Measurements are in seconds.</p><script src=//cdn.plot.ly/plotly-latest.min.js></script><div id=copy-benchmarks></div><script>
91(function(){
92 var request = new XMLHttpRequest();
93 request.open("GET", "/assets/do-fuse/copy-benchmarks.tsv", true);
94 request.onload = function() {
95 if (request.status >= 200 && request.status < 400) {
96 var payload = request.responseText.trim();
97 var tsv = payload.split("\n");
98 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
99 var traces = [];
100 var headers = tsv[0];
101 tsv.shift();
102 Array.prototype.forEach.call(headers, function(el, idx) {
103 var x = [];
104 var y = [];
105 for (var j=0; j<tsv.length; j++) {
106 x.push(j);
107 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
108 }
109 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
110 });
111 var copy = Plotly.newPlot("copy-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 40, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } }, xaxis: { title: "fn(i)", titlefont: { size: 12 } } });
112 } else { }
113 };
114 request.onerror = function() { };
115 request.send(null);
116})();
117</script><p>As far as these tests show, performance is quite stable and can be predicted
118which is fantastic. But this is a small test and spans only over couple of
119hours. So you should not completely trust them.<h3 id=measurement-experiment-2-sqlite-performanse>Measurement experiment 2: SQLite performanse</h3><p>I was unable to use database file directly from mounted drive so this is a no-go
120as I suspected. So I executed code below on a local disk just to get some
121benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY,
122FETCHALL, COMMIT for 1000 times to generate statistics. As you can see
123performance of SQLite is quite amazing. You could then potentially just copy
124file to mounted drive and be done with it.<pre tabindex=0 style=background-color:#fff><code><span style=display:flex><span><span style=color:#00f>import</span> time
125</span></span><span style=display:flex><span><span style=color:#00f>import</span> sqlite3
126</span></span><span style=display:flex><span><span style=color:#00f>import</span> sys
127</span></span><span style=display:flex><span>
128</span></span><span style=display:flex><span><span style=color:#00f>if</span> len(sys.argv) &lt; 3:
129</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT&#34;</span>)
130</span></span><span style=display:flex><span> exit()
131</span></span><span style=display:flex><span>
132</span></span><span style=display:flex><span><span style=color:#00f>def</span> data_iter(x):
133</span></span><span style=display:flex><span> <span style=color:#00f>for</span> i <span style=color:#00f>in</span> range(x):
134</span></span><span style=display:flex><span> <span style=color:#00f>yield</span> <span style=color:#a31515>&#34;m&#34;</span> + str(i), <span style=color:#a31515>&#34;f&#34;</span> + str(i*i)
135</span></span><span style=display:flex><span>
136</span></span><span style=display:flex><span>header_line = <span style=color:#a31515>&#34;</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\t</span><span style=color:#a31515>%s</span><span style=color:#a31515>\n</span><span style=color:#a31515>&#34;</span> % (<span style=color:#a31515>&#34;DROPTABLE&#34;</span>, <span style=color:#a31515>&#34;CREATETABLE&#34;</span>, <span style=color:#a31515>&#34;INSERTMANY&#34;</span>, <span style=color:#a31515>&#34;FETCHALL&#34;</span>, <span style=color:#a31515>&#34;COMMIT&#34;</span>)
137</span></span><span style=display:flex><span><span style=color:#00f>with</span> open(<span style=color:#a31515>&#34;sqlite-benchmarks.tsv&#34;</span>, <span style=color:#a31515>&#34;w&#34;</span>) <span style=color:#00f>as</span> fp:
138</span></span><span style=display:flex><span> fp.write(header_line)
139</span></span><span style=display:flex><span>
140</span></span><span style=display:flex><span>start_time = time.time()
141</span></span><span style=display:flex><span>conn = sqlite3.connect(sys.argv[1])
142</span></span><span style=display:flex><span>c = conn.cursor()
143</span></span><span style=display:flex><span>end_time = time.time()
144</span></span><span style=display:flex><span>result_time = CONNECT = end_time - start_time
145</span></span><span style=display:flex><span>print(<span style=color:#a31515>&#34;CONNECT: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
146</span></span><span style=display:flex><span>
147</span></span><span style=display:flex><span>start_time = time.time()
148</span></span><span style=display:flex><span>c.execute(<span style=color:#a31515>&#34;PRAGMA journal_mode=WAL&#34;</span>)
149</span></span><span style=display:flex><span>c.execute(<span style=color:#a31515>&#34;PRAGMA temp_store=MEMORY&#34;</span>)
150</span></span><span style=display:flex><span>c.execute(<span style=color:#a31515>&#34;PRAGMA synchronous=OFF&#34;</span>)
151</span></span><span style=display:flex><span>result_time = PRAGMA = end_time - start_time
152</span></span><span style=display:flex><span>print(<span style=color:#a31515>&#34;PRAGMA: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
153</span></span><span style=display:flex><span>
154</span></span><span style=display:flex><span><span style=color:#00f>for</span> i <span style=color:#00f>in</span> range(int(sys.argv[3])):
155</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;#</span><span style=color:#a31515>%i</span><span style=color:#a31515>&#34;</span> % (i))
156</span></span><span style=display:flex><span>
157</span></span><span style=display:flex><span> start_time = time.time()
158</span></span><span style=display:flex><span> c.execute(<span style=color:#a31515>&#34;drop table if exists test&#34;</span>)
159</span></span><span style=display:flex><span> end_time = time.time()
160</span></span><span style=display:flex><span> result_time = DROPTABLE = end_time - start_time
161</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;DROPTABLE: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
162</span></span><span style=display:flex><span>
163</span></span><span style=display:flex><span> start_time = time.time()
164</span></span><span style=display:flex><span> c.execute(<span style=color:#a31515>&#34;create table if not exists test(a,b)&#34;</span>)
165</span></span><span style=display:flex><span> end_time = time.time()
166</span></span><span style=display:flex><span> result_time = CREATETABLE = end_time - start_time
167</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;CREATETABLE: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
168</span></span><span style=display:flex><span>
169</span></span><span style=display:flex><span> start_time = time.time()
170</span></span><span style=display:flex><span> c.executemany(<span style=color:#a31515>&#34;INSERT INTO test VALUES (?, ?)&#34;</span>, data_iter(int(sys.argv[2])))
171</span></span><span style=display:flex><span> end_time = time.time()
172</span></span><span style=display:flex><span> result_time = INSERTMANY = end_time - start_time
173</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;INSERTMANY: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
174</span></span><span style=display:flex><span>
175</span></span><span style=display:flex><span> start_time = time.time()
176</span></span><span style=display:flex><span> c.execute(<span style=color:#a31515>&#34;select count(*) from test&#34;</span>)
177</span></span><span style=display:flex><span> res = c.fetchall()
178</span></span><span style=display:flex><span> end_time = time.time()
179</span></span><span style=display:flex><span> result_time = FETCHALL = end_time - start_time
180</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;FETCHALL: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
181</span></span><span style=display:flex><span>
182</span></span><span style=display:flex><span> start_time = time.time()
183</span></span><span style=display:flex><span> conn.commit()
184</span></span><span style=display:flex><span> end_time = time.time()
185</span></span><span style=display:flex><span> result_time = COMMIT = end_time - start_time
186</span></span><span style=display:flex><span> print(<span style=color:#a31515>&#34;COMMIT: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
187</span></span><span style=display:flex><span>
188</span></span><span style=display:flex><span> print
189</span></span><span style=display:flex><span> log_line = <span style=color:#a31515>&#34;</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\t</span><span style=color:#a31515>%f</span><span style=color:#a31515>\n</span><span style=color:#a31515>&#34;</span> % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT)
190</span></span><span style=display:flex><span> <span style=color:#00f>with</span> open(<span style=color:#a31515>&#34;sqlite-benchmarks.tsv&#34;</span>, <span style=color:#a31515>&#34;a&#34;</span>) <span style=color:#00f>as</span> fp:
191</span></span><span style=display:flex><span> fp.write(log_line)
192</span></span><span style=display:flex><span>
193</span></span><span style=display:flex><span>start_time = time.time()
194</span></span><span style=display:flex><span>conn.close()
195</span></span><span style=display:flex><span>end_time = time.time()
196</span></span><span style=display:flex><span>result_time = CLOSE = end_time - start_time
197</span></span><span style=display:flex><span>print(<span style=color:#a31515>&#34;CLOSE: </span><span style=color:#a31515>%g</span><span style=color:#a31515> seconds&#34;</span> % (result_time))
198</span></span></code></pre><p>You can download <a href=/assets/do-fuse/sqlite-benchmarks.tsv>raw result here</a>. And
199again, these results are done on a local block storage and do not represent
200capabilities of object storage. With my current approach and state of the test
201code these can not be done. I would need to make Python code much more robust
202and check locking etc.<div id=sqlite-benchmarks></div><script>
203(function(){
204 var request = new XMLHttpRequest();
205 request.open("GET", "/assets/do-fuse/sqlite-benchmarks.tsv", true);
206 request.onload = function() {
207 if (request.status >= 200 && request.status < 400) {
208 var payload = request.responseText.trim();
209 var tsv = payload.split("\n");
210 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
211 var traces = [];
212 var headers = tsv[0];
213 tsv.shift();
214 Array.prototype.forEach.call(headers, function(el, idx) {
215 var x = [];
216 var y = [];
217 for (var j=0; j<tsv.length; j++) {
218 x.push(j);
219 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
220 }
221 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
222 });
223 var sqlite = Plotly.newPlot("sqlite-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 50, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } } });
224 } else { }
225 };
226 request.onerror = function() { };
227 request.send(null);
228})();
229</script><h2 id=can-storage-be-mounted-on-multiple-machines-at-the-same-time-and-be-writable>Can storage be mounted on multiple machines at the same time and be writable?</h2><p>Well, this one didn't take long to test. And the answer is <strong>YES</strong>. I mounted
230space on both machines and measured same performance on both machines. But
231because file is downloaded before write and then uploaded on complete there
232could potentially be problems is another process is trying to access the same
233file.<h2 id=observations-and-conslusion>Observations and conslusion</h2><p>Using Spaces in this way makes it easier to access and manage files. But besides
234that you would need to write additional code to make this one play nice with you
235applications.<p>Nevertheless, this was extremely simple to setup and use and this is just
236another excellent product in DigitalOcean product line. I found this exercise
237very valuable and am thinking about implementing some sort of mechanism for
238SQLite, so data can be stored on Spaces and accessed by many VM's. For a project
239where data doesn't need to be accessible in real-time and can have couple of
240minutes old data this would be very interesting. If any of you find this
241proposal interesting please write in a comment box below or shoot me an email
242and I will keep you posted.</div></div></main><footer><hr><div><h3>Want to comment or have something to add?</h3>You can write me an email at
243<a href=mailto:m@mitjafelicijan.com>m@mitjafelicijan.com</a> or catch up
244with me
245<a href=https://telegram.me/mitjafelicijan target=_blank>on Telegram</a>.</div><hr><p>This website does not track you. Content is made available under
246the <a href=https://creativecommons.org/licenses/by/4.0/ target=_blank rel=noreferrer>CC BY 4.0 license</a> unless specified
247otherwise. Blog feed is available as <a href=/index.xml target=_blank>RSS feed</a>.</footer><script src=https://cdn.usefathom.com/script.js data-site=XHQARKXP defer></script> \ No newline at end of file
generated by cgit v1.2.3 (git 2.46.0) at 2026-05-01 00:34:36 +0000