aboutsummaryrefslogtreecommitdiff
path: root/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
diff options
context:
space:
mode:
Diffstat (limited to 'content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md')
-rw-r--r--content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md261
1 files changed, 0 insertions, 261 deletions
diff --git a/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md b/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
deleted file mode 100644
index 93a167e..0000000
--- a/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
+++ /dev/null
@@ -1,261 +0,0 @@
1~ title: Using DigitalOcean Spaces Object Storage with FUSE
2~ description: Using DigitalOcean Spaces Object Storage with FUSE
3~ slug: /using-digitalocean-spaces-object-storage-with-fuse.html
4~ date: 2018-01-16
5~ template: post
6~ hide: false
7
8Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced new product called [Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) which is Object Storage very similar to Amazon's S3. This really peaked my interest, because this was something I was missing and even the thought of going over the internet for such functionality was in no interest to me. Also in fashion with their previous pricing this also is very cheap and pricing page is a no-brainer compared to AWS or GCE. [Prices are clearly and precisely defined and outlined](https://www.digitalocean.com/pricing/). You must love them for that :)
9
10### Initial requirements
11
12* Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)
13* Will the performance degrade over time and over different sizes of objects? (tl;dr NO&YES)
14* Can storage be mounted on multiple machines at the same time and be writable? (tl;dr YES)
15
16> Let me be clear. This scripts I use are made just for benchmarking and are not intended to be used in real-life situations. Besides that, I am looking into using this approaches but adding caching service in front of it and then dumping everything as an object to storage. This could potentially be some interesting post of itself. But in case you would need real-time data without eventual consistency please take this scripts as they are: not usable in such situations.
17
18## Is it possible to use them as a mounted drive with FUSE?
19
20Well, actually they can be used in such manor. Because they are similar to [AWS S3](https://aws.amazon.com/s3/) many tools are available and you can find many articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse).
21
22To make this work you will need DigitalOcean account. If you don't have one you will not be able to test this code. But if you have an account then you go and [create new Droplet](https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&region=ams3&distro=debian&distroImage=debian-9-x64&options=private_networking,install_agent). If you click on this link you will already have preselected Debian 9 with smallest VM option.
23
24* Please be sure to add you SSH key, because we will login to this machine remotely.
25* If you change your region please remember which one you choose because we will need this information when we try to mount space to our machine.
26
27Instuctions on how to use SSH keys and how to setup them are available in article [How To Use SSH Keys with DigitalOcean Droplets](https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets).
28
29![DigitalOcean Droplets](/assets/do-fuse/fuse-droplets.png)
30
31After we created Droplet it's time to create new Space. This is done by clicking on a button [Create](https://cloud.digitalocean.com/spaces/new) (right top corner) and selecting Spaces. Choose pronounceable ```Unique name``` because we will use it in examples below. You can either choose Private or Public, it doesn't matter in our case. And you can always change that in the future.
32
33When you have created new Space we should [generate Access key](https://cloud.digitalocean.com/settings/api/tokens). This link will guide to the page when you can generate this key. After you create new one, please save provided Key and Secret because Secret will not be shown again.
34
35![DigitalOcean Spaces](/assets/do-fuse/fuse-spaces.png)
36
37Now that we have new Space and Access key we should SSH into our machine.
38
39```bash
40# replace IP with the ip of your newly created droplet
41ssh root@IP
42
43# this will install utilities for mounting storage objects as FUSE
44apt install s3fs
45
46# we now need to provide credentials (access key we created earlier)
47# replace KEY and SECRET with your own credentials but leave the colon between them
48# we also need to set proper permissions
49echo "KEY:SECRET" > .passwd-s3fs
50chmod 600 .passwd-s3fs
51
52# now we mount space to our machine
53# replace UNIQUE-NAME with the name you choose earlier
54# if you choose different region for your space be careful about -ourl option (ams3)
55s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp
56
57# now we try to create a file
58# once you mount it may take a couple of seconds to retrieve data
59echo "Hello cruel world" > /mnt/hello.txt
60```
61
62After all this you can return to your browser and go to [DigitalOcean Spaces](https://cloud.digitalocean.com/spaces) and click on your created space. If file hello.txt is present you have successfully mounted space to your machine and wrote data to it.
63
64I choose the same region for my Droplet and my Space but you don't have to. You can have different regions. What this actually does to performance I don't know.
65
66Additional information on FUSE:
67
68* [Github project page for s3fs](https://github.com/s3fs-fuse/s3fs-fuse)
69* [FUSE - Filesystem in Userspace](https://en.wikipedia.org/wiki/Filesystem_in_Userspace)
70
71## Will the performance degrade over time and over different sizes of objects?
72
73For this task I didn't want to just read and write text files or uploading images. I actually wanted to figure out if using something like SQlite is viable in this case.
74
75### Measurement experiment 1: File copy
76
77```bash
78# first we create some dummy files at different sizes
79dd if=/dev/zero of=10KB.dat bs=1024 count=10 #10KB
80dd if=/dev/zero of=100KB.dat bs=1024 count=100 #100KB
81dd if=/dev/zero of=1MB.dat bs=1024 count=1024 #1MB
82dd if=/dev/zero of=10MB.dat bs=1024 count=10240 #10MB
83
84# now we set time command to only return real
85TIMEFORMAT=%R
86
87# now lets test it
88(time cp 10KB.dat /mnt/) |& tee -a 10KB.results.txt
89
90# and now we automate
91# this will perform the same operation 100 times
92# this will output results into separated files based on objecty size
93n=0; while (( n++ < 100 )); do (time cp 10KB.dat /mnt/10KB.$n.dat) |& tee -a 10KB.results.txt; done
94n=0; while (( n++ < 100 )); do (time cp 100KB.dat /mnt/100KB.$n.dat) |& tee -a 100KB.results.txt; done
95n=0; while (( n++ < 100 )); do (time cp 1MB.dat /mnt/1MB.$n.dat) |& tee -a 1MB.results.txt; done
96n=0; while (( n++ < 100 )); do (time cp 10MB.dat /mnt/10MB.$n.dat) |& tee -a 10MB.results.txt; done
97```
98
99Files of size 100MB were not successfully transferred and ended up displaying error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted).
100
101As I suspected, object size is not really that important. Sadly I don't have the time to test performance over periods of time. But if some of you would do it please send me your data. I would be interested in seeing results.
102
103**Here are plotted results**
104
105You can download [raw result here](/assets/do-fuse/copy-benchmarks.tsv). Measurements are in seconds.
106
107<script src="//cdn.plot.ly/plotly-latest.min.js"></script>
108<div id="copy-benchmarks"></div>
109<script>
110(function(){
111 var request = new XMLHttpRequest();
112 request.open("GET", "/assets/do-fuse/copy-benchmarks.tsv", true);
113 request.onload = function() {
114 if (request.status >= 200 && request.status < 400) {
115 var payload = request.responseText.trim();
116 var tsv = payload.split("\n");
117 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
118 var traces = [];
119 var headers = tsv[0];
120 tsv.shift();
121 Array.prototype.forEach.call(headers, function(el, idx) {
122 var x = [];
123 var y = [];
124 for (var j=0; j<tsv.length; j++) {
125 x.push(j);
126 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
127 }
128 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
129 });
130 var copy = Plotly.newPlot("copy-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 40, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } }, xaxis: { title: "fn(i)", titlefont: { size: 12 } } });
131 } else { }
132 };
133 request.onerror = function() { };
134 request.send(null);
135})();
136</script>
137
138As far as these tests show, performance is quite stable and can be predicted which is fantastic. But this is a small test and spans only over couple of hours. So you should not completely trust them.
139
140### Measurement experiment 2: SQLite performanse
141
142I was unable to use database file directly from mounted drive so this is a no-go as I suspected. So I executed code below on a local disk just to get some benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT for 1000 times to generate statistics. As you can see performance of SQLite is quite amazing. You could then potentially just copy file to mounted drive and be done with it.
143
144```python
145import time
146import sqlite3
147import sys
148
149if len(sys.argv) < 3:
150 print("usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT")
151 exit()
152
153def data_iter(x):
154 for i in range(x):
155 yield "m" + str(i), "f" + str(i*i)
156
157header_line = "%s\t%s\t%s\t%s\t%s\n" % ("DROPTABLE", "CREATETABLE", "INSERTMANY", "FETCHALL", "COMMIT")
158with open("sqlite-benchmarks.tsv", "w") as fp:
159 fp.write(header_line)
160
161start_time = time.time()
162conn = sqlite3.connect(sys.argv[1])
163c = conn.cursor()
164end_time = time.time()
165result_time = CONNECT = end_time - start_time
166print("CONNECT: %g seconds" % (result_time))
167
168start_time = time.time()
169c.execute("PRAGMA journal_mode=WAL")
170c.execute("PRAGMA temp_store=MEMORY")
171c.execute("PRAGMA synchronous=OFF")
172result_time = PRAGMA = end_time - start_time
173print("PRAGMA: %g seconds" % (result_time))
174
175for i in range(int(sys.argv[3])):
176 print("#%i" % (i))
177
178 start_time = time.time()
179 c.execute("drop table if exists test")
180 end_time = time.time()
181 result_time = DROPTABLE = end_time - start_time
182 print("DROPTABLE: %g seconds" % (result_time))
183
184 start_time = time.time()
185 c.execute("create table if not exists test(a,b)")
186 end_time = time.time()
187 result_time = CREATETABLE = end_time - start_time
188 print("CREATETABLE: %g seconds" % (result_time))
189
190 start_time = time.time()
191 c.executemany("INSERT INTO test VALUES (?, ?)", data_iter(int(sys.argv[2])))
192 end_time = time.time()
193 result_time = INSERTMANY = end_time - start_time
194 print("INSERTMANY: %g seconds" % (result_time))
195
196 start_time = time.time()
197 c.execute("select count(*) from test")
198 res = c.fetchall()
199 end_time = time.time()
200 result_time = FETCHALL = end_time - start_time
201 print("FETCHALL: %g seconds" % (result_time))
202
203 start_time = time.time()
204 conn.commit()
205 end_time = time.time()
206 result_time = COMMIT = end_time - start_time
207 print("COMMIT: %g seconds" % (result_time))
208
209 print
210 log_line = "%f\t%f\t%f\t%f\t%f\n" % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT)
211 with open("sqlite-benchmarks.tsv", "a") as fp:
212 fp.write(log_line)
213
214start_time = time.time()
215conn.close()
216end_time = time.time()
217result_time = CLOSE = end_time - start_time
218print("CLOSE: %g seconds" % (result_time))
219```
220
221You can download [raw result here](/assets/do-fuse/sqlite-benchmarks.tsv). And again, these results are done on a local block storage and do not represent capabilities of object storage. With my current approach and state of the test code these can not be done. I would need to make Python code much more robust and check locking etc.
222
223<div id="sqlite-benchmarks"></div>
224<script>
225(function(){
226 var request = new XMLHttpRequest();
227 request.open("GET", "/assets/do-fuse/sqlite-benchmarks.tsv", true);
228 request.onload = function() {
229 if (request.status >= 200 && request.status < 400) {
230 var payload = request.responseText.trim();
231 var tsv = payload.split("\n");
232 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
233 var traces = [];
234 var headers = tsv[0];
235 tsv.shift();
236 Array.prototype.forEach.call(headers, function(el, idx) {
237 var x = [];
238 var y = [];
239 for (var j=0; j<tsv.length; j++) {
240 x.push(j);
241 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
242 }
243 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
244 });
245 var sqlite = Plotly.newPlot("sqlite-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 50, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } } });
246 } else { }
247 };
248 request.onerror = function() { };
249 request.send(null);
250})();
251</script>
252
253## Can storage be mounted on multiple machines at the same time and be writable?
254
255Well, this one didn't take long to test. And the answer is **YES**. I mounted space on both machines and measured same performance on both machines. But because file is downloaded before write and then uploaded on complete there could potentially be problems is another process is trying to access the same file.
256
257## Observations and conslusion
258
259Using Spaces in this way makes it easier to access and manage files. But besides that you would need to write additional code to make this one play nice with you applications.
260
261Nevertheless, this was extremely simple to setup and use and this is just another excellent product in DigitalOcean product line. I found this exercise very valuable and am thinking about implementing some sort of mechanism for SQLite, so data can be stored on Spaces and accessed by many VM's. For a project where data doesn't need to be accessible in real-time and can have couple of minutes old data this would be very interesting. If any of you find this proposal interesting please write in a comment box below or shoot me an email and I will keep you posted.
generated by cgit v1.2.3 (git 2.46.0) at 2026-04-30 23:01:38 +0000