aboutsummaryrefslogtreecommitdiff
path: root/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
diff options
context:
space:
mode:
Diffstat (limited to 'content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md')
-rw-r--r--content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md272
1 files changed, 0 insertions, 272 deletions
diff --git a/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md b/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
deleted file mode 100644
index f1c893a..0000000
--- a/content/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
+++ /dev/null
@@ -1,272 +0,0 @@
1---
2layout: post
3title: Using DigitalOcean Spaces Object Storage with FUSE
4description: Using DigitalOcean Spaces Object Storage with FUSE
5slug: using-digitalocean-spaces-object-storage-with-fuse
6type: research
7date: 2018-01-16
8---
9
10**Table of contents**
11
121. [Is it possible to use them as a mounted drive with FUSE?](#is-it-possible-to-use-them-as-a-mounted-drive-with-fuse)
132. [Will the performance degrade over time and over different sizes of objects?](#will-the-performance-degrade-over-time-and-over-different-sizes-of-objects)
14 1. [Measurement experiment 1: File copy](#measurement-experiment-1-file-copy)
15 2. [Measurement experiment 2: SQLite performanse](#measurement-experiment-2-sqlite-performanse)
163. [Can storage be mounted on multiple machines at the same time and be writable?](#can-storage-be-mounted-on-multiple-machines-at-the-same-time-and-be-writable)
174. [Observations and conslusion](#observations-and-conslusion)
18
19Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced new product called [Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) which is Object Storage very similar to Amazon's S3. This really peaked my interest, because this was something I was missing and even the thought of going over the internet for such functionality was in no interest to me. Also in fashion with their previous pricing this also is very cheap and pricing page is a no-brainer compared to AWS or GCE. [Prices are clearly and precisely defined and outlined](https://www.digitalocean.com/pricing/). You must love them for that :)
20
21### Initial requirements
22
23* Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)
24* Will the performance degrade over time and over different sizes of objects? (tl;dr NO&YES)
25* Can storage be mounted on multiple machines at the same time and be writable? (tl;dr YES)
26
27> Let me be clear. This scripts I use are made just for benchmarking and are not intended to be used in real-life situations. Besides that, I am looking into using this approaches but adding caching service in front of it and then dumping everything as an object to storage. This could potentially be some interesting post of itself. But in case you would need real-time data without eventual consistency please take this scripts as they are: not usable in such situations.
28
29## Is it possible to use them as a mounted drive with FUSE?
30
31Well, actually they can be used in such manor. Because they are similar to [AWS S3](https://aws.amazon.com/s3/) many tools are available and you can find many articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse).
32
33To make this work you will need DigitalOcean account. If you don't have one you will not be able to test this code. But if you have an account then you go and [create new Droplet](https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&region=ams3&distro=debian&distroImage=debian-9-x64&options=private_networking,install_agent). If you click on this link you will already have preselected Debian 9 with smallest VM option.
34
35* Please be sure to add you SSH key, because we will login to this machine remotely.
36* If you change your region please remember which one you choose because we will need this information when we try to mount space to our machine.
37
38Instuctions on how to use SSH keys and how to setup them are available in article [How To Use SSH Keys with DigitalOcean Droplets](https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets).
39
40![DigitalOcean Droplets](/files/fuse-droplets.png)
41
42After we created Droplet it's time to create new Space. This is done by clicking on a button [Create](https://cloud.digitalocean.com/spaces/new) (right top corner) and selecting Spaces. Choose pronounceable ```Unique name``` because we will use it in examples below. You can either choose Private or Public, it doesn't matter in our case. And you can always change that in the future.
43
44When you have created new Space we should [generate Access key](https://cloud.digitalocean.com/settings/api/tokens). This link will guide to the page when you can generate this key. After you create new one, please save provided Key and Secret because Secret will not be shown again.
45
46![DigitalOcean Spaces](/files/fuse-spaces.png)
47
48Now that we have new Space and Access key we should SSH into our machine.
49
50```bash
51# replace IP with the ip of your newly created droplet
52ssh root@IP
53
54# this will install utilities for mounting storage objects as FUSE
55apt install s3fs
56
57# we now need to provide credentials (access key we created earlier)
58# replace KEY and SECRET with your own credentials but leave the colon between them
59# we also need to set proper permissions
60echo "KEY:SECRET" > .passwd-s3fs
61chmod 600 .passwd-s3fs
62
63# now we mount space to our machine
64# replace UNIQUE-NAME with the name you choose earlier
65# if you choose different region for your space be careful about -ourl option (ams3)
66s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp
67
68# now we try to create a file
69# once you mount it may take a couple of seconds to retrieve data
70echo "Hello cruel world" > /mnt/hello.txt
71```
72
73After all this you can return to your browser and go to [DigitalOcean Spaces](https://cloud.digitalocean.com/spaces) and click on your created space. If file hello.txt is present you have successfully mounted space to your machine and wrote data to it.
74
75I choose the same region for my Droplet and my Space but you don't have to. You can have different regions. What this actually does to performance I don't know.
76
77Additional information on FUSE:
78
79* [Github project page for s3fs](https://github.com/s3fs-fuse/s3fs-fuse)
80* [FUSE - Filesystem in Userspace](https://en.wikipedia.org/wiki/Filesystem_in_Userspace)
81
82## Will the performance degrade over time and over different sizes of objects?
83
84For this task I didn't want to just read and write text files or uploading images. I actually wanted to figure out if using something like SQlite is viable in this case.
85
86### Measurement experiment 1: File copy
87
88```bash
89# first we create some dummy files at different sizes
90dd if=/dev/zero of=10KB.dat bs=1024 count=10 #10KB
91dd if=/dev/zero of=100KB.dat bs=1024 count=100 #100KB
92dd if=/dev/zero of=1MB.dat bs=1024 count=1024 #1MB
93dd if=/dev/zero of=10MB.dat bs=1024 count=10240 #10MB
94
95# now we set time command to only return real
96TIMEFORMAT=%R
97
98# now lets test it
99(time cp 10KB.dat /mnt/) |& tee -a 10KB.results.txt
100
101# and now we automate
102# this will perform the same operation 100 times
103# this will output results into separated files based on objecty size
104n=0; while (( n++ < 100 )); do (time cp 10KB.dat /mnt/10KB.$n.dat) |& tee -a 10KB.results.txt; done
105n=0; while (( n++ < 100 )); do (time cp 100KB.dat /mnt/100KB.$n.dat) |& tee -a 100KB.results.txt; done
106n=0; while (( n++ < 100 )); do (time cp 1MB.dat /mnt/1MB.$n.dat) |& tee -a 1MB.results.txt; done
107n=0; while (( n++ < 100 )); do (time cp 10MB.dat /mnt/10MB.$n.dat) |& tee -a 10MB.results.txt; done
108```
109
110Files of size 100MB were not successfully transferred and ended up displaying error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted).
111
112As I suspected, object size is not really that important. Sadly I don't have the time to test performance over periods of time. But if some of you would do it please send me your data. I would be interested in seeing results.
113
114**Here are plotted results**
115
116You can download [raw result here](/files/copy-benchmarks.tsv). Measurements are in seconds.
117
118<script src="//cdn.plot.ly/plotly-latest.min.js"></script>
119<div id="copy-benchmarks"></div>
120<script>
121(function(){
122 var request = new XMLHttpRequest();
123 request.open("GET", "/files/copy-benchmarks.tsv", true);
124 request.onload = function() {
125 if (request.status >= 200 && request.status < 400) {
126 var payload = request.responseText.trim();
127 var tsv = payload.split("\n");
128 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
129 var traces = [];
130 var headers = tsv[0];
131 tsv.shift();
132 Array.prototype.forEach.call(headers, function(el, idx) {
133 var x = [];
134 var y = [];
135 for (var j=0; j<tsv.length; j++) {
136 x.push(j);
137 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
138 }
139 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
140 });
141 var copy = Plotly.newPlot("copy-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 40, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } }, xaxis: { title: "fn(i)", titlefont: { size: 12 } } });
142 } else { }
143 };
144 request.onerror = function() { };
145 request.send(null);
146})();
147</script>
148
149As far as these tests show, performance is quite stable and can be predicted which is fantastic. But this is a small test and spans only over couple of hours. So you should not completely trust them.
150
151### Measurement experiment 2: SQLite performanse
152
153I was unable to use database file directly from mounted drive so this is a no-go as I suspected. So I executed code below on a local disk just to get some benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT for 1000 times to generate statistics. As you can see performance of SQLite is quite amazing. You could then potentially just copy file to mounted drive and be done with it.
154
155```python
156import time
157import sqlite3
158import sys
159
160if len(sys.argv) < 3:
161 print("usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT")
162 exit()
163
164def data_iter(x):
165 for i in range(x):
166 yield "m" + str(i), "f" + str(i*i)
167
168header_line = "%s\t%s\t%s\t%s\t%s\n" % ("DROPTABLE", "CREATETABLE", "INSERTMANY", "FETCHALL", "COMMIT")
169with open("sqlite-benchmarks.tsv", "w") as fp:
170 fp.write(header_line)
171
172start_time = time.time()
173conn = sqlite3.connect(sys.argv[1])
174c = conn.cursor()
175end_time = time.time()
176result_time = CONNECT = end_time - start_time
177print("CONNECT: %g seconds" % (result_time))
178
179start_time = time.time()
180c.execute("PRAGMA journal_mode=WAL")
181c.execute("PRAGMA temp_store=MEMORY")
182c.execute("PRAGMA synchronous=OFF")
183result_time = PRAGMA = end_time - start_time
184print("PRAGMA: %g seconds" % (result_time))
185
186for i in range(int(sys.argv[3])):
187 print("#%i" % (i))
188
189 start_time = time.time()
190 c.execute("drop table if exists test")
191 end_time = time.time()
192 result_time = DROPTABLE = end_time - start_time
193 print("DROPTABLE: %g seconds" % (result_time))
194
195 start_time = time.time()
196 c.execute("create table if not exists test(a,b)")
197 end_time = time.time()
198 result_time = CREATETABLE = end_time - start_time
199 print("CREATETABLE: %g seconds" % (result_time))
200
201 start_time = time.time()
202 c.executemany("INSERT INTO test VALUES (?, ?)", data_iter(int(sys.argv[2])))
203 end_time = time.time()
204 result_time = INSERTMANY = end_time - start_time
205 print("INSERTMANY: %g seconds" % (result_time))
206
207 start_time = time.time()
208 c.execute("select count(*) from test")
209 res = c.fetchall()
210 end_time = time.time()
211 result_time = FETCHALL = end_time - start_time
212 print("FETCHALL: %g seconds" % (result_time))
213
214 start_time = time.time()
215 conn.commit()
216 end_time = time.time()
217 result_time = COMMIT = end_time - start_time
218 print("COMMIT: %g seconds" % (result_time))
219
220 print
221 log_line = "%f\t%f\t%f\t%f\t%f\n" % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT)
222 with open("sqlite-benchmarks.tsv", "a") as fp:
223 fp.write(log_line)
224
225start_time = time.time()
226conn.close()
227end_time = time.time()
228result_time = CLOSE = end_time - start_time
229print("CLOSE: %g seconds" % (result_time))
230```
231
232You can download [raw result here](/files/sqlite-benchmarks.tsv). And again, these results are done on a local block storage and do not represent capabilities of object storage. With my current approach and state of the test code these can not be done. I would need to make Python code much more robust and check locking etc.
233
234<div id="sqlite-benchmarks"></div>
235<script>
236(function(){
237 var request = new XMLHttpRequest();
238 request.open("GET", "/files/sqlite-benchmarks.tsv", true);
239 request.onload = function() {
240 if (request.status >= 200 && request.status < 400) {
241 var payload = request.responseText.trim();
242 var tsv = payload.split("\n");
243 for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
244 var traces = [];
245 var headers = tsv[0];
246 tsv.shift();
247 Array.prototype.forEach.call(headers, function(el, idx) {
248 var x = [];
249 var y = [];
250 for (var j=0; j<tsv.length; j++) {
251 x.push(j);
252 y.push(parseFloat(tsv[j][idx].replace(",", ".")));
253 }
254 traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
255 });
256 var sqlite = Plotly.newPlot("sqlite-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 50, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } } });
257 } else { }
258 };
259 request.onerror = function() { };
260 request.send(null);
261})();
262</script>
263
264## Can storage be mounted on multiple machines at the same time and be writable?
265
266Well, this one didn't take long to test. And the answer is **YES**. I mounted space on both machines and measured same performance on both machines. But because file is downloaded before write and then uploaded on complete there could potentially be problems is another process is trying to access the same file.
267
268## Observations and conslusion
269
270Using Spaces in this way makes it easier to access and manage files. But besides that you would need to write additional code to make this one play nice with you applications.
271
272Nevertheless, this was extremely simple to setup and use and this is just another excellent product in DigitalOcean product line. I found this exercise very valuable and am thinking about implementing some sort of mechanism for SQLite, so data can be stored on Spaces and accessed by many VM's. For a project where data doesn't need to be accessible in real-time and can have couple of minutes old data this would be very interesting. If any of you find this proposal interesting please write in a comment box below or shoot me an email and I will keep you posted.
generated by cgit v1.2.3 (git 2.46.0) at 2026-04-30 23:45:31 +0000