aboutsummaryrefslogtreecommitdiff
path: root/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
diff options
context:
space:
mode:
authorMitja Felicijan <mitja.felicijan@gmail.com>2023-06-27 14:50:20 +0200
committerMitja Felicijan <mitja.felicijan@gmail.com>2023-06-27 14:50:20 +0200
commit8697555125c57ae64a0c9b78514b4aac4fd523de (patch)
treea699df53a7c35a4425f30bca86982c4341f6de40 /content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
parent33b2615a5038bc85036081e8b5e0da8584d88097 (diff)
downloadmitjafelicijan.com-8697555125c57ae64a0c9b78514b4aac4fd523de.tar.gz
Massive formatting and added figcaption
Diffstat (limited to 'content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md')
-rw-r--r--content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md152
1 files changed, 77 insertions, 75 deletions
diff --git a/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md b/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
index bfc77ee..3a62594 100644
--- a/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
+++ b/content/posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md
@@ -5,63 +5,67 @@ date: 2018-01-16T12:00:00+02:00
5draft: false 5draft: false
6--- 6---
7 7
8Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced 8Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced new
9new product called [Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) 9product called
10which is Object Storage very similar to Amazon's S3. This really peaked my 10[Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) which
11interest, because this was something I was missing and even the thought of 11is Object Storage very similar to Amazon's S3. This really peaked my interest,
12going over the internet for such functionality was in no interest to me. Also 12because this was something I was missing and even the thought of going over the
13in fashion with their previous pricing this also is very cheap and pricing 13internet for such functionality was in no interest to me. Also in fashion with
14page is a no-brainer compared to AWS or GCE. [Prices are clearly and precisely defined and outlined](https://www.digitalocean.com/pricing/). 14their previous pricing this also is very cheap and pricing page is a no-brainer
15You must love them for that :) 15compared to AWS or GCE. [Prices are clearly and precisely defined and
16outlined](https://www.digitalocean.com/pricing/). You must love them for that
17:)
16 18
17## Initial requirements 19## Initial requirements
18 20
19* Is it possible to use them as a mounted drive with FUSE? (tl;dr YES) 21* Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)
20* Will the performance degrade over time and over different sizes of objects? 22* Will the performance degrade over time and over different sizes of objects?
21 (tl;dr NO&YES) 23 (tl;dr NO&YES)
22* Can storage be mounted on multiple machines at the same time and be writable? 24* Can storage be mounted on multiple machines at the same time and be writable?
23 (tl;dr YES) 25 (tl;dr YES)
24 26
25> Let me be clear. This scripts I use are made just for benchmarking and are 27> Let me be clear. This scripts I use are made just for benchmarking and are not
26> not intended to be used in real-life situations. Besides that, I am looking 28> intended to be used in real-life situations. Besides that, I am looking into
27> into using this approaches but adding caching service in front of it and then 29> using this approaches but adding caching service in front of it and then
28> dumping everything as an object to storage. This could potentially be some 30> dumping everything as an object to storage. This could potentially be some
29> interesting post of itself. But in case you would need real-time data without 31> interesting post of itself. But in case you would need real-time data without
30> eventual consistency please take this scripts as they are: not usable in such 32> eventual consistency please take this scripts as they are: not usable in such
31> situations. 33> situations.
32 34
33## Is it possible to use them as a mounted drive with FUSE? 35## Is it possible to use them as a mounted drive with FUSE?
34 36
35Well, actually they can be used in such manor. Because they are similar to 37Well, actually they can be used in such manor. Because they are similar to [AWS
36[AWS S3](https://aws.amazon.com/s3/) many tools are available and you can find 38S3](https://aws.amazon.com/s3/) many tools are available and you can find many
37many articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse). 39articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse).
38 40
39To make this work you will need DigitalOcean account. If you don't have one you 41To make this work you will need DigitalOcean account. If you don't have one you
40will not be able to test this code. But if you have an account then you go and 42will not be able to test this code. But if you have an account then you go and
41[create new Droplet](https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&region=ams3&distro=debian&distroImage=debian-9-x64&options=private_networking,install_agent). 43[create new
42If you click on this link you will already have preselected Debian 9 with 44Droplet](https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&region=ams3&distro=debian&distroImage=debian-9-x64&options=private_networking,install_agent).
45If you click on this link you will already have preselected Debian 9 with
43smallest VM option. 46smallest VM option.
44 47
45* Please be sure to add you SSH key, because we will login to this machine 48* Please be sure to add you SSH key, because we will login to this machine
46 remotely. 49 remotely.
47* If you change your region please remember which one you choose because we 50* If you change your region please remember which one you choose because we will
48 will need this information when we try to mount space to our machine. 51 need this information when we try to mount space to our machine.
49 52
50Instuctions on how to use SSH keys and how to setup them are available in article 53Instuctions on how to use SSH keys and how to setup them are available in
51[How To Use SSH Keys with DigitalOcean Droplets](https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets). 54article [How To Use SSH Keys with DigitalOcean
55Droplets](https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets).
52 56
53![DigitalOcean Droplets](/assets/do-fuse/fuse-droplets.png) 57![DigitalOcean Droplets](/assets/do-fuse/fuse-droplets.png)
54 58
55After we created Droplet it's time to create new Space. This is done by clicking 59After we created Droplet it's time to create new Space. This is done by clicking
56on a button [Create](https://cloud.digitalocean.com/spaces/new) (right top 60on a button [Create](https://cloud.digitalocean.com/spaces/new) (right top
57corner) and selecting Spaces. Choose pronounceable ```Unique name``` because we 61corner) and selecting Spaces. Choose pronounceable ```Unique name``` because we
58will use it in examples below. You can either choose Private or Public, it 62will use it in examples below. You can either choose Private or Public, it
59doesn't matter in our case. And you can always change that in the future. 63doesn't matter in our case. And you can always change that in the future.
60 64
61When you have created new Space we should [generate Access key](https://cloud.digitalocean.com/settings/api/tokens). 65When you have created new Space we should [generate Access
62This link will guide to the page when you can generate this key. After you 66key](https://cloud.digitalocean.com/settings/api/tokens). This link will guide
63create new one, please save provided Key and Secret because Secret will not 67to the page when you can generate this key. After you create new one, please
64be shown again. 68save provided Key and Secret because Secret will not be shown again.
65 69
66![DigitalOcean Spaces](/assets/do-fuse/fuse-spaces.png) 70![DigitalOcean Spaces](/assets/do-fuse/fuse-spaces.png)
67 71
@@ -90,14 +94,13 @@ s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tm
90echo "Hello cruel world" > /mnt/hello.txt 94echo "Hello cruel world" > /mnt/hello.txt
91``` 95```
92 96
93After all this you can return to your browser and go to 97After all this you can return to your browser and go to [DigitalOcean
94[DigitalOcean Spaces](https://cloud.digitalocean.com/spaces) and click on your 98Spaces](https://cloud.digitalocean.com/spaces) and click on your created
95created space. If file hello.txt is present you have successfully mounted 99space. If file hello.txt is present you have successfully mounted space to your
96space to your machine and wrote data to it. 100machine and wrote data to it.
97 101
98I choose the same region for my Droplet and my Space but you don't have to. 102I choose the same region for my Droplet and my Space but you don't have to. You
99You can have different regions. What this actually does to performance I 103can have different regions. What this actually does to performance I don't know.
100don't know.
101 104
102Additional information on FUSE: 105Additional information on FUSE:
103 106
@@ -106,9 +109,9 @@ Additional information on FUSE:
106 109
107## Will the performance degrade over time and over different sizes of objects? 110## Will the performance degrade over time and over different sizes of objects?
108 111
109For this task I didn't want to just read and write text files or uploading 112For this task I didn't want to just read and write text files or uploading
110images. I actually wanted to figure out if using something like SQlite is 113images. I actually wanted to figure out if using something like SQlite is viable
111viable in this case. 114in this case.
112 115
113### Measurement experiment 1: File copy 116### Measurement experiment 1: File copy
114 117
@@ -134,12 +137,12 @@ n=0; while (( n++ < 100 )); do (time cp 1MB.dat /mnt/1MB.$n.dat) |& tee -a 1MB.r
134n=0; while (( n++ < 100 )); do (time cp 10MB.dat /mnt/10MB.$n.dat) |& tee -a 10MB.results.txt; done 137n=0; while (( n++ < 100 )); do (time cp 10MB.dat /mnt/10MB.$n.dat) |& tee -a 10MB.results.txt; done
135``` 138```
136 139
137Files of size 100MB were not successfully transferred and ended up displaying 140Files of size 100MB were not successfully transferred and ended up displaying
138error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted). 141error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted).
139 142
140As I suspected, object size is not really that important. Sadly I don't have 143As I suspected, object size is not really that important. Sadly I don't have the
141the time to test performance over periods of time. But if some of you would do 144time to test performance over periods of time. But if some of you would do it
142it please send me your data. I would be interested in seeing results. 145please send me your data. I would be interested in seeing results.
143 146
144**Here are plotted results** 147**Here are plotted results**
145 148
@@ -177,18 +180,18 @@ Measurements are in seconds.
177})(); 180})();
178</script> 181</script>
179 182
180As far as these tests show, performance is quite stable and can be predicted 183As far as these tests show, performance is quite stable and can be predicted
181which is fantastic. But this is a small test and spans only over couple of 184which is fantastic. But this is a small test and spans only over couple of
182hours. So you should not completely trust them. 185hours. So you should not completely trust them.
183 186
184### Measurement experiment 2: SQLite performanse 187### Measurement experiment 2: SQLite performanse
185 188
186I was unable to use database file directly from mounted drive so this is a 189I was unable to use database file directly from mounted drive so this is a no-go
187no-go as I suspected. So I executed code below on a local disk just to get 190as I suspected. So I executed code below on a local disk just to get some
188some benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, 191benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY,
189INSERTMANY, FETCHALL, COMMIT for 1000 times to generate statistics. As you can 192FETCHALL, COMMIT for 1000 times to generate statistics. As you can see
190see performance of SQLite is quite amazing. You could then potentially just 193performance of SQLite is quite amazing. You could then potentially just copy
191copy file to mounted drive and be done with it. 194file to mounted drive and be done with it.
192 195
193```python 196```python
194import time 197import time
@@ -305,24 +308,23 @@ and check locking etc.
305 308
306## Can storage be mounted on multiple machines at the same time and be writable? 309## Can storage be mounted on multiple machines at the same time and be writable?
307 310
308Well, this one didn't take long to test. And the answer is **YES**. I mounted 311Well, this one didn't take long to test. And the answer is **YES**. I mounted
309space on both machines and measured same performance on both machines. But 312space on both machines and measured same performance on both machines. But
310because file is downloaded before write and then uploaded on complete there 313because file is downloaded before write and then uploaded on complete there
311could potentially be problems is another process is trying to access the same 314could potentially be problems is another process is trying to access the same
312file. 315file.
313 316
314## Observations and conslusion 317## Observations and conslusion
315 318
316Using Spaces in this way makes it easier to access and manage files. But 319Using Spaces in this way makes it easier to access and manage files. But besides
317besides that you would need to write additional code to make this one play 320that you would need to write additional code to make this one play nice with you
318nice with you applications. 321applications.
319 322
320Nevertheless, this was extremely simple to setup and use and this is just 323Nevertheless, this was extremely simple to setup and use and this is just
321another excellent product in DigitalOcean product line. I found this exercise 324another excellent product in DigitalOcean product line. I found this exercise
322very valuable and am thinking about implementing some sort of mechanism for 325very valuable and am thinking about implementing some sort of mechanism for
323SQLite, so data can be stored on Spaces and accessed by many VM's. For a 326SQLite, so data can be stored on Spaces and accessed by many VM's. For a project
324project where data doesn't need to be accessible in real-time and can have 327where data doesn't need to be accessible in real-time and can have couple of
325couple of minutes old data this would be very interesting. If any of you find 328minutes old data this would be very interesting. If any of you find this
326this proposal interesting please write in a comment box below or shoot me an 329proposal interesting please write in a comment box below or shoot me an email
327email and I will keep you posted. 330and I will keep you posted.
328