diff options
Diffstat (limited to '_posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md')
| -rw-r--r-- | _posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md | 21 |
1 files changed, 15 insertions, 6 deletions
diff --git a/_posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md b/_posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md index 7442956..6b89e2b 100644 --- a/_posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md +++ b/_posts/2018-01-16-using-digitalocean-spaces-object-storage-with-fuse.md | |||
| @@ -4,17 +4,26 @@ title: Using DigitalOcean Spaces Object Storage with FUSE | |||
| 4 | description: Using DigitalOcean Spaces Object Storage with FUSE | 4 | description: Using DigitalOcean Spaces Object Storage with FUSE |
| 5 | --- | 5 | --- |
| 6 | 6 | ||
| 7 | **Table of content** | ||
| 8 | |||
| 9 | - [Is it possible to use them as a mounted drive with FUSE?](#is-it-possible-to-use-them-as-a-mounted-drive-with-fuse) | ||
| 10 | - [Will the performance degrade over time and over different sizes of objects?](#will-the-performance-degrade-over-time-and-over-different-sizes-of-objects) | ||
| 11 | - [Measurement experiment 1: File copy](#measurement-experiment-1-file-copy) | ||
| 12 | - [Measurement experiment 2: SQLite performanse](#measurement-experiment-2-sqlite-performanse) | ||
| 13 | - [Can storage be mounted on multiple machines at the same time and be writable?](#can-storage-be-mounted-on-multiple-machines-at-the-same-time-and-be-writable) | ||
| 14 | - [Observations and conslusion](#observations-and-conslusion) | ||
| 15 | |||
| 7 | Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced new product called [Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) which is Object Storage very similar to Amazon's S3. This really peaked my interest, because this was something I was missing and even the thought of going over the internet for such functionality was in no interest to me. Also in fashion with their previous pricing this also is very cheap and pricing page is a no-brainer compared to AWS or GCE. [Prices are clearly and precisely defined and outlined](https://www.digitalocean.com/pricing/). You must love them for that :) | 16 | Couple of months ago [DigitalOcean](https://www.digitalocean.com) introduced new product called [Spaces](https://blog.digitalocean.com/introducing-spaces-object-storage/) which is Object Storage very similar to Amazon's S3. This really peaked my interest, because this was something I was missing and even the thought of going over the internet for such functionality was in no interest to me. Also in fashion with their previous pricing this also is very cheap and pricing page is a no-brainer compared to AWS or GCE. [Prices are clearly and precisely defined and outlined](https://www.digitalocean.com/pricing/). You must love them for that :) |
| 8 | 17 | ||
| 9 | ### Initial requirements | 18 | ### Initial requirements |
| 10 | 19 | ||
| 11 | * [Is it possible to use them as a mounted drive with FUSE?](#fuse) (tl;dr YES) | 20 | * Is it possible to use them as a mounted drive with FUSE? (tl;dr YES) |
| 12 | * [Will the performance degrade over time and over different sizes of objects?](#performanse) (tl;dr NO&YES) | 21 | * Will the performance degrade over time and over different sizes of objects? (tl;dr NO&YES) |
| 13 | * [Can storage be mounted on multiple machines at the same time and be writable?](#multiuser) (tl;dr YES) | 22 | * Can storage be mounted on multiple machines at the same time and be writable? (tl;dr YES) |
| 14 | 23 | ||
| 15 | > Let me be clear. This scripts I use are made just for benchmarking and are not intended to be used in real-life situations. Besides that, I am looking into using this approaches but adding caching service in front of it and then dumping everything as an object to storage. This could potentially be some interesting post of itself. But in case you would need real-time data without eventual consistency please take this scripts as they are: not usable in such situations. | 24 | > Let me be clear. This scripts I use are made just for benchmarking and are not intended to be used in real-life situations. Besides that, I am looking into using this approaches but adding caching service in front of it and then dumping everything as an object to storage. This could potentially be some interesting post of itself. But in case you would need real-time data without eventual consistency please take this scripts as they are: not usable in such situations. |
| 16 | 25 | ||
| 17 | ## <a name="fuse"></a>Is it possible to use them as a mounted drive with FUSE? | 26 | ## Is it possible to use them as a mounted drive with FUSE? |
| 18 | 27 | ||
| 19 | Well, actually they can be used in such manor. Because they are similar to [AWS S3](https://aws.amazon.com/s3/) many tools are available and you can find many articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse). | 28 | Well, actually they can be used in such manor. Because they are similar to [AWS S3](https://aws.amazon.com/s3/) many tools are available and you can find many articles and [Stackoverflow items](https://stackoverflow.com/search?q=s3+fuse). |
| 20 | 29 | ||
| @@ -67,7 +76,7 @@ Additional information on FUSE: | |||
| 67 | * [Github project page for s3fs](https://github.com/s3fs-fuse/s3fs-fuse) | 76 | * [Github project page for s3fs](https://github.com/s3fs-fuse/s3fs-fuse) |
| 68 | * [FUSE - Filesystem in Userspace](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) | 77 | * [FUSE - Filesystem in Userspace](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) |
| 69 | 78 | ||
| 70 | ## <a name="performanse"></a>Will the performance degrade over time and over different sizes of objects? | 79 | ## Will the performance degrade over time and over different sizes of objects? |
| 71 | 80 | ||
| 72 | For this task I didn't want to just read and write text files or uploading images. I actually wanted to figure out if using something like SQlite is viable in this case. | 81 | For this task I didn't want to just read and write text files or uploading images. I actually wanted to figure out if using something like SQlite is viable in this case. |
| 73 | 82 | ||
| @@ -249,7 +258,7 @@ You can download [raw result here](/files/sqlite-benchmarks.tsv). And again, the | |||
| 249 | })(); | 258 | })(); |
| 250 | </script> | 259 | </script> |
| 251 | 260 | ||
| 252 | ## <a name="multiuser"></a>Can storage be mounted on multiple machines at the same time and be writable? | 261 | ## Can storage be mounted on multiple machines at the same time and be writable? |
| 253 | 262 | ||
| 254 | Well, this one didn't take long to test. And the answer is **YES**. I mounted space on both machines and measured same performance on both machines. But because file is downloaded before write and then uploaded on complete there could potentially be problems is another process is trying to access the same file. | 263 | Well, this one didn't take long to test. And the answer is **YES**. I mounted space on both machines and measured same performance on both machines. But because file is downloaded before write and then uploaded on complete there could potentially be problems is another process is trying to access the same file. |
| 255 | 264 | ||
