diff options
| author | Mitja Felicijan <mitja.felicijan@gmail.com> | 2023-06-27 14:50:20 +0200 |
|---|---|---|
| committer | Mitja Felicijan <mitja.felicijan@gmail.com> | 2023-06-27 14:50:20 +0200 |
| commit | 8697555125c57ae64a0c9b78514b4aac4fd523de (patch) | |
| tree | a699df53a7c35a4425f30bca86982c4341f6de40 /content/posts/2020-09-09-digitalocean-sync.md | |
| parent | 33b2615a5038bc85036081e8b5e0da8584d88097 (diff) | |
| download | mitjafelicijan.com-8697555125c57ae64a0c9b78514b4aac4fd523de.tar.gz | |
Massive formatting and added figcaption
Diffstat (limited to 'content/posts/2020-09-09-digitalocean-sync.md')
| -rw-r--r-- | content/posts/2020-09-09-digitalocean-sync.md | 83 |
1 files changed, 42 insertions, 41 deletions
diff --git a/content/posts/2020-09-09-digitalocean-sync.md b/content/posts/2020-09-09-digitalocean-sync.md index 2eeea2e..aa3cce4 100644 --- a/content/posts/2020-09-09-digitalocean-sync.md +++ b/content/posts/2020-09-09-digitalocean-sync.md | |||
| @@ -5,25 +5,25 @@ date: 2020-09-09T12:00:00+02:00 | |||
| 5 | draft: false | 5 | draft: false |
| 6 | --- | 6 | --- |
| 7 | 7 | ||
| 8 | I've been using [Dropbox](https://www.dropbox.com/) for probably **10+ years** | 8 | I've been using [Dropbox](https://www.dropbox.com/) for probably **10+ years** |
| 9 | now and I-ve became so used to it that it runs in the background that I don't | 9 | now and I-ve became so used to it that it runs in the background that I don't |
| 10 | even imagine a world without it. But it's not without problems. | 10 | even imagine a world without it. But it's not without problems. |
| 11 | 11 | ||
| 12 | At first I had problems with `.venv` environments for Python and the only | 12 | At first I had problems with `.venv` environments for Python and the only |
| 13 | solution for excluding synchronization for this folder was to manually exclude | 13 | solution for excluding synchronization for this folder was to manually exclude a |
| 14 | a specific folder which is not really scalable. FYI, my whole project folder is | 14 | specific folder which is not really scalable. FYI, my whole project folder is |
| 15 | synced on [Dropbox](https://www.dropbox.com/). This of course introduced a lot | 15 | synced on [Dropbox](https://www.dropbox.com/). This of course introduced a lot |
| 16 | of syncing of files and folders that are not needed or even break things on | 16 | of syncing of files and folders that are not needed or even break things on |
| 17 | other machines. In the case of **Python**, I couldn't use that on my second | 17 | other machines. In the case of **Python**, I couldn't use that on my second |
| 18 | machine. I needed to delete `.venv` folder and pip it again which synced files | 18 | machine. I needed to delete `.venv` folder and pip it again which synced files |
| 19 | again to the main machine. This was very frustrating. **Nodejs** handles this | 19 | again to the main machine. This was very frustrating. **Nodejs** handles this |
| 20 | much nicer and I can just run the scripts without deleting `node_modules` again | 20 | much nicer and I can just run the scripts without deleting `node_modules` again |
| 21 | and reinstalling. However, `node_modules` is a beast of its own. It creates so | 21 | and reinstalling. However, `node_modules` is a beast of its own. It creates so |
| 22 | many files that OS has a problem counting them when you check the folder | 22 | many files that OS has a problem counting them when you check the folder |
| 23 | contents for size. | 23 | contents for size. |
| 24 | 24 | ||
| 25 | I wanted something similar to Dropbox. I could without the instant syncing | 25 | I wanted something similar to Dropbox. I could without the instant syncing but |
| 26 | but it would need to be fast and had the option for me to exclude folders like | 26 | it would need to be fast and had the option for me to exclude folders like |
| 27 | `node_modules, .venv, .git` and folders like that. | 27 | `node_modules, .venv, .git` and folders like that. |
| 28 | 28 | ||
| 29 | I went on a hunt for an alternative to [Dropbox](https://www.dropbox.com/) | 29 | I went on a hunt for an alternative to [Dropbox](https://www.dropbox.com/) |
| @@ -33,48 +33,51 @@ and found: | |||
| 33 | - [Sync.com](https://sync.com) | 33 | - [Sync.com](https://sync.com) |
| 34 | - [Box](https://www.box.com/) | 34 | - [Box](https://www.box.com/) |
| 35 | 35 | ||
| 36 | You know, the usual list of suspects. I didn't include [Google drive](https://drive.google.com) | 36 | You know, the usual list of suspects. I didn't include [Google |
| 37 | or [One drive](https://onedrive.live.com/) since they are even more draconian | 37 | drive](https://drive.google.com) or [One drive](https://onedrive.live.com/) |
| 38 | than Dropbox. | 38 | since they are even more draconian than Dropbox. |
| 39 | 39 | ||
| 40 | > All this does not stem from me being paranoid but recently these companies | 40 | > All this does not stem from me being paranoid but recently these companies |
| 41 | > have became more and more aggressive and they keep violating our privacy when | 41 | > have became more and more aggressive and they keep violating our privacy when |
| 42 | > they share our data with 3rd party services. It is getting out of control. | 42 | > they share our data with 3rd party services. It is getting out of control. |
| 43 | 43 | ||
| 44 | So, my main problem was still there. No way of excluding a specific folder | 44 | So, my main problem was still there. No way of excluding a specific folder from |
| 45 | from syncing. And before we go into "*But you have git, isn't that enough?*", | 45 | syncing. And before we go into "*But you have git, isn't that enough?*", I must |
| 46 | I must say, that many of the files (PDFs, spreadsheets, etc) I have in a `git` | 46 | say, that many of the files (PDFs, spreadsheets, etc) I have in a `git` repo |
| 47 | repo don't get pushed upstream to Git and I still want to have them synced | 47 | don't get pushed upstream to Git and I still want to have them synced across my |
| 48 | across my computers. | 48 | computers. |
| 49 | 49 | ||
| 50 | I initially wanted to use [rsync](https://linux.die.net/man/1/rsync) but I | 50 | I initially wanted to use [rsync](https://linux.die.net/man/1/rsync) but I would |
| 51 | would need to then have a remote VPS or transfer between my computers directly. | 51 | need to then have a remote VPS or transfer between my computers directly. I |
| 52 | I wanted a solution where all my files could be accessible to me without my | 52 | wanted a solution where all my files could be accessible to me without my |
| 53 | machine. | 53 | machine. |
| 54 | 54 | ||
| 55 | > **WARNING: This solution will cost you money!** DigitalOcean Spaces are $5 | 55 | > **WARNING: This solution will cost you money!** DigitalOcean Spaces are $5 per |
| 56 | per month and there are some bandwidth limitations and if you go beyond that | 56 | month and there are some bandwidth limitations and if you go beyond that you get |
| 57 | you get billed additionally. | 57 | billed additionally. |
| 58 | 58 | ||
| 59 | Then I remembered that I could use something like [S3](https://en.wikipedia.org/wiki/Amazon_S3) | 59 | Then I remembered that I could use something like |
| 60 | since it has versioning and is fully managed. I didn't want to go down the AWS | 60 | [S3](https://en.wikipedia.org/wiki/Amazon_S3) since it has versioning and is |
| 61 | rabbit hole with this so I choose [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/). | 61 | fully managed. I didn't want to go down the AWS rabbit hole with this so I |
| 62 | choose [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/). | ||
| 62 | 63 | ||
| 63 | Then I needed a command-line tool to sync between source and target. I found | 64 | Then I needed a command-line tool to sync between source and target. I found |
| 64 | this nice tool [s3cmd](https://s3tools.org/s3cmd) and it is in the Ubuntu | 65 | this nice tool [s3cmd](https://s3tools.org/s3cmd) and it is in the Ubuntu |
| 65 | repositories. | 66 | repositories. |
| 66 | 67 | ||
| 67 | ```bash | 68 | ```bash |
| 68 | sudo apt install s3cmd | 69 | sudo apt install s3cmd |
| 69 | ``` | 70 | ``` |
| 70 | 71 | ||
| 71 | After installation will I create a new Space bucket on DigitalOcean. Remember | 72 | After installation will I create a new Space bucket on DigitalOcean. Remember |
| 72 | the zone you will choose because you will need it when you will configure `s3cmd`. | 73 | the zone you will choose because you will need it when you will configure |
| 74 | `s3cmd`. | ||
| 73 | 75 | ||
| 74 | Then I visited [Digitalocean Applications & API](https://cloud.digitalocean.com/account/api/tokens) | 76 | Then I visited [Digitalocean Applications & |
| 75 | and generated **Spaces access keys**. Save both key and secret somewhere safe | 77 | API](https://cloud.digitalocean.com/account/api/tokens) and generated **Spaces |
| 76 | because when you will leave the page secret will not be available anymore to | 78 | access keys**. Save both key and secret somewhere safe because when you will |
| 77 | you and you will need to re-generate it. | 79 | leave the page secret will not be available anymore to you and you will need to |
| 80 | re-generate it. | ||
| 78 | 81 | ||
| 79 | ```bash | 82 | ```bash |
| 80 | # enter your key and secret and correct endpoint | 83 | # enter your key and secret and correct endpoint |
| @@ -101,10 +104,8 @@ When syncing int he other direction you will need to change the order of the | |||
| 101 | I am planning to implement some sort of a `.ignore` file that will enable me to | 104 | I am planning to implement some sort of a `.ignore` file that will enable me to |
| 102 | have a project-specific exclude options. | 105 | have a project-specific exclude options. |
| 103 | 106 | ||
| 104 | I am currently running this every hour as a cronjob which is perfectly fine | 107 | I am currently running this every hour as a cronjob which is perfectly fine for |
| 105 | for now when I am testing how this whole thing works and how it all will turn | 108 | now when I am testing how this whole thing works and how it all will turn out. |
| 106 | out. | ||
| 107 | 109 | ||
| 108 | I have also created a small Gnome extension which is still very unstable, but | 110 | I have also created a small Gnome extension which is still very unstable, but |
| 109 | when/if this whole experiment pays of I will share on Github. | 111 | when/if this whole experiment pays of I will share on Github. |
| 110 | |||
