aboutsummaryrefslogtreecommitdiff
path: root/content/posts/2020-09-09-digitalocean-sync.md
diff options
context:
space:
mode:
authorMitja Felicijan <mitja.felicijan@gmail.com>2023-06-27 14:50:20 +0200
committerMitja Felicijan <mitja.felicijan@gmail.com>2023-06-27 14:50:20 +0200
commit8697555125c57ae64a0c9b78514b4aac4fd523de (patch)
treea699df53a7c35a4425f30bca86982c4341f6de40 /content/posts/2020-09-09-digitalocean-sync.md
parent33b2615a5038bc85036081e8b5e0da8584d88097 (diff)
downloadmitjafelicijan.com-8697555125c57ae64a0c9b78514b4aac4fd523de.tar.gz
Massive formatting and added figcaption
Diffstat (limited to 'content/posts/2020-09-09-digitalocean-sync.md')
-rw-r--r--content/posts/2020-09-09-digitalocean-sync.md83
1 files changed, 42 insertions, 41 deletions
diff --git a/content/posts/2020-09-09-digitalocean-sync.md b/content/posts/2020-09-09-digitalocean-sync.md
index 2eeea2e..aa3cce4 100644
--- a/content/posts/2020-09-09-digitalocean-sync.md
+++ b/content/posts/2020-09-09-digitalocean-sync.md
@@ -5,25 +5,25 @@ date: 2020-09-09T12:00:00+02:00
5draft: false 5draft: false
6--- 6---
7 7
8I've been using [Dropbox](https://www.dropbox.com/) for probably **10+ years** 8I've been using [Dropbox](https://www.dropbox.com/) for probably **10+ years**
9now and I-ve became so used to it that it runs in the background that I don't 9now and I-ve became so used to it that it runs in the background that I don't
10even imagine a world without it. But it's not without problems. 10even imagine a world without it. But it's not without problems.
11 11
12At first I had problems with `.venv` environments for Python and the only 12At first I had problems with `.venv` environments for Python and the only
13solution for excluding synchronization for this folder was to manually exclude 13solution for excluding synchronization for this folder was to manually exclude a
14a specific folder which is not really scalable. FYI, my whole project folder is 14specific folder which is not really scalable. FYI, my whole project folder is
15synced on [Dropbox](https://www.dropbox.com/). This of course introduced a lot 15synced on [Dropbox](https://www.dropbox.com/). This of course introduced a lot
16of syncing of files and folders that are not needed or even break things on 16of syncing of files and folders that are not needed or even break things on
17other machines. In the case of **Python**, I couldn't use that on my second 17other machines. In the case of **Python**, I couldn't use that on my second
18machine. I needed to delete `.venv` folder and pip it again which synced files 18machine. I needed to delete `.venv` folder and pip it again which synced files
19again to the main machine. This was very frustrating. **Nodejs** handles this 19again to the main machine. This was very frustrating. **Nodejs** handles this
20much nicer and I can just run the scripts without deleting `node_modules` again 20much nicer and I can just run the scripts without deleting `node_modules` again
21and reinstalling. However, `node_modules` is a beast of its own. It creates so 21and reinstalling. However, `node_modules` is a beast of its own. It creates so
22many files that OS has a problem counting them when you check the folder 22many files that OS has a problem counting them when you check the folder
23contents for size. 23contents for size.
24 24
25I wanted something similar to Dropbox. I could without the instant syncing 25I wanted something similar to Dropbox. I could without the instant syncing but
26but it would need to be fast and had the option for me to exclude folders like 26it would need to be fast and had the option for me to exclude folders like
27`node_modules, .venv, .git` and folders like that. 27`node_modules, .venv, .git` and folders like that.
28 28
29I went on a hunt for an alternative to [Dropbox](https://www.dropbox.com/) 29I went on a hunt for an alternative to [Dropbox](https://www.dropbox.com/)
@@ -33,48 +33,51 @@ and found:
33- [Sync.com](https://sync.com) 33- [Sync.com](https://sync.com)
34- [Box](https://www.box.com/) 34- [Box](https://www.box.com/)
35 35
36You know, the usual list of suspects. I didn't include [Google drive](https://drive.google.com) 36You know, the usual list of suspects. I didn't include [Google
37or [One drive](https://onedrive.live.com/) since they are even more draconian 37drive](https://drive.google.com) or [One drive](https://onedrive.live.com/)
38than Dropbox. 38since they are even more draconian than Dropbox.
39 39
40> All this does not stem from me being paranoid but recently these companies 40> All this does not stem from me being paranoid but recently these companies
41> have became more and more aggressive and they keep violating our privacy when 41> have became more and more aggressive and they keep violating our privacy when
42> they share our data with 3rd party services. It is getting out of control. 42> they share our data with 3rd party services. It is getting out of control.
43 43
44So, my main problem was still there. No way of excluding a specific folder 44So, my main problem was still there. No way of excluding a specific folder from
45from syncing. And before we go into "*But you have git, isn't that enough?*", 45syncing. And before we go into "*But you have git, isn't that enough?*", I must
46I must say, that many of the files (PDFs, spreadsheets, etc) I have in a `git` 46say, that many of the files (PDFs, spreadsheets, etc) I have in a `git` repo
47repo don't get pushed upstream to Git and I still want to have them synced 47don't get pushed upstream to Git and I still want to have them synced across my
48across my computers. 48computers.
49 49
50I initially wanted to use [rsync](https://linux.die.net/man/1/rsync) but I 50I initially wanted to use [rsync](https://linux.die.net/man/1/rsync) but I would
51would need to then have a remote VPS or transfer between my computers directly. 51need to then have a remote VPS or transfer between my computers directly. I
52I wanted a solution where all my files could be accessible to me without my 52wanted a solution where all my files could be accessible to me without my
53machine. 53machine.
54 54
55> **WARNING: This solution will cost you money!** DigitalOcean Spaces are $5 55> **WARNING: This solution will cost you money!** DigitalOcean Spaces are $5 per
56per month and there are some bandwidth limitations and if you go beyond that 56month and there are some bandwidth limitations and if you go beyond that you get
57you get billed additionally. 57billed additionally.
58 58
59Then I remembered that I could use something like [S3](https://en.wikipedia.org/wiki/Amazon_S3) 59Then I remembered that I could use something like
60since it has versioning and is fully managed. I didn't want to go down the AWS 60[S3](https://en.wikipedia.org/wiki/Amazon_S3) since it has versioning and is
61rabbit hole with this so I choose [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/). 61fully managed. I didn't want to go down the AWS rabbit hole with this so I
62choose [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/).
62 63
63Then I needed a command-line tool to sync between source and target. I found 64Then I needed a command-line tool to sync between source and target. I found
64this nice tool [s3cmd](https://s3tools.org/s3cmd) and it is in the Ubuntu 65this nice tool [s3cmd](https://s3tools.org/s3cmd) and it is in the Ubuntu
65repositories. 66repositories.
66 67
67```bash 68```bash
68sudo apt install s3cmd 69sudo apt install s3cmd
69``` 70```
70 71
71After installation will I create a new Space bucket on DigitalOcean. Remember 72After installation will I create a new Space bucket on DigitalOcean. Remember
72the zone you will choose because you will need it when you will configure `s3cmd`. 73the zone you will choose because you will need it when you will configure
74`s3cmd`.
73 75
74Then I visited [Digitalocean Applications & API](https://cloud.digitalocean.com/account/api/tokens) 76Then I visited [Digitalocean Applications &
75and generated **Spaces access keys**. Save both key and secret somewhere safe 77API](https://cloud.digitalocean.com/account/api/tokens) and generated **Spaces
76because when you will leave the page secret will not be available anymore to 78access keys**. Save both key and secret somewhere safe because when you will
77you and you will need to re-generate it. 79leave the page secret will not be available anymore to you and you will need to
80re-generate it.
78 81
79```bash 82```bash
80# enter your key and secret and correct endpoint 83# enter your key and secret and correct endpoint
@@ -101,10 +104,8 @@ When syncing int he other direction you will need to change the order of the
101I am planning to implement some sort of a `.ignore` file that will enable me to 104I am planning to implement some sort of a `.ignore` file that will enable me to
102have a project-specific exclude options. 105have a project-specific exclude options.
103 106
104I am currently running this every hour as a cronjob which is perfectly fine 107I am currently running this every hour as a cronjob which is perfectly fine for
105for now when I am testing how this whole thing works and how it all will turn 108now when I am testing how this whole thing works and how it all will turn out.
106out.
107 109
108I have also created a small Gnome extension which is still very unstable, but 110I have also created a small Gnome extension which is still very unstable, but
109when/if this whole experiment pays of I will share on Github. 111when/if this whole experiment pays of I will share on Github.
110