aboutsummaryrefslogtreecommitdiff
path: root/_posts/2020-09-09-digitalocean-sync.md
diff options
context:
space:
mode:
authorMitja Felicijan <mitja.felicijan@gmail.com>2024-02-23 10:35:22 +0100
committerMitja Felicijan <mitja.felicijan@gmail.com>2024-02-23 10:35:22 +0100
commit4abcce013c9ee3053badf2abda77190233066676 (patch)
tree450de7e8fed3c3c7501a9d2e2eb60a676bdfa09e /_posts/2020-09-09-digitalocean-sync.md
parentcdf50cb2e3051200c6ea0628c318d66220b7d1a1 (diff)
downloadmitjafelicijan.com-4abcce013c9ee3053badf2abda77190233066676.tar.gz
Testing thoughts page
Diffstat (limited to '_posts/2020-09-09-digitalocean-sync.md')
-rw-r--r--_posts/2020-09-09-digitalocean-sync.md113
1 files changed, 0 insertions, 113 deletions
diff --git a/_posts/2020-09-09-digitalocean-sync.md b/_posts/2020-09-09-digitalocean-sync.md
deleted file mode 100644
index 38696a9..0000000
--- a/_posts/2020-09-09-digitalocean-sync.md
+++ /dev/null
@@ -1,113 +0,0 @@
1---
2title: Using Digitalocean Spaces to sync between computers
3permalink: /digitalocean-spaces-to-sync-between-computers.html
4date: 2020-09-09T12:00:00+02:00
5layout: post
6type: post
7draft: false
8---
9
10I've been using [Dropbox](https://www.dropbox.com/) for probably **10+ years**
11now and I-ve became so used to it that it runs in the background that I don't
12even imagine a world without it. But it's not without problems.
13
14At first I had problems with `.venv` environments for Python and the only
15solution for excluding synchronization for this folder was to manually exclude a
16specific folder which is not really scalable. FYI, my whole project folder is
17synced on [Dropbox](https://www.dropbox.com/). This of course introduced a lot
18of syncing of files and folders that are not needed or even break things on
19other machines. In the case of **Python**, I couldn't use that on my second
20machine. I needed to delete `.venv` folder and pip it again which synced files
21again to the main machine. This was very frustrating. **Nodejs** handles this
22much nicer and I can just run the scripts without deleting `node_modules` again
23and reinstalling. However, `node_modules` is a beast of its own. It creates so
24many files that OS has a problem counting them when you check the folder
25contents for size.
26
27I wanted something similar to Dropbox. I could without the instant syncing but
28it would need to be fast and had the option for me to exclude folders like
29`node_modules, .venv, .git` and folders like that.
30
31I went on a hunt for an alternative to [Dropbox](https://www.dropbox.com/)
32and found:
33
34- [Tresorit](https://tresorit.com/)
35- [Sync.com](https://sync.com)
36- [Box](https://www.box.com/)
37
38You know, the usual list of suspects. I didn't include [Google
39drive](https://drive.google.com) or [One drive](https://onedrive.live.com/)
40since they are even more draconian than Dropbox.
41
42> All this does not stem from me being paranoid but recently these companies
43> have became more and more aggressive and they keep violating our privacy when
44> they share our data with 3rd party services. It is getting out of control.
45
46So, my main problem was still there. No way of excluding a specific folder from
47syncing. And before we go into "*But you have git, isn't that enough?*", I must
48say, that many of the files (PDFs, spreadsheets, etc) I have in a `git` repo
49don't get pushed upstream to Git and I still want to have them synced across my
50computers.
51
52I initially wanted to use [rsync](https://linux.die.net/man/1/rsync) but I would
53need to then have a remote VPS or transfer between my computers directly. I
54wanted a solution where all my files could be accessible to me without my
55machine.
56
57> **WARNING: This solution will cost you money!** DigitalOcean Spaces are $5 per
58month and there are some bandwidth limitations and if you go beyond that you get
59billed additionally.
60
61Then I remembered that I could use something like
62[S3](https://en.wikipedia.org/wiki/Amazon_S3) since it has versioning and is
63fully managed. I didn't want to go down the AWS rabbit hole with this so I
64choose [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/).
65
66Then I needed a command-line tool to sync between source and target. I found
67this nice tool [s3cmd](https://s3tools.org/s3cmd) and it is in the Ubuntu
68repositories.
69
70```bash
71sudo apt install s3cmd
72```
73
74After installation will I create a new Space bucket on DigitalOcean. Remember
75the zone you will choose because you will need it when you will configure
76`s3cmd`.
77
78Then I visited [Digitalocean Applications &
79API](https://cloud.digitalocean.com/account/api/tokens) and generated **Spaces
80access keys**. Save both key and secret somewhere safe because when you will
81leave the page secret will not be available anymore to you and you will need to
82re-generate it.
83
84```bash
85# enter your key and secret and correct endpoint
86# my endpoint is ams3.digitaloceanspaces.com because
87# I created my bucket in Amsterdam regiin
88s3cmd --configure
89```
90
91After that I played around with options for `s3cmd` and got to the following
92command.
93
94```bash
95# I executed this command from my projects folder
96cd projects
97s3cmd sync --delete-removed --exclude 'node_modules/*' --exclude '.git/*' --exclude '.venv/*' ./ s3://my-bucket-name/projects/
98```
99
100When syncing int he other direction you will need to change the order of the
101`SOURCE` and `TARGET` to `s3://my-bucket-name/projects/` and `./`.
102
103> Be sure that all the paths have trailing slash so that sync knows that this
104> are directories.
105
106I am planning to implement some sort of a `.ignore` file that will enable me to
107have a project-specific exclude options.
108
109I am currently running this every hour as a cronjob which is perfectly fine for
110now when I am testing how this whole thing works and how it all will turn out.
111
112I have also created a small Gnome extension which is still very unstable, but
113when/if this whole experiment pays of I will share on Github.