1---
  2title: Using Digitalocean Spaces to sync between computers
  3url: digitalocean-spaces-to-sync-between-computers.html
  4date: 2020-09-09T12:00:00+02:00
  5type: post
  6draft: false
  7---
  8
  9I've been using [Dropbox](https://www.dropbox.com/) for probably **10+ years**
 10now and I-ve became so used to it that it runs in the background that I don't
 11even imagine a world without it. But it's not without problems.
 12
 13At first I had problems with `.venv` environments for Python and the only
 14solution for excluding synchronization for this folder was to manually exclude a
 15specific folder which is not really scalable. FYI, my whole project folder is
 16synced on [Dropbox](https://www.dropbox.com/). This of course introduced a lot
 17of syncing of files and folders that are not needed or even break things on
 18other machines. In the case of **Python**, I couldn't use that on my second
 19machine. I needed to delete `.venv` folder and pip it again which synced files
 20again to the main machine. This was very frustrating. **Nodejs** handles this
 21much nicer and I can just run the scripts without deleting `node_modules` again
 22and reinstalling. However, `node_modules` is a beast of its own. It creates so
 23many files that OS has a problem counting them when you check the folder
 24contents for size.
 25
 26I wanted something similar to Dropbox. I could without the instant syncing but
 27it would need to be fast and had the option for me to exclude folders like
 28`node_modules, .venv, .git` and folders like that.
 29
 30I went on a hunt for an alternative to [Dropbox](https://www.dropbox.com/) 
 31and found:
 32
 33- [Tresorit](https://tresorit.com/)
 34- [Sync.com](https://sync.com)
 35- [Box](https://www.box.com/)
 36
 37You know, the usual list of suspects. I didn't include [Google
 38drive](https://drive.google.com) or [One drive](https://onedrive.live.com/)
 39since they are even more draconian than Dropbox.
 40
 41> All this does not stem from me being paranoid but recently these companies 
 42> have became more and more aggressive and they keep violating our privacy when 
 43> they share our data with 3rd party services. It is getting out of control.
 44
 45So, my main problem was still there. No way of excluding a specific folder from
 46syncing. And before we go into "*But you have git, isn't that enough?*", I must
 47say, that many of the files (PDFs, spreadsheets, etc) I have in a `git` repo
 48don't get pushed upstream to Git and I still want to have them synced across my
 49computers.
 50
 51I initially wanted to use [rsync](https://linux.die.net/man/1/rsync) but I would
 52need to then have a remote VPS or transfer between my computers directly.  I
 53wanted a solution where all my files could be accessible to me without my
 54machine.
 55
 56> **WARNING: This solution will cost you money!** DigitalOcean Spaces are $5 per
 57month and there are some bandwidth limitations and if you go beyond that you get
 58billed additionally.
 59
 60Then I remembered that I could use something like
 61[S3](https://en.wikipedia.org/wiki/Amazon_S3) since it has versioning and is
 62fully managed. I didn't want to go down the AWS rabbit hole with this so I
 63choose [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces/).
 64
 65Then I needed a command-line tool to sync between source and target. I found
 66this nice tool [s3cmd](https://s3tools.org/s3cmd) and it is in the Ubuntu
 67repositories.
 68
 69```bash
 70sudo apt install s3cmd
 71```
 72
 73After installation will I create a new Space bucket on DigitalOcean. Remember
 74the zone you will choose because you will need it when you will configure
 75`s3cmd`.
 76
 77Then I visited [Digitalocean Applications &
 78API](https://cloud.digitalocean.com/account/api/tokens) and generated **Spaces
 79access keys**. Save both key and secret somewhere safe because when you will
 80leave the page secret will not be available anymore to you and you will need to
 81re-generate it.
 82
 83```bash
 84# enter your key and secret and correct endpoint
 85# my endpoint is ams3.digitaloceanspaces.com because
 86# I created my bucket in Amsterdam regiin
 87s3cmd --configure
 88```
 89
 90After that I played around with options for `s3cmd` and got to the following
 91command.
 92
 93```bash
 94# I executed this command from my projects folder
 95cd projects
 96s3cmd sync --delete-removed --exclude 'node_modules/*' --exclude '.git/*' --exclude '.venv/*' ./ s3://my-bucket-name/projects/
 97```
 98
 99When syncing int he other direction you will need to change the order of the 
100`SOURCE` and `TARGET` to `s3://my-bucket-name/projects/` and `./`.
101
102> Be sure that all the paths have trailing slash so that sync knows that this 
103> are directories.
104
105I am planning to implement some sort of a `.ignore` file that will enable me to
106have a project-specific exclude options.
107
108I am currently running this every hour as a cronjob which is perfectly fine for
109now when I am testing how this whole thing works and how it all will turn out.
110
111I have also created a small Gnome extension which is still very unstable, but
112when/if this whole experiment pays of I will share on Github.