-
Notifications
You must be signed in to change notification settings - Fork 8
Home
Welcome to the iceshelf wiki!
This page (currently) only provides a step-by-step guide on how to setup iceshelf
Before we begin, please note that this guide assumes you're running on Ubuntu and that you have root access.
With that out of the way, let's get started.
First, you need to install all the necessary dependencies
sudo apt-get install gpg par2 git python-dev python-pip
This will install the following tools
- OpenGPG : should be pre-installed, but adding it won't hurt
- PAR2 : tool which can create, verify and repair using parity files
- Git : to retreive [iceshelf] and [amazon-glacier-cmd-interface]
- Python development environment and PIP : required for OpenGPG wrapper we're going to install
Next up, we need to install the python wrapper for OpenGPG. Unfortunately, there are two versions of this, so it's very important that you install the correct one, since the incorrect variant lacks functionality that iceshelf needs.
sudo pip install python-gnupg
sudo pip install awscli
Start with adding a new user who will run the backup. I chose iceshelf but as you can see, it leads to some fun path names, but other than that, has no impact. However, feel free to change it to something else, just make sure to follow through with all relevant changes.
sudo adduser --system --disabled-password iceshelf
This sets up a new user without password so no-one can login as that user. Now, having said that, we're going to "login" as the user by virtue of changing our identity.
sudo sudo -Hu iceshelf bash
While it might look like a copy-n-paste error, this changes you to root, who then executes bash as iceshelf while also making sure the HOME variable points to /home/iceshelf. The benefit is that we now appear as the user iceshelf and that our home directory is changed to /home/iceshelf.
As iceshelf, we'll now configure AWS (if you don't intend to use glacier, you can skip this configuration step)
aws configure
You'll be asked for both the AWS Access Key and the AWS Secret Access Key. I highly recommend creating a new user and then creating these keys for this specific user (see "Identity and Access Management" and also this tutorial).
You also need to choose a region and an output format. The last one should be set to json
AWS Access Key ID [None]: MyAwesomeAccessKey
AWS Secret Access Key [None]: MySuperDuperS3cretAwesomeKey
Default region name [None]: eu-west-1
Default output format [None]: json
Next, time to download iceshelf to the home holder
cd
git clone https://github.com/mrworf/iceshelf.git
To be able to use encryption, iceshelf needs a gpg key
~$ gpg --gen-key
gpg (GnuPG) 1.4.16; Copyright (C) 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: directory `/home/iceshelf/.gnupg' created
gpg: new configuration file `/home/iceshelf/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/iceshelf/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/iceshelf/.gnupg/secring.gpg' created
gpg: keyring `/home/iceshelf/.gnupg/pubring.gpg' created
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection?
We're going to choose 1
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0)
I don't want my backup key to expire, so we simply hit enter, which gives us the default of zero, no expiration.
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N)
It sure is, so answer y and continue
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
"Heinrich Heine (Der Dichter) <[email protected]>"
This next couple of questions are up to you, the only thing you MUST change is the email, since it does not make any sense to use mine.
Real name: Iceshelf
Email address: [email protected]
Comment: Automated backup
You selected this USER-ID:
"Iceshelf (Automated backup) <[email protected]>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?
Almost done, we need to okay this and continue
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.
You may or may not get a message saying gpg: gpg-agent is not available in this session which is fine, if you get the message it just means gpg was unable to use fancy graphical UI to ask you for a passphrase.
In my case, I chose no passphrase (simply hitting enter) but if you want additional security, you can add something here.
JUST MAKE SURE TO REMEMBER IT
You don't want a passphrase - this is probably a *bad* idea!
I will do it anyway. You can change your passphrase at any time,
using this program with the option "--edit-key".
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
............+++++
...+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
............+++++
+++++
gpg: /home/iceshelf/.gnupg/trustdb.gpg: trustdb created
gpg: key xxxxxxxx marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
pub 4096R/xxxxxxxx 2016-11-15
Key fingerprint = xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx
uid Iceshelf (Automated backup) <[email protected]>
sub 4096R/xxxxxxxx 2016-11-15
Congratulations, you've just created a key that can be used for encrypting your backup as well as signing the files associated with it (nevermind the x:es in the text above, I didn't really feel like sharing those details).
Of course, now you might want to make a backup of this key. One way is to exporting the private key and saving it somewhere safe
gpg --export-private-key [email protected] >secret-key.gpg
which is probably not a bad idea. But if you're paranoid enough to encrypt and sign, you're probably also aware that USB keys loose data and so do SSDs ([bitflip]), harddrives with spinning platters will fail due to mechanical issues or simply gum up if you let it sit without power. CD-Rs are fairly good if you burn with 1x speed so the pattern is clear, but even they can fail.
So what are you supposed to do? Well, lets turn to paper. Yes, of all easily available material, paper is the one most likely to survive the longest as long as it's kept reasonably dry and out of sunlight. I mean, they still find parchment paper and we have books which are 100+ years old. Just saying.
To accomplish a paper backup of your key, there is a little helper in the other folder of iceshelf, where you'll find a script called analog-key.sh which can aid you with producing a printable key. The same script will also parse and reconstitute back into a gpg key.
But whatever you decide to do, please make a backup of the key, because once it's lost, so are your backups.
A good starting point for your configuration is the example conf file, iceshelf.sample.conf, which is why we're starting with that as a baseline
To keep things simple, we'll stay as the iceshelf user (check using id to see what user you're currently logged in as). If you logged out previously, fret not, just issue
sudo sudo -Hu iceshelf bash
And you'll be back as mr iceshelf.
Now, let's copy the configuration and open it up so we can edit it
cd
cp iceshelf/iceshelf.sample.conf iceshelf.conf
nano -w iceshelf.conf
Locate the [sources] section and get ready to add the files and/or directories you'd like to back up. Remember, all paths should be entered in the absolute form, meaning that you provide every directory starting from root. So if you'd like to backup the iceshelf user, you'd add this:
iceshelf=/home/iceshelf
The iceshelf part before the equal sign is just for your reference. Just keep in mind that it must be unique or only the last version of it will be used. The part after the equal sign is the directory or file you want to have backed up. As you can see, we wrote /home/iceshelf since that's the absolute path of the user's home directory.
This was just an example, and while you can backup iceshelf with iceshelf, there rarely is a need for it and I would highly recommend taking some time to consider which folders and files you want backed up.
While it may seem like a great idea to backup the key itself, it's pointless since we're going to encrypt the backup with it making it impossible to retreive if lost.
There's quite a few options to tweak your backup, so I'm just going to highlight the good ones here and you can explore the rest in the sample configuration's documentation or the README.md file. Most of the defaults are sensible and will not require review for this inital setup.
detect move: no
This is the default behavior, which for most people is fine. What it means in reality is that if you have a file called IMAGE.JPG and you renamed it to MONALISA.JPG, iceshelf would record two changes, the deletion of IMAGE.JPG and the addition of MONALISA.JPG. While this is correct, it has one obvious drawback, if you have a tendency to move or rename a lot of the content you're backing up, it will generate an unnecessary amount of extra data since each move or rename essentially duplicates the file.
By changing this option to yes
detect move: yes
Iceshelf will now do its best to detect when you move or rename a file. When it detects the change, it will simply record the change, and not add a new version of the same file. It will do this for as many times as you move or rename the file, always tracking the original and just storing the change of name/location. However, the moment you modify the file, it will upload a new copy of the modified file.
Why is this behavior merely mentioned and not enabled by default? Because it can be difficult to understand the side effects. Any move/rename with this option enabled will result in a single entry in the manifest.json file. It also complicates restoration of backup, since you now have to manually rename or move the file into place. But if you have a lot of files which move around, this is by far the most efficient way of dealing with it.
But I would suggest testing it out first so you understand how it works.
Another handy option is
skip empty: no
Again, by default, it's not enabled. But when it's enabled, it will avoid creating a backup if there are no changes. What it means is that with skip empty set to no, even if there is no changes, a manifest.json will be created. This manifest is of course empty. Why would you want this? To be consistent, to know that the backup was made but no changes were detected.
Setting it to yes
skip empty: yes
will make Iceshelf cancel the backup if no changes are detected. Benefit? No empty manifest files and no unnecessary uploads to Amazon Glacier.
First thing we're going to do is add your newly created key, since mine is called [email protected], we're adding it to both encryption and signature. Find the [security] section and add the following
encrypt: [email protected]
sign: [email protected]
This will make sure that we can validate that no-one has tampered with the backup (or more likely, confirm it's not corrupt) and also obfuscate the data so no-one but me can access it. Since I didn't put in a passphrase for the key, I'm safely ignoring the encrypt phrase and sign phrase.
I'm also turning on a 5% parity, allowing up to 5% of the contents to be damaged
add parity: 5
This will of course grow the backup by 5% so keep this in mind if you decide to go up in percentages. And really, while 100% sounds cool, it isn't really a good idea. If you're this worried about data loss, you're better off making multiple backups in multiple locations. Typically 5% is enough and I'd be hesitant to go above 15%.
Last step is setting up glacier, which you do under the [glacier] section. It's quite simple, you just point out a suitable name for your vault.
For me, once configured, I simple add the following:
vault: iceshelf-backup
If you're seeing poor performance during uploads to glacier (due to latency or what not) you can try to increase the number of threads being used for the upload. By default it uses 4, but by specifying threads you can set it to whatever you want.
For sending files from California, US to Europe, I tend to set it to 8, like so:
threads: 8
Which yields me about 200K/s per connection.
While the tool was created to allow backup using the Amazon Glacier storage, it will function just as well without glacier. To disable the glacier integration, you can simply comment out or remove the entries under the [glacier] section, and iceshelf will simply skip that step and move the finished backup into the previously defined done dir.
Now that you have a config file for iceshelf, it's time to try it out. Make sure you're still running as iceshelf, otherwise look in the previous section on instructions for changing user.
The easiest way to try it, is to run in --changes mode, which will do all but create a backup
/home/iceshelf/iceshelf/iceshelf --changes /home/iceshelf/iceshelf.conf
Depending on your configuration, this may take a while, but in the end, this is what you should see
max size is limited to 32GB when using parity, changing setting accordingly
First run, no previous checksums
Setting up the prep directory
Checking sources for changes
Processing "iceshelf" (/home/iceshelf)
Detected changes:
"/home/iceshelf/iceshelf/exclusions/dovecot.excl" is new
"/home/iceshelf/iceshelf/testsuite/README.md" is new
"/home/iceshelf/iceshelf/.git/hooks/pre-applypatch.sample" is new
"/home/iceshelf/iceshelf/.git/hooks/pre-rebase.sample" is new
"/home/iceshelf/.bash_history" is new
"/home/iceshelf/iceshelf/testsuite/test.sh" is new
"/home/iceshelf/iceshelf/TODO.md" is new
"/home/iceshelf/.gnupg/trustdb.gpg" is new
"/home/iceshelf/iceshelf/.git/info/exclude" is new
"/home/iceshelf/iceshelf/.git/hooks/pre-push.sample" is new
"/home/iceshelf/iceshelf/other/README.md" is new
"/home/iceshelf/iceshelf/.git/HEAD" is new
"/home/iceshelf/iceshelf/modules/configuration.py" is new
"/home/iceshelf/iceshelf/modules/glacier.py" is new
"/home/iceshelf/iceshelf/testsuite/.gitignore" is new
"/home/iceshelf/iceshelf/modules/fileutils.py" is new
"/home/iceshelf/iceshelf/exclusions/README.md" is new
"/home/iceshelf/iceshelf/.git/description" is new
"/home/iceshelf/iceshelf/iceshelf" is new
"/home/iceshelf/iceshelf/modules/configuration.pyc" is new
"/home/iceshelf/iceshelf/.git/hooks/commit-msg.sample" is new
"/home/iceshelf/iceshelf/modules/helper.py" is new
"/home/iceshelf/iceshelf/.gitignore" is new
"/home/iceshelf/iceshelf/modules/glacier.pyc" is new
"/home/iceshelf/iceshelf/.git/logs/refs/heads/master" is new
"/home/iceshelf/iceshelf/.git/hooks/prepare-commit-msg.sample" is new
"/home/iceshelf/iceshelf/.git/config" is new
"/home/iceshelf/iceshelf/.git/logs/HEAD" is new
"/home/iceshelf/iceshelf/modules/__init__.pyc" is new
"/home/iceshelf/iceshelf/DATABASE.md" is new
"/home/iceshelf/iceshelf/LICENSE" is new
"/home/iceshelf/.gnupg/secring.gpg" is new
"/home/iceshelf/iceshelf/.git/hooks/pre-commit.sample" is new
"/home/iceshelf/.gnupg/pubring.gpg~" is new
"/home/iceshelf/iceshelf/other/analog-key.sh" is new
"/home/iceshelf/iceshelf/.git/packed-refs" is new
"/home/iceshelf/iceshelf/.git/objects/pack/pack-f30d8d9756cc8bb8c962fe6cf8ccaf6889c02edf.pack" is new
"/home/iceshelf/.gnupg/pubring.gpg" is new
"/home/iceshelf/iceshelf/modules/fileutils.pyc" is new
"/home/iceshelf/.gnupg/random_seed" is new
"/home/iceshelf/iceshelf/modules/helper.pyc" is new
"/home/iceshelf/iceshelf/.git/hooks/update.sample" is new
"/home/iceshelf/iceshelf/.git/refs/remotes/origin/HEAD" is new
"/home/iceshelf/iceshelf/.git/hooks/applypatch-msg.sample" is new
"/home/iceshelf/iceshelf/.git/refs/heads/master" is new
"/home/iceshelf/iceshelf/.git/index" is new
"/home/iceshelf/iceshelf/.git/objects/pack/pack-f30d8d9756cc8bb8c962fe6cf8ccaf6889c02edf.idx" is new
"/home/iceshelf/iceshelf.conf" is new
"/home/iceshelf/.lesshst" is new
"/home/iceshelf/iceshelf/.git/logs/refs/remotes/origin/HEAD" is new
"/home/iceshelf/iceshelf/.git/hooks/post-update.sample" is new
"/home/iceshelf/.gnupg/gpg.conf" is new
"/home/iceshelf/iceshelf/iceshelf.sample.conf" is new
"/home/iceshelf/iceshelf/modules/__init__.py" is new
"/home/iceshelf/iceshelf/README.md" is new
"/home/iceshelf/iceshelf/other/iceshelf.service" is new
===============
56 files (305K) to be backed up
One thing you'll realize immediately is that this backup contains a lot of files we didn't want. Which is why it's a good idea to do a "dry run" first so you don't backup a bunch of unnecessary files.
Back to the configuration file, it's time to learn about [exclude] and what it can do for you.
If you read through the configuration file, you might find it daunting in terms of what exclude can do. But in reality, it's fairly simple. By telling iceshelf which files, directories or file sizes you're not interested in, it will filter those out during backup. In our case, we want to get rid of any .git folder and the .gnupg folder.
nogit=?/.git/
nogpg=?/.gnupg/
Another test run and
"/home/iceshelf/iceshelf/exclusions/dovecot.excl" is new
"/home/iceshelf/iceshelf/testsuite/README.md" is new
"/home/iceshelf/.bash_history" is new
"/home/iceshelf/iceshelf/testsuite/test.sh" is new
"/home/iceshelf/iceshelf/TODO.md" is new
"/home/iceshelf/iceshelf/other/README.md" is new
"/home/iceshelf/iceshelf/modules/configuration.py" is new
"/home/iceshelf/iceshelf/modules/glacier.py" is new
"/home/iceshelf/iceshelf/testsuite/.gitignore" is new
"/home/iceshelf/iceshelf/modules/fileutils.py" is new
"/home/iceshelf/iceshelf/exclusions/README.md" is new
"/home/iceshelf/iceshelf/iceshelf" is new
"/home/iceshelf/iceshelf/modules/configuration.pyc" is new
"/home/iceshelf/iceshelf/modules/helper.py" is new
"/home/iceshelf/iceshelf/.gitignore" is new
"/home/iceshelf/iceshelf/modules/glacier.pyc" is new
"/home/iceshelf/iceshelf/modules/__init__.pyc" is new
"/home/iceshelf/iceshelf/DATABASE.md" is new
"/home/iceshelf/iceshelf/LICENSE" is new
"/home/iceshelf/iceshelf/other/analog-key.sh" is new
"/home/iceshelf/iceshelf/modules/fileutils.pyc" is new
"/home/iceshelf/iceshelf/modules/helper.pyc" is new
"/home/iceshelf/iceshelf.conf" is new
"/home/iceshelf/.lesshst" is new
"/home/iceshelf/iceshelf/iceshelf.sample.conf" is new
"/home/iceshelf/iceshelf/modules/__init__.py" is new
"/home/iceshelf/iceshelf/README.md" is new
"/home/iceshelf/iceshelf/other/iceshelf.service" is new
is going to be backed up now. Much better, don't you think? But we can do better. Let's get rid of all *.pyc files, we also don't want the .bash_history or .lesshst files.
nopyc=*.pyc
nobash=/home/iceshelf/.bash_history
noless=/home/iceshelf/.lesshst
And we end up with
"/home/iceshelf/iceshelf/DATABASE.md" is new
"/home/iceshelf/iceshelf/LICENSE" is new
"/home/iceshelf/iceshelf/testsuite/README.md" is new
"/home/iceshelf/iceshelf/modules/fileutils.py" is new
"/home/iceshelf/iceshelf/exclusions/README.md" is new
"/home/iceshelf/iceshelf.conf" is new
"/home/iceshelf/iceshelf/exclusions/dovecot.excl" is new
"/home/iceshelf/iceshelf/other/analog-key.sh" is new
"/home/iceshelf/iceshelf/iceshelf" is new
"/home/iceshelf/iceshelf/testsuite/test.sh" is new
"/home/iceshelf/iceshelf/TODO.md" is new
"/home/iceshelf/iceshelf/modules/helper.py" is new
"/home/iceshelf/iceshelf/.gitignore" is new
"/home/iceshelf/iceshelf/modules/glacier.py" is new
"/home/iceshelf/iceshelf/other/README.md" is new
"/home/iceshelf/iceshelf/iceshelf.sample.conf" is new
"/home/iceshelf/iceshelf/modules/configuration.py" is new
"/home/iceshelf/iceshelf/modules/__init__.py" is new
"/home/iceshelf/iceshelf/README.md" is new
"/home/iceshelf/iceshelf/testsuite/.gitignore" is new
"/home/iceshelf/iceshelf/other/iceshelf.service" is new
Alright, we're good to go.
Now, we could just run the tool manually, like so
/home/iceshelf/iceshelf/iceshelf /home/iceshelf/iceshelf.conf
But that seems a bit inconvenient, there has to be a better way. There is!
If you look under the other folder, you'll find a file called iceshelf-cronjob. This file contains a script which can be called from your crontab (the scheduler which is responsible for running certain tools at certain times). if you've followed the guide this far and stuck to the naming conventions used here for the configuration files and user setup, then you can simply drop the script into the /etc/cron.daily/ folder (or /etc/cron.weekly/ if you want it to run once a week instead).
But wait! If you copy the file, you won't get any updates, so instead of copying the file into /etc/cron.daily/, let's symlink it so it automatically gets updated when you refresh the git repository. If you're still running as user iceshelf (check using id command) then it's time to exit by pressing CTRL-D.
Once you're ready, let's make the symlink
sudo ln -s /home/iceshelf/iceshelf/other/iceshelf-cronjob /etc/cron.daily/
That was it, the cronjob will now run every day using the /home/iceshelf/iceshelf.conf file and save any output to /var/log/iceshelf.log and also send you an email with the result (if you have configured email for your server).
Congratulations, you now have a complete iceshelf setup which is set to run every day.
If you run into issues or have question, feel free to post them on the github.com page