How to Set Up an Ultimate Backup System, Using Duplicity, Systemd and Thunar on Arch Linux Part Three

How to Setup Automated Duplicity Backups for the User: 

Now that we have set the automated backups for the system, it is time to do the same for the user. The process is similar with some minor modifications. First things first, we need an operation folder where we will place all of the duplicity related files. Create the following folder:

mkdir /home/my_user/.config/duplicity_backup

In order to simplify our life when we are implementing the front end, we will not place all of the duplicity parameters into one file. Create the following file:

vim /home/my_user/.config/duplicity_backup/exclude_list.txt 

This file will contain the list of files and folders that should be excluded from backup. It should be filled in similar to this:

Note that Music, Videos and Pictures are excluded in this case, this is done to conserve space on the remote server. Fill free to include those folders, by removing them from the list.  Now we need to specify the remote repository where the files will be stored:

cd /home/my_user/.config/duplicity_backup/ 
echo "scp://my_user@server///home/my_user/repos/me" > repo_location

 Finally we need to create the backup scripts:

vim back_home_up.sh

with the following content:

Make this script executable and run it:

chmod +x back_home_up.sh
back_home_up.sh

Now you should have your first full backup of home folder on the remote repository. It is time to set up user units for systemd to automate it. Navigate to the ~/.config/systemd/user folder, if it is not there create it. Than create duplicity-remote-backup.service and duplicity-remote-backup.timer

files with the following content:


Now enable the timer by executing:

systemctl --user enable duplicity-remote-backup.timer

This shall conclude the back-end of the duplicity backup system for the user. Next we will discuss the front-end.

How to Implement a Front-end to Duplicity for Thunar, Using thunarx-python:

Thunarx-python provides python bindings for the Thunar Extension Framework. We will be using it to implement a Thunar plugin that provides a submeny with restore options. To install it on Arch Linux you either need to download it from AUR and install it manually or run the following command: 

yaourt thunarx-python

Once the installation is complete, create the following file:

sudo vim /usr/share/thunarx-python/extensions/thunarx-submenu-plugin.py 

with this content: 

Don't forget to edit line 12 and set the dup_folder to the correct location. This will add the submenu option "Restore"  to Thunar when you right click on a file. For this to work you need to add one more additional file to   /usr/local/lib/python2.7 . Execute the following commands in command line:

cd /usr/local/lib/python2.7 
sudo vim dupController.py

and fill it in with:

Now you should have it all up and running, test it by restoring and arbitrary file.

How to Set Up an Ultimate Backup System, Using Duplicity, Systemd and Thunar on Arch Linux Part Two

 

How to Automate Script Calls Using Systemd:

How to Write Systemd Service Units:

Now that we have a working script for performing a system backup, we want to automate it. To get familiarized with systemd I would recommend to read through the wiki pages of arch Linux they are pretty great [1], [2], [3]. The back_sys_up.sh is responsible for backing up the system, independent of any users. That is why we want it to run whenever the machine is powered, and not depend on whether the user is logged in or not. To do so we need to create our service for calling the backup script in  /etc/systemd/system folder:

cd  /etc/systemd/systemsudo 
sudo vim duplicity-system-remote-backup.service

This file must contain the following content:

It is also recommended to find out which services are responsible for the network connection on your machine and add the requirements in this file at the bottom of the [Unit] section:

Requires=your_network_connection.service 
After=yout_network_connection.service

This will ensure that your machine is connected to the network before running the backup service. The [Unit] section contains all of the parameters responsible for the systemd unit, and it is universal for all the systemd units. In our case we just have a short description of what the unit does and its requirements. The [Service] section is unique for systemd service units. We have only two parameters here. Type specifies the type of the service in our case oneshot, which means that the service will stop upon the completion of the script. Since we are going to be using a timer to activate the script every time this is a desired behavior.  The second parameter is the ExecStart, which specifies the script to be executed.

How to Write Systemd Timer Units:

Now lets configure the timer for our service. To do so execute the following command:

sudo vim duplicity-system-remote-backup.timer

It is important that the timer and the service files have the same name. The content of the timer should be as follows:

Here there are two important parameters. OnBootSec specifies the time that should pass after boot in order for the service to start, in this case 5 minutes. OnUnitActiveSec specifies the time that should pass from the last time the unit was active in order to start the service, in this case 1 hour. Now it is time to enable the timer and do some testing. Run the following command to enable the timer:

sudo systemctl enable duplicity-system-remote-backup.timer

In order to test you need to restart. After reboot open two terminal windows. In one of them type:

 journalctl -f

Leave this terminal visible, here you can monitor all the systemd activities. In the second terminal you can type:

systemctl list-timers 

 To see which timers are currently present, how much time is left until they execute, when was the list time they have been active, etc.

Housekeeping:

Since full backups can take a long time on slow networks, and since there is no guaranty that your machine will stay up for the duration of this time, duplicity allows the restarting of interrupted backup processes from the point of interruption. To do so duplicity divides the bulk of your files into blocks and uploads them block by block. Every time a block is being uploaded a lock file is being created by duplicity. This can cause issues when duplicity is unexpectedly interrupted, i.e. the machine is turned off while duplicity is performing a backup, or something along this lines. The problem is that after reboot the lock file is still in place, and duplicity will refuse to perform backups if there is a lock file from another instance. To overcome this issue a small script must be written that will be activated every time after boot, and that will look for the lock files and delete them before duplicity is activated. To do so create the following file:

sudo vim /root/scripts/system_backup/remove_locks.sh

and fill it in as follows:

In order to save time I took a shortcut and also added the remove functionality for the my_user in to this script as well. By default the lock files are stored in ~/.cache/duplicity folder. If you did not explicitly specify a different folder this script should do the trick. To automate this process we need to create the systemd service and timer units for this script. To do so create duplicity-cleanup-onboot.service  duplicity-cleanup-onboot.timer  files in /etc/systemd/system/ folder, with the following content:

Now you should have s fully functioning system backup using duplicity. In the next part we will describe the creation of the back-end as well as the front-end for the backup system of the user.