How to Set Up an Ultimate Backup System, Using Duplicity, Systemd and Thunar on Arch Linux Part Three

How to Setup Automated Duplicity Backups for the User: 

Now that we have set the automated backups for the system, it is time to do the same for the user. The process is similar with some minor modifications. First things first, we need an operation folder where we will place all of the duplicity related files. Create the following folder:

mkdir /home/my_user/.config/duplicity_backup

In order to simplify our life when we are implementing the front end, we will not place all of the duplicity parameters into one file. Create the following file:

vim /home/my_user/.config/duplicity_backup/exclude_list.txt 

This file will contain the list of files and folders that should be excluded from backup. It should be filled in similar to this:

	/home/my_user/Music
	/home/my_user/.cache
	/home/my_user/Downloads
	/home/my_user/.gnupg
	/home/my_user/.ssh/id_rsa
	/home/my_user/.local
	/home/my_user/Pictures
	/home/my_user/Templates
	/home/my_user/tmp
	/home/my_user/Videos
	/home/my_user/.config/google-chrome/

view raw exclude_list.txt hosted with ❤ by GitHub

Note that Music, Videos and Pictures are excluded in this case, this is done to conserve space on the remote server. Fill free to include those folders, by removing them from the list.  Now we need to specify the remote repository where the files will be stored:

cd /home/my_user/.config/duplicity_backup/ 
echo "scp://my_user@server///home/my_user/repos/me" > repo_location

 Finally we need to create the backup scripts:

vim back_home_up.sh

with the following content:

Make this script executable and run it:

chmod +x back_home_up.sh
back_home_up.sh

Now you should have your first full backup of home folder on the remote repository. It is time to set up user units for systemd to automate it. Navigate to the ~/.config/systemd/user folder, if it is not there create it. Than create duplicity-remote-backup.service and duplicity-remote-backup.timer

files with the following content:

	[Unit]
	Description=Duplicity remote home backup
	[Service]
	Type=oneshot
	ExecStart=/home/my_user/.config/duplicity_backup/back_home_up.sh

view raw duplicity-remote-backup.service hosted with ❤ by GitHub

	[Unit]
	Description=Duplicity remote home backup timer
	[Timer]
	OnBootSec=5m
	OnUnitActiveSec=1h
	[Install]
	WantedBy=timers.target

view raw duplicity-remote-backup.timer hosted with ❤ by GitHub

Now enable the timer by executing:

systemctl --user enable duplicity-remote-backup.timer

This shall conclude the back-end of the duplicity backup system for the user. Next we will discuss the front-end.

How to Implement a Front-end to Duplicity for Thunar, Using thunarx-python:

Thunarx-python provides python bindings for the Thunar Extension Framework. We will be using it to implement a Thunar plugin that provides a submeny with restore options. To install it on Arch Linux you either need to download it from AUR and install it manually or run the following command: 

yaourt thunarx-python

Once the installation is complete, create the following file:

sudo vim /usr/share/thunarx-python/extensions/thunarx-submenu-plugin.py 

with this content: 

	import thunarx
	import gtk
	import sys
	import urllib
	from os import path
	sys.path.append("/usr/local/lib/python2.7")
	from dupController import DupController
	home_folder = path.expanduser("~") + "/"
	dup_folder = home_folder + ".config/duplicity_backup"
	__dpc = DupController(dup_folder)
	def __extractAddress(obj):
	return urllib.unquote(obj.get_uri()[7:])
	"""
	Thunarx Submenu Plugin
	This plugin shows an example of a MenuProvider plugin that implements
	sub-menus. The example used here requires the developer to sub-class
	gtk.Action and override the create_menu_item virtual method.
	"""
	class MyAction(gtk.Action):
	__gtype_name__ = "MyAction"
	def __init__(self, name, label, tooltip=None, stock_id=None, menu_handler=None):
	gtk.Action.__init__(self, name, label, tooltip, stock_id)
	self.menu_handler = menu_handler
	self.obj_type = None
	self.obj_addresses = None
	def create_menu_item(self):
	menuitem = gtk.MenuItem(self.get_label())
	if self.menu_handler is not None:
	menu = gtk.Menu()
	menuitem.set_submenu(menu)
	self.menu_handler(menu, self.obj_type, self.obj_addresses)
	return menuitem
	do_create_menu_item = create_menu_item
	# Defined By me!
	def ManageDuplicityController(self, objects, option):
	for obj in objects:
	__dpc.restore_file(obj, option)
	def PyFileActionMenu(menu, obj_type, obj_addresses):
	create_menu = False
	backed_up_objects = []
	try:
	for obj in obj_addresses:
	obj_name = __extractAddress(obj)
	if __dpc.is_backed_up(obj_name):
	create_menu = True
	backed_up_objects.append(obj_name)
	except TypeError:
	pass
	if create_menu:
	options = __dpc.get_options()
	for option in options:
	action = gtk.Action("TMP:"+option[0], option[0], None, None)
	subitem = action.create_menu_item()
	menu.append(subitem)
	subitem.show()
	action.connect("activate", ManageDuplicityController, backed_up_objects, option[1])
	class ThunarxSubMenuProviderPlugin(thunarx.MenuProvider):
	def __init__(self):
	pass
	def get_file_actions(self, window, files):
	res = MyAction("TMP:Restore", "Restore", "Backup Management",
	gtk.STOCK_FILE, menu_handler=PyFileActionMenu)
	res.obj_type = "File"
	res.obj_addresses = files
	return [res]
	def get_folder_actions(self, window, folder):
	res = MyAction("TMP:Restore", "Restore",
	"Backup Management", gtk.STOCK_DIRECTORY, menu_handler=PyFileActionMenu)
	res.obj_type = "Folder"
	res.obj_addresses = folder
	return [res]

view raw thunarx-submenu-plugin.py hosted with ❤ by GitHub

Don't forget to edit line 12 and set the dup_folder to the correct location. This will add the submenu option "Restore"  to Thunar when you right click on a file. For this to work you need to add one more additional file to   /usr/local/lib/python2.7 . Execute the following commands in command line:

cd /usr/local/lib/python2.7 
sudo vim dupController.py

and fill it in with:

	from __future__ import print_function
	from os import path
	import shlex
	from collections import deque
	from itertools import islice
	from subprocess import Popen, PIPE, STDOUT
	from threading import Thread
	try:
	import Tkinter as tk
	except ImportError:
	import tkinter as tk # Python 3
	try:
	from Queue import Queue, Empty
	except ImportError:
	from queue import Queue, Empty # Python 3
	info = print
	def iter_except(function, exception):
	"""Works like builtin 2-argument `iter()`, but stops on `exception`."""
	try:
	while True:
	yield function()
	except exception:
	return
	class DupControllerGUI:
	"""
	Disclaimer: The original structure of this class has been borrowed and from:
	https://gist.github.com/zed/42324397516310c86288
	it has been modified to fit our purpose.
	Implements a threaded gui and a background call to duplicity
	"""
	def __init__(self, root, opt_args):
	self.root = root
	# start duplicity backup process
	self.proc = Popen(opt_args, stdout=PIPE, stderr=STDOUT)
	# launch thread to read the subprocess output
	# (put the subprocess output into the queue in a background thread,
	# get output from the queue in the GUI thread.
	# Output chain: proc.readline -> queue -> stringvar -> label)
	q = Queue()
	t = Thread(target=self.reader_thread, args=[q]).start()
	# GUI Elements
	self._message_var = tk.StringVar()
	self._message_var.set("Restoring: Please restrain from stopping until the process has finished")
	tk.Label(root, font=("Times", 13, "bold"), textvariable=self._message_var).pack()
	# show subprocess' stdout in GUI
	self._stdout_text = tk.Text(root, font=("Mono", 12, "bold"), bg="gray11", fg="antique white")
	self._stdout_text.insert(tk.END, "Starting restore process: \n")
	self._stdout_text.pack()
	self._stdout_text.config(state="disabled")
	# stop subprocess using a button
	self._button = tk.Button(root, text="Stop", font=("Mono", 11, "bold"), command=self.stop)
	self._button.pack()
	self.update(q) # start update loop
	# Set to Busy
	self.set_cursor("watch")
	def set_cursor(self, param):
	"""Set cursor to param"""
	self.root.config(cursor=param)
	self._stdout_text.config(cursor=param)
	def reader_thread(self, q):
	""" Read from stdout in subprocess thread"""
	info('Start Duplicity')
	is_finished = False
	"""Read subprocess output and put it into the queue."""
	for line in iter(self.proc.stdout.readline, b''):
	q.put([line, is_finished])
	line = "Restoring process has successfully finished \n"
	is_finished = True
	q.put([line, is_finished])
	info('Finish Duplicity')
	def update(self, q):
	"""Update GUI with items from the queue."""
	# read no more than 10000 lines, use deque to discard lines except the last one,
	for data in deque(islice(iter_except(q.get_nowait, Empty), 10000), maxlen=1):
	if data is None:
	return # stop updating
	else:
	line = data[0]
	is_finished = data[1]
	self._stdout_text.config(state="normal")
	self._stdout_text.insert(tk.END, line) # update GUI
	self._stdout_text.config(state="disabled")
	if is_finished:
	self._button.config(text="Finish")
	self.set_cursor("")
	self.root.after(40, self.update, q) # schedule next update
	def stop(self):
	"""Stop subprocess and quit GUI."""
	info('stoping')
	self.proc.terminate() # tell the subprocess to exit
	# kill subprocess if it hasn't exited after a countdown
	def kill_after(countdown):
	if self.proc.poll() is None: # subprocess hasn't exited yet
	countdown -= 1
	if countdown < 0: # do kill
	info('killing')
	self.proc.kill() # more likely to kill on *nix
	else:
	self.root.after(1000, kill_after, countdown)
	return # continue countdown in a second
	# clean up
	self.proc.stdout.close() # close fd
	self.proc.wait() # wait for the subprocess' exit
	self.root.destroy() # exit GUI
	kill_after(countdown=5)
	class DupController:
	"""
	Back-end control of duplicity restore process. Made for integration with thunarx-python
	"""
	def __init__(self, duplicity_folder):
	"""
	:param duplicity_folder: Root operation folder of duplicity.
	"""
	if type(duplicity_folder) is not type(str()):
	raise TypeError("Type of the Duplicity operation folder must be str")
	if duplicity_folder.endswith("/"):
	self.__op_folder = duplicity_folder
	else:
	self.__op_folder = duplicity_folder + "/"
	if not path.exists(self.__op_folder):
	raise ValueError("The Duplicity operation folder does not exist")
	self.exclude_list = "exclude_list.txt"
	self.root_folder = path.expanduser("~") + "/"
	with open(self.__op_folder+"repo_location", "r") as f:
	self.repo_location = f.readline()
	def is_backed_up(self, obj_address):
	""" Parse the exclude file list, determine if the file is excluded or not
	:param obj_address the address of the object to be restored
	"""
	is_excluded = False
	excl_path = self.__op_folder + self.exclude_list
	if path.exists(excl_path):
	with open(excl_path, 'r') as f:
	for line in f:
	if line.strip("\n") in obj_address:
	is_excluded = True
	return not is_excluded
	def __extract_rel_file_path(self, full_path):
	"""
	:param full_path: Absolute path to the file located in users home directory.
	:return: Relative path to the, in relation to home folder.
	"""
	root_folder_len = len(self.root_folder)
	curr_root_folder = full_path[0:root_folder_len]
	if curr_root_folder != self.root_folder:
	raise ValueError("Invalid root folder \n " +
	" is : " + curr_root_folder + "\n" +
	"should : " + self.root_folder)
	return full_path[root_folder_len:len(full_path)]
	@staticmethod
	def get_options():
	"""
	Duplicity restore options. Feel free to modify or append in case a different structure is needed
	:return: list of options in [display name, duplicity command] format.
	"""
	return [["Jump to 1h ago", "-t1h"],
	["Jump to 2h ago", "-t2h"],
	["Jump to 3h ago", "-t3h"],
	["Jump to 1D ago", "-t1D"],
	["Jump to 2D ago", "-t2D"],
	["Jump to 1W ago", "-t1W"],
	["Jump to 2W ago", "-t2W"]]
	def restore_file(self, file_path, date_str):
	"""
	Restore a file to a particular date in time. The restored file will have a prefix _restored
	and wont override the current file
	:param file_path: Absolute file path
	:param date_str: Date string obtained from self.gat_options()
	"""
	rel_file_str = "--file-to-restore " + self.__extract_rel_file_path(file_path)
	opt_string = "duplicity --no-encryption "
	opt_string += date_str + " " + rel_file_str + " "
	opt_string += self.repo_location + " " + file_path + "_restored"
	opt_args = shlex.split(opt_string)
	info(opt_args)
	root = tk.Tk()
	app = DupControllerGUI(root, opt_args)
	root.protocol("WM_DELETE_WINDOW", app.stop) # exit subprocess if GUI is closed
	root.title("Restoring")
	# Set icon, in case the file is missing fill free to set it to whatever icon you like or comment it out
	icon_img = tk.Image("photo", file='/usr/share/icons/gnome/16x16/devices/gnome-dev-cdrom-audio.png')
	root.tk.call('wm', 'iconphoto', root._w, icon_img)
	root.mainloop()
	info('exited')

view raw dupController.py hosted with ❤ by GitHub

Now you should have it all up and running, test it by restoring and arbitrary file.