Skip to Content

DIY encrypted, version controlled & synchronised filesystem for Ubuntu

whirlpool's picture
I want to explain how I managed to set up a user home filesystem that has the following features:
  • Encrypted
  • Backed up regularly
  • Backups are encrypted
  • Enables you to roll back to previous versions of files.
  • Synchronises between my laptop and desktop (like Dropbox or Ubuntu One) but for almost all files.
  • Synchronisation can occur over LAN or internet.

Software used:

  • Ubuntu 10.10 Maverick Meerkat
  • Deja Dup
  • Unison
  • Libnotify-bin
  • SSH
  • Cron
  • Custom bash script

Steps:

  1. Checked if I want any files from the desktop computer and copied them to laptop. You can use unison for this to get acquainted with it.
  2. Backed up my laptop using Deja Dup.
  3. Installed Ubuntu 10.10 and formatted the home partition. I enabled the encrypted home option from the installation. You can't have an encrypted home partition without formatting it and creating a new one. The installer fails to tell you so if you don't ask it to format and proceeded with the encryption option.
  4. Install Deja Dup and restore the backup using Deja Dup. There is a bug in Deja Dup that will not allow you to restore your home partition files in their original location. So restore them in a folder called restore/ and then move them one level up. A fix was committed but I don't know which version works with it. Restoring will take some time.
  5. Do the the same thing as in step 2 for the desktop.
  6. Restore the same back up of your laptop on to the desktop as in step 3. This step isn't necessary but it saves time copying files through the network.
  7. Install SSH server and client metapackage on desktop
  8. Set up passwordless SSH
  9. Install Unison (not unison-gtk) on both machines
  10. Register with http://www.dyndns.com/ (if you don't have a static IP and want to synchronise through the internet)
  11. Setup your router NAT settings to redirect port 22 to the internal IP of your desktop
  12. Setup your router to update DynDNS (most new versions can do this)
  13. Setup your Unison profiles. I created two one for LAN sync (called lansync.prf) and internet sync (called netsync.prf) I copied Micah's setup. And save them in ~/.unison on your laptop.
  14. ### NETSYNC ###
    ### ROOT SYNC PATHS ###
    
    # first root is my home directory on this laptop
    root = /home/username/
    
    # second directory is my desktop's home folder over SSH 
    root = ssh://[email protected]//home/username/
    
    ### OPTIONS ###
    
    # place new files at the top of the list
    # sortnewfirst = true
    
    # turn on ssh compression
    rshargs = -C
    
    ### PATHS TO SYNCHRONIZE ###
    
    # sync all of my email data
    # path = .evolution/
    
    # sync up firefox
    path = .mozilla/firefox/moftasa.default/
    
    # sync all gFTP for the bookmarks and cache
    # path = .gftp/
    
    # gaim/pidgin IM client logs and settings
    path = .purple/
    
    # Liferea
    path = .liferea_1.6/
    
    # configuration
    path = .config/Lyx/
    path = .config/zim/
    
    # local data
    path = .local/share/Empathy/
    path = .fonts/
    path = .alexandria/
    
    
    # Personal folders
    path = Templates/
    path = Calibre/
    path = bin/
    path = Notes/
    path = Downloads/
    path = Documents/
    path = Videos/
    path = Personal/
    path = Pictures/
    path = Desktop/
    
    ### IGNORE RULES ###
    
    # I don't think these will break anything, but let's ignore anyway
    ignore = Path .mozilla/firefox/moftasa.default/Cache/*
    ignore = Path .liferea_1.6/cache/*
    
    ignore = Name Thumbs.db
    ignore = Name *.tmp
    ignore = Name *.tmp-bad
    ignore = Name temp.*
    ignore = Name *~
    ignore = Name .*~
    
    ######
    ######
    
    
    ### LANSYNC ###
    ###### ROOT SYNC PATHS ###
    
    # first root is my home directory on this laptop
    root = /home/username/
    
    # second directory is my desktop's home folder over SSH 
    root = ssh://[email protected]//home/username/
    
    ### OPTIONS ###
    
    # place new files at the top of the list
    # sortnewfirst = true
    
    # turn on ssh compression
    rshargs = -C
    
    ### PATHS TO SYNCHRONIZE ###
    
    # sync all of my email data
    # path = .evolution/
    
    # sync up firefox 
    path = .mozilla/firefox/moftasa.default/
    
    # sync all gFTP for the bookmarks and cache
    # path = .gftp/
    
    # gaim/pidgin IM client logs and settings
    path = .purple/
    
    # Liferea
    path = .liferea_1.6/
    
    # configuration
    path = .config/Lyx/
    path = .config/zim/
    
    # local data
    path = .local/share/Empathy/
    path = .fonts/
    path = .alexandria/
    
    
    # Personal folders
    path = Templates/
    path = Calibre/
    path = bin/
    path = Notes/
    path = Downloads/
    path = Documents/
    path = Videos/
    path = Personal/
    path = Pictures/
    path = Desktop/
    
    ### IGNORE RULES ###
    
    # I don't think these will break anything, but let's ignore anyway
    ignore = Path .mozilla/firefox/moftasa.default/Cache/*
    ignore = Path .liferea_1.6/cache/*
    
    ignore = Name Thumbs.db
    ignore = Name *.tmp
    ignore = Name *.tmp-bad
    ignore = Name temp.*
    ignore = Name *~
    ignore = Name .*~
    
    ######
    ######
    
    
  15. Install libnotify-bin if you nice notifications like these
  16. Notification
  17. Put the following bash script in your ~/bin/ directory as unison-notify.sh
    #!/bin/bash
    message=`grep 'Synchronization\|Fatal error' ~/.unison/unison.log | tail -n 1`
    icon=/usr/share/icons/gnome/48x48/emblems/emblem-synchronizing.png
    notify-send -i $icon "Unison" "$message"
    
  18. Run this to make it executable:
    $ chmod a+x unison-notify.sh
  19. Adjust cron by adding the following line (run crontab -e to edit the cron table) this will run unison every 15 minutes.
    0, 15, 30, 45 * * * * unison netsync -batch -silent ; unison lansync -batch -silent ; DISPLAY=:0.0 /home/**username**/bin/unison-notify.sh
    

Discussion:

Having your desktop and laptop synchronised magically like that is very impressive and takes a bit of time to get used to. It's also good to know that everything is backed up and encrypted. Compared to Dropbox or Ubuntu One this wins on the ability to have unlimited amounts of data synchronised without paying anything. Though, I'll need to buy a UPS for the desktop. No web or mobile interface but you can access your desktop data via SSH or SFTP.

However, this is a hack that might not be suitable for everyone and takes a considerable amount of time to set up (It took me two days as I had to learn more about Unison and the annoying Deja Dup bug wasted time). It's very important to make sure you have a good back up of your files and that everything is set up in the way that works for you. Read the pages I linked to (they are listed again at the bottom of this post). They explain things in more detail, think the above is just an outline for what to do. You also need to be acquainted with Unison as you might want to run it manually to force certain decisions regarding conflicts. With good set up this should be very rare (like when you update the same file from two different machines).

I decided that my laptop is the computer that starts the synchronisation every time. If you want to check the details of the synchronsiations Unison's log file is at ~/.unison/unison.log

Ubuntu enables the encryption of the filesystem and this is the easiest part. Deja Dup is a very easy user backup tool and allows you to roll back to previous versions of each file. The different versions are the back up files themselves. This means that the version information is stored on your back up drive which saves hard disk space. Power users will not be happy to know that you can't use cron with it. That's because it uses it's own program that checks for USB drives inserted etc..

I don't sync all files, specially Gnome settings as they tend to change often and change between synchronisations which Unison will skip. The same problem happened with a few firefox files that change very often. It's important to say that Unison seems to be very safe and doesn't corrupt data and when it fails or disconnects it does so gracefully.

Links:

Comments

great howto, nice formating,

great howto, nice formating, simply AWESOME



Linux
Let's Free The World

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.


Dr. Radut | blog