Marcel's Linux Walkabout: Keeping in Sync with Your Data
- "Data, Meet rsync."
- It's Time To Get in rsync
- That Secure Feeling
- Moving On
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
In my latest Linux Walkabout, I found myself visiting an old problem: keeping data safe. If your system suffers a disk crash or other equally painful mishap, you can always reinstall the operating system and put back the applications, but data is paramount. In the end, it is really the only thing that is important. Knowing all that, it's amazing how difficult it can be for people to get that data backed up.
There are reasons, of course. Aside from just "not getting around to it," the obvious reason is that we have so much data these days. It has been a long time since we moved from diskettes to CD-ROMs, but even they are quickly filled. Many people I know who do regular backups are doing theirs to a disk on another machine. Machine A backs to machine B, which backs to C, which backs to A. In the absence of another machine, others still are backing up to a second disk on the same machine. Given that an extra hard drive added to a system is extremely inexpensive these days and high capacity tape drives can still cost substantially more, it isn't that unusual to find this kind of solution being used.
"Data, Meet rsync."
The ideal solution in any of these scenarios is to do one big backup; then simply make sure that the backup is always up-to-date. In other words, back up only what has changed. The result is a perfect mirror: One directory (and the accompanying subdirectories) is mirrored either locally or on a remote system. One of the best tools available to achieve this mirroring is a program called rsync.
With rsync, you can synchronize files and directories on one system with that of another. Because only those files that have been modified are transferred, the process can be very quick. You can do this with single files, whole directories, and subdirectories while maintaining file ownership and permissions, links, symbolic links, and so on. rsync has its own transport, or you can use OpenSSH to secure the transfer (something I'll cover later). You can even mirror with anonymous rsync if you wish...but as they say, I am getting ahead of myself.
You can find rsync at http://rsync.samba.org, and it is freely distributed under the GPL. That said, you may not have to go hunting for rsync. Your Linux distribution may already come with this handy piece of software, so check your distribution CDs first. If you go for the source instead, you'll find that building the package is simply a matter of the classic extract and build five-step.
tar -xzvf rsync-2.5.6.tar.gz cd rsync-2.5.6 ./configure make su -c "make install"
This magic synchronization of files and directories is done using a client and server setup. At least one machine must play the role of server; although nothing stops you from running an rsync daemon on every one of your machines, however, the client machine must also have the rsync program installed. The server gets its information about who can access what from a configuration file called rsyncd.conf. You'll find that it probably lives in the /etc directory. The following listing is from an rsync server I've just set up:
hosts allow = 192.168.22.0/24
use chroot = no
max connections = 4
log file = /var/log/rsyncd.log
gid = nogroup
uid = nobody
[website]
path = /mnt/data1/website
read only = no
comment = All our websites
[mailman]
path = /mnt/extradrive/mailman
read only = no
comment = Mailman lists and archives
[marcel]
path = /mnt/backups/marcel
read only = no
comment = Backup area for Marcel
The above configuration file needs a little explaining, but it is all very simple once you get the hang of it. For starters, if you've used Samba at all, you might have noticed that the format of the file is quite similar. Given that the author of rsync, Andrew Tridgell, is also one of the authors of Samba, that won't seem so strange. Backup areas are identified by a name in square brackets (website, mailman, etc.) The chief bits of information there are the path to the disk area and some kind of comment.
In each of these sections, I also specified "read only = no", but I could just as easily have added that to the top section (the one without a name in square brackets). That's the global section. Anything put up there applies to all other sections (which can then be overridden). Pay particular attention to the gid and uid sectionsβnamely, the group and user id that the file transfer takes place as. The default is nobody, but you need to make sure that this is the case on your system. One of my servers has no nobody group, but rather a nogroup.
The "hosts allow" section identifies my local subnet as being the only set of addresses from which transfers can take place. The "log file" identifies a file to log information from the daemon. You can also specify a maximum number of connections, specific users who are allowed to transfer files (auth users), and a whole lot more. Run "man rsyncd.conf" for the full details. When your configuration is set, you can launch the rsync daemon, which interestingly enough is exactly the same program as the rsync command itself. rsync can be run from xinetd (or inetd) or as a daemon. To run it as a daemon (as I have chosen to do), do the following.
rsync --daemon