Basic Backup Script in BASH

[ehansen]

Introduction
Everyone should be backing up their data. This just doesn't go towards sysadmins, but even people at home who never even think of it. Just like everything else in I.T., hard drives were built to fail. If you do not make efficient back ups, you are at your own mercy when your drive will no longer spin up, meaning all that data is now gone.

I try to make as many of my tasks as easy as possible simply by using Bash scripting. Its the most portable language meaning 99.9% of the time, you aren't going to need bash to run the script...and even if so, most of the code I write can work with other shells as well.

I'm going to copy/paste parts of my script, and give details about each block of code. At the end, I'll also provide a link to the full script.

What WIll This (Not) Do?
This script will do the following:

• Back up the specified directories
• Create a directory based on month & date
• Create a separate backup for each given device
• E-mail you the results (this assumes a working SSMTP, which is out of the scope of this particular guide)

Pretty much everything else is not going to be done. This is a quick and dirty backup solution, and isn't intended to be the absolute answer. I wrote this originally for my own needs and my needs only.

Lets Begin

DAY=`date +%d`

MONTH=`date +%m`

YEAR=`date +%Y`

Here we get various information needed for storing our backup information. Pretty straight forward with a smile at the end if I do say so myself. Pretty easy, eh?

# Backup directory to use (2011/08/31 for 08.31.2011)
BKDIR="/backups/$YEAR/$MONTH/$DAY"

if [ ! -d "$BKDIR" ]; then
        mkdir -p $BKDIR
fi

We make the directory where the backup files will be stored (-p ensures any missing directories will be made). The script must have write permissions to the BKDIR (in this case /backups/), or else it will fail.

BKLOG="/backups/$YEAR/$MONTH/$DAY.log"

Log of information (will make more sense later). I like things being consistent.

ARRPOS=0

I'll be honest, I can't really explain this, but it is used, and should make more sense when you actually see its use. Its like trying to explain to someone new to computers how a keyboard makes a letter appear...you just tell them "just press the key to see its power" if you want to keep them interested in you.

DRIVE=('sda' 'sdb')

This backup solution is device-based, and my server has two devices (one main, another with misc. data). The end result will basically be $BKDIR/$DRIVE[$ARRPOS].tar.gz (i.e.: /backups/2012/01/26/sda.tar.gz)

BACKUP=('/' '/pub')

What to back up on each device (for me, this is backing up everything).

SDAEX=('/media' '/tmp' '/dev' '/proc' '/sys' '/mnt' '/pub' '/var/cache' '/backups')

A lot of these aren't needed, and we also do not want to back up our back ups by default.

touch $BKLOG

Create an empty file for the log file, making sure it can be made.

echo "To: someone@somewhere.com" > $BKLOG
echo "From:Backups <backups@secrets>" >> $BKLOG
echo -e "Subject: Generated backup report for `hostname` on $YEAR.$MONTH.$DAY\n" >> $BKLOG

echo -e ">> Backup for: $YEAR.$MONTH.$DAY started @ `date +%H:%M:%S`\n" >> $BKLOG

The purpose of $BKLOG is to log the status of the back up process. When it is done, we will be e-mailing the report out (see "To:" field). You can change this however you want, this is how I did it for myself.

# Checks to see if day = 1, and if so, backs up the last month's backups
if [ "$DAY" == "01" ]; then
        M=`echo -n $MONTH | awk '{printf substr($1,2)}'`
        let OLD=M-1

        echo "- New month detected.  Backing up previous month's ($OLD) backups." >> $BKLOG
        echo "   + Backup file: /backups/$YEAR/$OLD.tar.gz" >> $BKLOG
        SD=$( { time tar -cpPzf /backups/$YEAR/$OLD.tar.gz /backups/$YEAR/$OLD/; } 2>&1 )

        # Got stats, delete folder
        rm -rf /backups/$YEAR/$OLD

        SD=`echo -n "$SD" | grep real`
        MIN=`echo -n "$SD" | awk '{printf substr($2,0,2)}'`
        SEC=`echo -n "$SD" | awk '{printf substr($2,3)}'`
        echo -e "- done [ $MIN $SEC ].\n" >> $BKLOG
fi

As the comment block states, this is a monthly backup that occurs. It backs up the previous month's backups before starting one for the current day. This, combined with other routines put into the system keeps backups for a lengthy amount of time. This is also why we excluded /backups from our routine...WAY too many back ups of back ups if you ask me.

One thing I want to talk about, since its the meat of the actual back up routine, is this line:

tar -cpPzf /backups/$YEAR/$OLD.tar.gz /backups/$YEAR/$OLD/

This is basically telling tar to create (-c) a gunzipped (-z) back up file (-f) named /backups/$YEAR/$OLD.tar.gz containing the data found in /backups/$YEAR/$OLD/ directory, preserving permissions (-p), using absolute file names (-P) basically not stripping "/" from the beginning of the file name. The -P switch is used because it makes the output ugly, and it can lead to a broken back up.

Continuing on...

# Cycle through each drive and back up each
for d in "${DRIVE[@]}"; do
        echo "- Backing up drive $d" >> $BKLOG

        # By default, at least don't backup lost+found directories
        EX="--exclude=lost+found"

        # If we are backing up drive 1 (/dev/sda), there's to exclude
        if [ $d == "sda" ]; then
                for e in "${SDAEX[@]}"; do
                        EX="`echo -n $EX` --exclude=$e"
                done
        fi

        # Do the magic work and display some cool info
        SD=$( { time tar -cpPzf $BKDIR/$d.tar.gz $EX ${BACKUP[$ARRPOS]}; } 2>&1 )
        SD=`echo -n "$SD" | grep real`
        MIN=`echo -n "$SD" | awk '{printf substr($2,0,2)}'`
        SEC=`echo -n "$SD" | awk '{printf substr($2,3)}'`
        SD=$(ls -liha $BKDIR/$d.tar.gz)
        SIZE=`echo -n $SD | awk '{printf $6}'`

        let ARRPOS++
done

This is the code that does the backing up of current data. This is also where we need ARRPOS. This is all pretty much self explanatory as well to be honest. Biggest change here is, besides the array wrapped in a for block, we get the file size of the created back up file. So, lets re-wind a little bit here and look at the for block...

Another block of code I didn't discuss earlier (since its in a couple of spots) is this:

        SD=$( { time tar -cpPzf $BKDIR/$d.tar.gz $EX ${BACKUP[$ARRPOS]}; } 2>&1 )
        SD=`echo -n "$SD" | grep real`
        MIN=`echo -n "$SD" | awk '{printf substr($2,0,2)}'`
        SEC=`echo -n "$SD" | awk '{printf substr($2,3)}'`

If you run the time command, you'll get an output like this:

[ehansen@sfu ~]$ time

real	0m0.000s
user	0m0.000s
sys	0m0.000s

What the block of code does is measure how long it takes to create the back up file (tar command), and then we only measure the "real" time. The reason why this is done is because even in a multithreaded environment like Linux lets you have, a process may have to stop its thread for a moment to either let another user's program (or system action like signal handling) occur. The "real" time is the actual time it took for a command to execute. We use my best friend, Mr. awk, to parse the data from the information.

for d in "${DRIVE[@]}"; do

d will be whatever value is currently at $DRIVE[$ARRPOS]. For example, the first time around, d will be sda, second time it will be sdb.

        # By default, at least don't backup lost+found directories
        EX="--exclude=lost+found"

        # If we are backing up drive 1 (/dev/sda), there's to exclude
        if [ $d == "sda" ]; then
                for e in "${SDAEX[@]}"; do
                        EX="`echo -n $EX` --exclude=$e"
                done
        fi

ext3 & 4 file systems create this wonderful file called lost+found. Personally, I'm not a fan of it, because every time I try to restore corrupted data from it, I just get my bottom handed to me, but besides this, its a folder that really should be pointless to include in a routine back up measure. If we're working on the first partition (where /boot, /home, /var, etc... are stored), we exclude some of the more minor files that really mean nothing when the system is shut down. The reason why there's the line "EX="`echo -n $EX` --exclude=$e" is its basically the same as, for example, in PHP or Perl where you can do EX .= " --exclude=$e". Bash, however, is not as friendly with strings and concating.

# Mail this script out...ssmtp for GMail accounts, otherwise change for appropriate MTA
/usr/sbin/ssmtp -t < $BKLOG

This is the last of the back up script, just giving out some generic details, and then using ssmtp to send out the report. Nothing to really discuss here.

There you have it...a (surprisingly) simple back up solution for a server. Is it robust? Not really. A better implementation would be to do incremental daily with a full backup only once a week, and to also consider possible locked database files for example, but that's your homework. I encourage people to not steal other people's code, but to build upon it and also learn from it. What I provide will not be a solution for everyone, maybe anyone, but a lot of the techniques found here can be implemented in other languages even (such as PHP...which I converted this script to for fun).

As promised, here is the script, in full (with some poor comments in the header) that you can wget with: http://eric-hansen.info/old/scripts/script/backup.sh

[goodselfme]

There is a lot of good information in this post.thank you. How long do the entries remain on the site? There are times when I need to review things more than once.Thank you.

[Akendo]

That looks right nice, but i don't like tar as a backup solution. It makes backup to heavy and hard to time. But year, nice writing style and some nice script.

Can't test it now. But so far: "Nice Work!"
Thank for posting

so far
akendo

[ehansen]

There is a lot of good information in this post.thank you. How long do the entries remain on the site? There are times when I need to review things more than once.Thank you.

How to's stay on here for as long as the site is active as far as I know. Glad you found some useful information from this by the way.

[ehansen]

That looks right nice, but i don't like tar as a backup solution. It makes backup to heavy and hard to time. But year, nice writing style and some nice script.

Can't test it now. But so far: "Nice Work!"
Thank for posting

so far
akendo

Thanks Akendo. I used tar as it has some of the better compression and plus its essentially native to Linux these days. You're more than welcome to use a different compression set up though, its just my preference for tar.

[kInOzAwA]

http://eric-hansen.info/old/scripts/script/backup.sh link has been remove?

[ehansen]

http://eric-hansen.info/old/scripts/script/backup.sh link has been remove?

Ah, sorry about that. I'll have to fix that. Give it a day or so and I should hopefully at least have it hosted somewhere else.

[kInOzAwA]

nevermind.. nice script overthere dude

[Darwin]

This is more or less the method I have been using, took the dive in at the deep end which did need a few tweaks but overall what you state is a great way to backup for a noob like me. Nice and simple and relatively quick.

Shame the link is down...

[ehansen]

!/bin/bash

mkdir ~/lab3 && cd ~/lab3

sequence=`seq 9`
# could also do "sequence=[ 1 2 3 4 5 6 7 8 9 ] but `seq 9` evaluates to that,
#which is what the back-ticks are for
#I also pulled this out so you can change the number of files to any arbitrary number
# by changing the code int one place

for number in $sequence
do touch ./test${number}.txt
done

mkdir backup

for number in $sequence
do cp ./test${number}.txt ./backup/file${number}.back
done

Interesting script, I'm just not sure how this is a backup script. It'd be easier to create a tarball of a directory (i.e.: what I did) and just store it somewhere. Saves on disk space and you can easy extract a single file from a tarball.