No more worrying about data loss: use rsync to build your automated backup solution

In modern IT environments, data backup is a vital task. Whether it is personal files or corporate data, a reliable backup mechanism is needed to prevent data loss. Today, we will introduce an efficient backup solution: using rsync to automate the backup directory.

What is rsync?

rsync is an open source, fast, and universal file synchronization tool. Its main function is to synchronize files and directories from one location to another, and only transfer the changed parts, which greatly improves the transmission efficiency. It supports local replication and remote replication through SSH, rsync protocol, etc.

For more information about rsync, please refer to the official documentation: https://rsync.samba.org/

Why choose rsync?

• Efficiency: rsync only transfers the changed parts, which greatly reduces the amount of data transferred.
• Flexibility: Backup can be performed locally and remotely, supporting multiple protocols.
• Reliability: rsync maintains file attributes, permissions, and timestamps to ensure the integrity of the backup.
• Incremental backup: Incremental backup is achieved through symbolic links, saving storage space.

Automated backup scripts

The following is a shell script that uses rsync to implement automated backup. It backs up the specified directory and compresses the old backup into an archive.

Script Details

(1) Check and install rsync: Check whether rsync is installed in the system. If not, install it according to the system type.

 if ! command -v rsync &> /dev/null; then echo "rsync 未安装。正在安装..." if [[ "$OSTYPE" == "linux-gnu"* ]]; then if [ -x "$(command -v apt-get)" ]; then sudo apt-get update && sudo apt-get install -y rsync elif [ -x "$(command -v yum)" ]; then sudo yum install -y rsync elif [ -x "$(command -v dnf)" ]; then sudo dnf install -y rsync else echo "无法检测到合适的包管理器，请手动安装rsync。" exit 1 fi elif [[ "$OSTYPE" == "darwin"* ]]; then if ! command -v brew &> /dev/null; then echo "Homebrew 未安装。请先安装Homebrew。" exit 1 fi brew install rsync else echo "不支持的操作系统，请手动安装rsync。" exit 1 fi echo "rsync 安装完成。" else echo "rsync 已安装。" fi

(2) Get the current time and yesterday's date: used to name the backup directory and archive files.

 NOW=$(date +%Y%m%d%H%M) YESTERDAY=$(date -d "yesterday" +%Y%m%d)

(3) Configure the directory where the backup system is stored: define the storage location of backup files, snapshots, and archives.

 BACKUP_HOME="/srv/backups" CURRENT_LINK="$BACKUP_HOME/current" SNAPSHOT_DIR="$BACKUP_HOME/snapshots" ARCHIVES_DIR="$BACKUP_HOME/archives"

(4) File directory to be backed up: Specify the source file directory to be backed up.

 BACKUP_SOURCE_DIR="/etc"

(5) Create a directory for storing backup files: Make sure the directory for storing backup files exists. If it does not exist, create it.

 mkdir -p "$SNAPSHOT_DIR" "$ARCHIVES_DIR" &> /dev/null

(6) Use rsync for backup: Use the rsync command to perform incremental backup and update the symbolic link of the current backup.

 rsync -azH --link-dest="$CURRENT_LINK" "$BACKUP_SOURCE_DIR" "$SNAPSHOT_DIR/$NOW" \ && ln -snf "$(ls -1d "$SNAPSHOT_DIR"/* | tail -n 1)" "$CURRENT_LINK"

The rsync command uses the -a option (archive mode), the -z option (compressed files), the -H option (keep hard links), and the --link-dest option (incremental backup using symbolic links). After the backup is complete, the CURRENT_LINK symbolic link is updated to point to the latest backup directory.

(7) Archiving: Check whether there is a backup from yesterday. If so, compress it and archive it, and delete the original backup file.

 if [ $(ls -d "$SNAPSHOT_DIR"/"$YESTERDAY"* 2> /dev/null | wc -l) -ne 0 ]; then tar -czf "$ARCHIVES_DIR/$YESTERDAY.tar.gz" "$SNAPSHOT_DIR/$YESTERDAY"* \ && rm -rf "$SNAPSHOT_DIR/$YESTERDAY"* fi

How to use

• Create a script file: Save the above script as rsync.sh.
• Grant execution permission: Run chmod +x rsync.sh in the terminal.
• Execute the script file: ./rsyn.sh, and enter the directory to be backed up according to the prompts, as shown in the figure below:

Modify the system time and execute the script again, as shown in the following figure:

From the above results, we can see that yesterday's backup has been compressed and archived.

Summarize

By using rsync and a simple shell script, we can easily implement an automated backup system. This solution is not only efficient and flexible, but also ensures data integrity and security. I hope this article can help you better understand the use of rsync and effectively apply it in actual work.

How to obtain the script

The above scripts have been uploaded to gitee. You can get them if you need them. The repository on gitee mainly shares some commonly used scripts in work. You can frok or watch the repository so that you can pay attention to updates in time.

Script Repository

Warehouse address: https://gitee.com/didiplus/script