PDA

View Full Version : Easiest way to backup data



stratacast1
2nd December 2014, 03:08 AM
What's the most efficient way of backing up data? I was thinking maybe rsync because that just updates the files that were changed right? I'd prefer a method like that vs replacing the entire file/directory. Any suggestions? I'm not afraid of doing it in terminal

solo2101
2nd December 2014, 03:20 AM
Easiest way?
Copy the files....

Efficiency?
Depends, are you copying over a slow network?

rclark
2nd December 2014, 04:26 AM
rsync is what I use. The first time takes awhile if you have terabytes of data, but after that only the files that are changed/deleted/added are updated to the backup system/drive. All I use from the command-line. I formatted my backup drive ext4 so none of the permission stuff is lost when transferring files. I don't backup the OS folders, just the data/user folders. OS can always be restored if necessary by re-installing. Also setup a chron job to do it automatically once a night. Aproximately, once a month I'll backup to another drive to have a monthly backup incase something drastic occurs that wipes out files on both backup and server. Of course there a yearly backup to go back to that is off site. Not a perfect setup or interprise ready, but works for me.

sea
2nd December 2014, 04:55 AM
Easiest:

cd
tar acf $HOME/backup.tar.gz --one-file-system --exclude=Downloads --exclude=.cache * .[a-zA-Z]*

I use rsync too: 'hardcopy'

#!/bin/sh

# Probleme mit NTFS
# rsync .... --no-perms --no-owner --no-group ...

# Backups nachträglich 'verifizieren'/check if all good
# rsync ... -c ...

usr=sea
quelle=/home/$usr
ziel=/media/backups/$(hostname)
heute=$(date +%F)

# Change to source and set ownage to user
cd "$quelle"
#sudo chown "$usr:$usr" {.[a-zA-Z]*,*} -R # This might cause verbose error messages
[[ -d "$ziel" ]] || mkdir -p "$ziel"
umask 077 "$ziel"

# Delete tempfiles
#rm -fr {.c,c}ache/*

# Sync meine files zum NAS, ohne binds/symlinks
printf "\nStart sync...\n"

time rsync -aR \
--progress \
--no-owner --no-group \
"${quelle}" "${ziel}/" \
--exclude=/home/$usr/priv/backups \
--exclude=/home/$usr/priv/cloud \
--exclude=/home/$usr/net/dls \
--exclude=/home/$usr/net/pub \
--exclude=/home/$usr/{mm,media}/{sn{d,ds},vi{d,ds}}/[wW]in* \
--exclude=/home/$usr/notepad/.ignore \
--exclude=/home/$usr/.cache

cd "$OLDPWD"
echo
echo "Done."


Hope this helps

stratacast1
2nd December 2014, 05:41 AM
I'm backing up through USB, nothing fancy over networks or terabytes of data...maybe 300GB-500GB of data. Just doing home backups :) I'd kind of like to just run a script rather than answer to all the prompts and such. My plan is to just hook up my drive maybe once every week or 2 and run a script to backup select directories. Specifically:
~/Documents
~/Pictures
~/Videos
*Other stuff in home*
/steam/SteamLibrary/steamapps/common/*

Along with a couple of files in ~/.local

I should probably watch some videos on this too
Maybe something like:

rsync -u ~/Documents /media/courtney/Seagate/Linux_backup/Documents

Should I do -p for preserving permissions? And I see that -u adds the new files, what's the option to update files that have been changed, like if I made changes to a document?

stevea
2nd December 2014, 07:44 AM
rdiff-backup

rsync makes a nice efficient COPY/MIRROR, not a backup.
That's probably OK if you are just copying collections that are never modified (and hopefully never re-wriiten or modified by accident) like a collection of photos or music.
It's not OK for files that are regularly modified.

Here is the problem .....


# monday - write brilliant speech
[stevea@mycena Desktop]$ mkdir from to
[stevea@mycena Desktop]$ touch from/{a,b,c}
[stevea@mycena Desktop]$ echo "Four score and ..." > from/d

# monday night script creates a duplicate/mirror/copy
rsync -ar from/ to

# tuesday morning make bad editing mistake
[stevea@mycena Desktop]$ echo 4 > from/d


# tuesday night 'COPY' completes the destruction of all the effort.
[stevea@mycena Desktop]$ rsync -ar from/ to
[stevea@mycena Desktop]$ ls -l to
total 4
-rw-rw-r--. 1 stevea stevea 0 Dec 2 01:01 a
-rw-rw-r--. 1 stevea stevea 0 Dec 2 01:01 b
-rw-rw-r--. 1 stevea stevea 0 Dec 2 01:01 c
-rw-rw-r--. 1 stevea stevea 2 Dec 2 01:06 d

# wednesday - recognize the error and find someone to blame & a stiff drink.


Yes you could use rsync --backup .... but managing these backups is awkward.



Alternative:

# monday - write brilliant speech
[stevea@mycena Desktop]$ mkdir from to
[stevea@mycena Desktop]$ touch from/{a,b,c}
[stevea@mycena Desktop]$ echo "Four score and ..." > from/d

# monday make an incremental BACKUP
[stevea@mycena Desktop]$ rdiff-backup from to

# tuesday make a dumb mistake, then another backup
[stevea@mycena Desktop]$ sed -i s/Four/Seven/ from/d
[stevea@mycena Desktop]$ rdiff-backup from to

# wednesday - recognize the error and revert to the older copy
[stevea@mycena Desktop]$ cat from/d
Seven score and ...
# Uh oh .... let's restore from the incremental backup ....
[stevea@mycena Desktop]$ rdiff-backup -l to
Found 1 increments:
increments.2014-12-02T01:19:57-05:00.dir Tue Dec 2 01:19:57 2014
Current mirror: Tue Dec 2 01:21:24 2014

[stevea@mycena Desktop]$ rdiff-backup --force --restore-as-of "2014-12-02T01:19:57-05:00" to/d from/d
[stevea@mycena Desktop]$ cat from/d
Four score and ...

The last backup is kept as files under "to/" and the older stuff as gzip'ed diff file off the current. File metadata is also stored & restored.

You have a lot of latitude in the restore date description, like "--restore-as-of 10D" for the data as of ten days ago.


================
rdiff-backup has very similar syntax as rsync, and can make backups to/from remote systems given a transport mechanism, like ssh.
'safekeep' is a decent config wrapper for rdiff-backup.

sidebrnz
2nd December 2014, 08:11 AM
There are a number of backup programs available that are basically a convenient front-end for rsync; just pick the one that feels right to you. The one I use is backintime, and I put my backups on a flash drive that's been reformatted to ext4 so that my backups can take advantage of hard links to represent files that haven't changed since the last backup. Right now, I have about 35GB of data and 8GB of free space on a 16GB drive because most of what I'm backing up rarely changes. I'm fairly sure that most, if not all of the rsync front-ends can do this, if saving space is an issue, so again, pick whichever one seems right for you and you won't have to worry about writing, debugging and maintaining backup scripts on your own.

stevea
2nd December 2014, 10:24 AM
There are a number of backup programs available that are basically a convenient front-end for rsync; just pick the one that feels right to you. The one I use is backintime, and I put my backups on a flash drive that's been reformatted to ext4 so that my backups can take advantage of hard links to represent files that haven't changed since the last backup. Right now, I have about 35GB of data and 8GB of free space on a 16GB drive because most of what I'm backing up rarely changes. I'm fairly sure that most, if not all of the rsync front-ends can do this, if saving space is an issue, so again, pick whichever one seems right for you and you won't have to worry about writing, debugging and maintaining backup scripts on your own.

Saving space is the primary issue wrt backups, otherwise we'd just make a hourly complete copy of everything for decade-long periods.

backintime appears to be a competent tool that makes differential backups by keeping full uncompressed file of changed files, and appears to hide the relatively ugly rsync syntax. That's perfectly fine as far as functionality is concerned, except those uncompressed full copies take up a lot more disk space than compressed diffs.

As for tools ...
duplicity, deja-dup, duply & safekeep, rdiff-backup, are incrementals based in the librsync.
backintime uses the rsync command.
Amanda, BackupPC do incrementals, pretty old school.
grsync and luckybackup - look like thin wrappers on rsync.

Others are ...
backup-manager, bacula, backup-light, bontmia, dar, dirvish, fwbackups, obnam and I certainly missed some.


I still think I'd tend toward ... deja-dup, duply or safekeep.

wintonian
2nd December 2014, 10:53 AM
Probably lots of things wrong with this but...

Periodically clone /home (which is on a separate drive) and in between clones copy and paste any files/ directory that have changed. Plus I have at least 1 image of my system drive.

Simple and the home drive can just be swapped out in case of failure.

Works if it's mainly personal files and a couple of hidden files that you have customised. I wouldn't use such a haphazard method in an enterprise situation though.

As for rsync luckybackup seems quite a good one to me, but take what ever frontend works for you.

glennzo
2nd December 2014, 12:18 PM
With plenty of disk space available I just keep 90 days of daily .tar backups for any data that I hold near and dear. Seems simple enough to me.

A script to create the backup with the date in the file name.
A script to delete backups older than 90 days.

Tested and working per my specifications.

Both triggered by cron.

pete_1967
3rd December 2014, 12:02 AM
Been running BackupPC for multiple hosts, including laptops with weekly full and daily incremental backups for years now. For my purposes it is perfect (Bacula is too heavy weight). Restoring the data from it is simple and reliable, you can either let BackupPC to do it or download Zip or Tar file, or even directly from its own storage with internal utility and extract yourself. I do push weeklies to NAS drive where I keep them for 3 months.

Only setback is that, depending on what you want to backup, because it's using pull as opposed to push, it requires either root user privileges or other user that can fetch files & restore them (not strictly necessary). I am using ssh but it has plethora of methods from Samba to ftp to suit every need.