Backup Strategies
FedoraForum.org - Fedora Support Forums and Community
Results 1 to 14 of 14
  1. #1
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    3,148
    Mentioned
    1 Post(s)
    Tagged
    1 Thread(s)

    Backup Strategies

    I was just reading Leslie's post about using hashes to remove duplicates from his backups.
    Rather than hijack a thread that started out about Timeshift, I thought I would start a new one.

    My approach is to first consider why I am making a backup.
    I can see several scenarios where having a backup would be useful:

    1/ Finger problems: This is where I accidentally rm something or overwrite something or edit the wrong file - you get the idea.

    2/ Hardware problems: This where the storage medium goes bad. Previously we often had some warning that a drive was getting a bit beyond its use-by date, but it remains to be seen how SSDs fail in the real world.

    3/ Property problems: This is where you get home and find the house has been emptied of its contents, or has been involved in a fire, flood, earthquake, ...

    As a programmer I would also need something for version control of my source code. This can also double as a backup system.

    Recovery of data needs to be an important consideration. For finger problems it needs to be quick and easy to get to - I don't want to be trying to find a file inside a 100GB tar archive. For hardware and property problems I don't want to be hunting around for some software that can actually read the archive - I will not be in any emotional state to clearly handle such problems.

    In the past I have used a binary tree of snap shots as my primary backup.
    The idea is that there are 5 directories containing one or two previous snap shots.
    Every second backup goes into dir-1. Every second one of the others goes into dir-2. Ditto for the dir-3 and dir-4, while dir-5 keeps an ongoing sequence of the remainder.
    This has the advantage of keeping access to a broad range of past times.

    I am now thinking that combing the binary tree with the techniques described in
    https://forums.fedoraforum.org/showt...25#post1824825
    might be a good approach.

    So I would propose using version control for code (and similar). Some stuff would be in public repositories, such as VICI on SourceForge, others would be on my own server.
    Then rsync (probably daily depending on how much work was done) for the home directory onto a NAS.
    Snapshots of the home directory, plus databases and repositories merged into a binary tree on a drive that is normally off-line.

    Finally a copy of the snapshot is dumped to a removable media and held remotely at a relative or friend's place (or a bank).
    Last edited by ocratato; 5th July 2019 at 03:29 PM.

    User error. Please replace user and try again

  2. #2
    Join Date
    Jun 2005
    Location
    Montreal, Que, Canada
    Posts
    6,081
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    Hi Ocratato.
    I do several backups Monthly or bi-weekly daily I rsync /scratch to /scratch1. My /development1 is part of /scratch and /scratch1 is an update of yesterday's /scratch.

    When I do test compiles, etc. I do so from /scratch and I do use git. Sometimes I occasionally want to pull back yesterday's stuff.
    almost all my $HOME directories are relocated to /scratch/leslie/
    I have a /usr/local/bin -->/scratch/usr/local/bin (my utilities)
    I have $HOME/bin-->/scratch/leslie/bin
    I have $HOME/documents -->/scratch/leslie/documents
    etc
    I use the following rsync script for daily backup
    Code:
    #!/bin/bash  
    # rsyncScratch/Scratch1
    SUDO=$(which sudo)
    
    function do_rsync()
    {
       local TO  From  wasmounted 
       From=$1  TO=${2}           
       wasmounted=3
    
       if [ -d ${TO}/bin ];
       then
          echo "${TO} was mounted"
          wasmounted=1
       else
          printf "%s\n%s\n" "Mounting ${TO}" "Do not Interrupt this script. It will restore ${TO} state on conclusion"
          ${SUDO} mount "$TO"
          wasmounted=0
       fi
    
       ${SUDO} chown -R leslie:leslie ${From}
       ${SUDO} chmod -R 755 ${From}
       echo "${SUDO} rsync -rauv ${From}/* $EXCLUDES  ${TO}"
       ${SUDO} rsync -rauv ${From}/*  $EXCLUDES     ${TO}
       if [  $wasmounted == 0 ];
       then
          printf "%s\n%s\n" "Restoring state of ${TO}" "Unmounting ${TO}"
          ${SUDO} umount  "${TO}"
          wasmounted=0
       else
         echo "${TO} remains mounted"
    fi
    }
    
    EXCLUDES="--exclude=lock --exclude=tmp "
    do_rsync /scratch /scratch1
    EXCLUDES="--exclude=lock --exclude=tmp --delete"
    do_rsync /scratch /scratch2
    echo "done"
    For my 3 external disks, I have another script that takes todays date and backs up /home and /scratch


    Why did I move my stuff to /scratch? I have 6 disks and I have 6 different distributions. The /scratch and /scratch1 are common to the six distributions. Softlinks are a wonderful Linux feature. most of my $HOME is relocated to /scratch In the worst case, my restore of files I am working on are no older than "yesterday's backup"

    To complete the daily backup info, here are my important crontab lines
    Code:
    SHELL=/bin/bash
    MAILTO=leslie
    PATH=$PATH:/sbin:/bin:/usr/local/bin:/usr/sbin
    LOGDIR='/scratch/log' 
    @reboot
                     sleep 90; /usr/local/bin/rsyncScratchScratch1.sh ; date > ${LOGDIR}/rsyncScratchScratch1.sda

    My archived backups go back 9 years to 2010. And yes, I sometimes refer back to them to answer the question "How did I do that xyz thing. Can I use it again...? "
    Leslie in Montreal

    Interesting web sites list
    http://forums.fedoraforum.org/showth...40#post1697840

  3. #3
    Join Date
    Dec 2011
    Posts
    82
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    My approach is to do local backups using rsnapshot (essentially a wrapper around rsync) and a remote backup to the cloud using a combination of duply (front-end to duplicity) and rsync.

    The local snapshots go to a second hard-disk. This helps against scenarios 1 and 2; it contains the last 14 daily backups, the 3 weeks before that and the 3 month before that. This gets you back approx 5 month with increasing gaps in time, see below for status today (no backup run yesterday as PC remained off):
    Code:
    >  ls -dltog *ly.*
    drwxr-x---. 3 4096 Jul  5 09:54 daily.0
    drwxr-x---. 3 4096 Jul  4 09:46 daily.1
    drwxr-x---. 3 4096 Jul  3 11:46 daily.2
    drwxr-x---. 3 4096 Jul  2 13:46 daily.3
    drwxr-x---. 3 4096 Jul  1 12:46 daily.4
    drwxr-x---. 3 4096 Jun 30 07:46 daily.5
    drwxr-x---. 3 4096 Jun 29 12:00 daily.6
    drwxr-x---. 3 4096 Jun 28 13:46 daily.7
    drwxr-x---. 3 4096 Jun 27 12:46 daily.8
    drwxr-x---. 3 4096 Jun 25 08:52 daily.9
    drwxr-x---. 3 4096 Jun 24 18:46 daily.10
    drwxr-x---. 3 4096 Jun 23 14:46 daily.11
    drwxr-x---. 3 4096 Jun 22 11:46 daily.12
    drwxr-x---. 3 4096 Jun 21 12:46 daily.13
    drwxr-x---. 3 4096 May 28 16:37 weekly.0
    drwxr-x---. 3 4096 May 17 10:47 weekly.1
    drwxr-x---. 3 4096 May 11 17:29 weekly.2
    drwxr-x---. 3 4096 Apr 18 21:34 monthly.0
    drwxr-x---. 3 4096 Mar 18 12:40 monthly.1
    drwxr-x---. 3 4096 Feb 20 12:55 monthly.2
    The backup to cloud storage covers me against mishaps in scenario 3. It is done with duply (front-end to duplicity) for data that I want encrypted before they leave my PC and rsync for data that can get copied unencrypted (and is thus readable to cloud provider). This rsync does not bother with versioning, only securing the last state. It also is called with --delete, i.e. deleting something on my PC also removes it immediately from this backup. Duplicity by design allows you to roll back several steps in time.

  4. #4
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    3,148
    Mentioned
    1 Post(s)
    Tagged
    1 Thread(s)

    Re: Backup Strategies

    Quote Originally Posted by pindakoe
    My approach is to do local backups using rsnapshot (essentially a wrapper around rsync) and a remote backup to the cloud using a combination of duply (front-end to duplicity) and rsync.

    The local snapshots go to a second hard-disk. This helps against scenarios 1 and 2; it contains the last 14 daily backups, the 3 weeks before that and the 3 month before that. This gets you back approx 5 month with increasing gaps in time, see below for status today (no backup run yesterday as PC remained off):
    ...
    The backup to cloud storage covers me against mishaps in scenario 3. It is done with duply (front-end to duplicity) for data that I want encrypted before they leave my PC and rsync for data that can get copied unencrypted (and is thus readable to cloud provider). This rsync does not bother with versioning, only securing the last state. It also is called with --delete, i.e. deleting something on my PC also removes it immediately from this backup. Duplicity by design allows you to roll back several steps in time.
    The difference between what you do and what I do highlights the need to tailor your backup strategy to your own circumstances.
    If you have a relatively small amount of data and it is changing frequently then daily (or even more often) are a reasonable approach. On the other hand if there is a lot of data and it is not changed much (apart from code which is held in a cloud repository) then weekly or even monthly snapshots would be sufficient.

    I think I still prefer to give a snapshot on an SD card to a friend over a cloud solution.
    Have you tested recovery from the cloud? Start by assuming that your house and all its contents are gone - can you still get your data back?

    User error. Please replace user and try again

  5. #5
    Join Date
    Nov 2004
    Location
    MT USA
    Posts
    1,038
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    Mine is a keep it simple strategy. Backup/Snapshot (whatever you want to call it) is a manual process using rsync. Absolutely no, notta, cloud backups... I want full control over my data. I also only save my data files, no operating system backups. For me, having a machine down for a couple of hours/day is not a problem here at home. All I need is to eventually be able to get my data back where it belongs.

    I have a 'data' server. All my other machines access the data stored on the server and it is really the only computer that needs backup support. Occasionally I'll rysnc the client computers home folder to the server when I feel I should. That takes care of the client machines. So now, what do I do with the server data.

    When important files are laid down on the server (say wedding pictures), I'll kick off a rsync session to a USB attached drive on the server. That way data is always in two places. Breathe easy at that point. Every couple of months, I'll start an rsync session to another 8Tb external drive that has four backup folders on it, that I simply 'rotate' over. At the end of the year, I buy a new drive and do a yearly backup and store it off-site. That is the general plan. However if we am going to be gone (say vacation for a week or two), I'll do a full backup on a spare drive and store off site before we go. Ie. The plan verifies depending on circumstances.

    As for programming I'll use git (local, not cloud), or just store the project files from the server, to the client's local disk temporarily as a 'backup' so again data is in two places.
    Last edited by rclark; 8th July 2019 at 04:28 AM.

  6. #6
    Join Date
    Dec 2011
    Posts
    82
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    Quote Originally Posted by ocratato
    The difference between what you do and what I do highlights the need to tailor your backup strategy to your own circumstances.
    If you have a relatively small amount of data and it is changing frequently then daily (or even more often) are a reasonable approach. On the other hand if there is a lot of data and it is not changed much (apart from code which is held in a cloud repository) then weekly or even monthly snapshots would be sufficient.

    I think I still prefer to give a snapshot on an SD card to a friend over a cloud solution.
    Have you tested recovery from the cloud? Start by assuming that your house and all its contents are gone - can you still get your data back?
    As to frequency: fully agree. Note that rsnapshot does not define a time interval of backups, just multiple levels. You can also (with the appropriate cron or systemd-timer jobs) define the backup frequencies more or less frequent. Check up on the 'retain' parameter in its or for example here where they talk about alpha, beta and gamma bakups to avoid any preconceived notion on time resolution

    And yes ... I did a restore from cloud after one of my (software) experiments went a step too far and wiped out part of my normal storage and first line backup (the one with the snapshots). That worked (as I had taken the precaution to have a copy of the signing key that duplicity uses for encryption at a separate off-site storage).

  7. #7
    Join Date
    Jun 2005
    Location
    Montreal, Que, Canada
    Posts
    6,081
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    Well, instead of reinventing the wheel, I discovered plasma' Kup backup program

    It uses some logic (checksums for version management or rsync for image backups
    It can be trained to do backups based on hours of uptime (40 hrs), or crontab or manual initiation. Backup won't start unless the backup is mounted.
    I am going for version management.

    https://www.youtube.com/watch?v=y53IE0G8Brg&t=178s

    This is a correction to the android doing it's spelling corruption algorithm.
    I include the youtube link which explains it's use.

    You may also want to explore bup I believe that Kup drives bup.
    Last edited by lsatenstein; 13th July 2019 at 06:50 AM.
    Leslie in Montreal

    Interesting web sites list
    http://forums.fedoraforum.org/showth...40#post1697840

  8. #8
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    3,148
    Mentioned
    1 Post(s)
    Tagged
    1 Thread(s)

    Re: Backup Strategies

    Quote Originally Posted by lsatenstein
    Well, instead of reinventing the wheel, I discovered plasma' lip backup program
    Any links so we can have a look?

    User error. Please replace user and try again

  9. #9
    Join Date
    Nov 2004
    Location
    MT USA
    Posts
    1,038
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    I did a search and didn't see anything called "plasma' lip backup program" either. Looked on my Ubuntu KDE system and all I find is Timeshift.

  10. #10
    Join Date
    Jun 2005
    Location
    Montreal, Que, Canada
    Posts
    6,081
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    Well, instead of reinventing the wheel, I discovered plasma' Kup backup program

    It uses some logic (checksums for version management or rsync for image backups
    It can be trained to do backups based on hours of uptime (40 hrs), or crontab or manual initiation. Backup won't start unless the backup is mounted.
    I am going for version management.

    https://www.youtube.com/watch?v=y53IE0G8Brg&t=178s
    Leslie in Montreal

    Interesting web sites list
    http://forums.fedoraforum.org/showth...40#post1697840

  11. #11
    Join Date
    Oct 2010
    Location
    Canberra
    Posts
    3,148
    Mentioned
    1 Post(s)
    Tagged
    1 Thread(s)

    Re: Backup Strategies

    Here is a link to some real info, rather than a video

    https://www.linux-apps.com/content/s...content=147465

    User error. Please replace user and try again

  12. #12
    Join Date
    Jun 2005
    Location
    Montreal, Que, Canada
    Posts
    6,081
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    I downloaded and installed timeshift. I tried it, and it does not do the job for me. It is ok if you only want to backup a /dev/sda
    The following is my disk layout and may explain why timeshift is inappropriate.

    To understand why, I have a six disk system with large drives. Configured as follows:
    Code:
    /dev/sda Gnome/home/leslie-------------------->/scratch/leslie  used for Gnome
    /dev/sdb  Centos/home/leslie ----------------->/scratch/leslie  For Older C compiler
    /dev/sdc  Tumbleweed/home/leslie-------------->/scratch/leslie  For KDE
    /dev/sdd  Deepin/Elementary/home/leslie------->/scratch/leslie  Using Fedora30
    /dev/sde  MX-Linux/home/leslie---------------->/scratch/leslie   test of other distros
    
    /dev/sda8 /scratch    is a partition that is  common to all distributions.

    Timeshift does not appear to offer multi-disk target to one backup external disk.

    Further information
    ============
    I have a dedicated 1 terrabyte drive, within a partition titled /scratch.
    I have a /scratch2 partition which is a merged backup of /scratch
    I have a /scratch1 partition which is a nightly mirror of /scratch (actually, mirrored at reboot or on demand)

    And I have three external terrabyte backup drives.
    Why so many backups?

    I write C code, I write scripts and some C++ for myself to handle a large variety of Linux stuff that is of interest to me. I also test the programs on the above list of distros. The following is a list of my $HOME folder as it is on each distro.
    What I do not show are the individual hidden files. These cannot be shared across distros. eg .cache, .local, .mozilla, .profile ... etc.

    Code:
    lrwxrwxrwx. 1 leslie leslie     19 Nov 29  2018  bin -> /scratch/leslie/bin
    lrwxrwxrwx. 1 leslie leslie     31 Nov 29  2018 'Calibre Library' -> '/scratch/leslie/Calibre Library'
    lrwxrwxrwx. 1 leslie leslie     23 Nov 29  2018  Desktop -> /scratch/leslie/Desktop
    lrwxrwxrwx. 1 leslie leslie     14 Nov 29  2018  Development -> /scratch/Devel
    lrwxrwxrwx. 1 leslie leslie     25 Nov 29  2018  Documents -> /scratch/leslie/Documents
    lrwxrwxrwx. 1 leslie leslie     25 Nov 29  2018  Downloads -> /scratch/leslie/Downloads
    drwx------. 6 leslie leslie   4096 May 16 21:30  Dropbox
    -rw-r--r--. 1 root   root      936 May  4 12:10  my-gdbus.pp
    -rw-r--r--. 1 root   root      166 May  4 12:10  my-gdbus.te
    lrwxrwxrwx. 1 leslie leslie     24 Nov 29  2018  Pictures -> /scratch/leslie/Pictures
    lrwxrwxrwx. 1 leslie leslie     22 Nov 29  2018  Public -> /scratch/leslie/Public
    lrwxrwxrwx. 1 leslie leslie      8 Apr 22 11:28  scratch -> /scratch
    -rwxr-xr-x. 1 leslie leslie   2076 Jan 24 23:34  taskbar.dconf
    lrwxrwxrwx. 1 leslie leslie     25 Nov 29  2018  Templates -> /scratch/leslie/Templates
    lrwxrwxrwx. 1 leslie leslie     22 Nov 29  2018  Videos -> /scratch/leslie/Videos
    One system folder is also shared.
    /usr/local/bin-->/scratch/Devel/LinuxStuff/usr/local/bin
    Any object that can be common between the distros is moved to /scratch/LinuxStuff.

    Ocratato indicated in a penultimate post that Kup (not in Fedora) appears to be a good backup as us bup

    So far, kup is a utility for kernel patch uploads and the name distinction between Kup and kup is the first letter.
    So, I am continuing to do rsync backups, onto my external backup drives until I am forced to delete some of the oldest backups.
    Leslie in Montreal

    Interesting web sites list
    http://forums.fedoraforum.org/showth...40#post1697840

  13. #13
    Join Date
    Jun 2004
    Location
    Maryland, US
    Posts
    7,624
    Mentioned
    3 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    I did a search and didn't see anything called "plasma' lip backup program" either. Looked on my Ubuntu KDE system and all I find is Timeshift. - rclark
    Apparently it's not in the normal Fedora dnf repos, and confusingly there is a package called "kup" which is some kind of kernel uploader

    Code:
    kup - kernel.org upload utility
    This  utility is used to upload files to kernel.org and other systems using the same upload system (kup-server).
    that is not related to this Plasma "kup" backup tool at all. You have to get KDE "kup" backup tool here

    https://github.com/spersson/Kup/
    or
    https://www.linux-apps.com/content/s...65#files-panel

  14. #14
    Join Date
    Jun 2005
    Location
    Montreal, Que, Canada
    Posts
    6,081
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)

    Re: Backup Strategies

    The Plasma one is spelled Kup, the kernel backup is spelled kup. The difference is the capital K in the first letter.

    Its using Python 2, and that means that it is incompatible with Fedora 30. As far as I know, its for Ubuntu (perhaps Debian or arch derivatives)
    Leslie in Montreal

    Interesting web sites list
    http://forums.fedoraforum.org/showth...40#post1697840

Similar Threads

  1. GNOME: Seven Possible Recovery Strategies
    By Wayne in forum Linux Chat
    Replies: 18
    Last Post: 23rd August 2012, 09:26 AM
  2. Jiggery Pokery - strategies for modifying fedora installs
    By mikeym in forum Installation, Upgrades and Live Media
    Replies: 0
    Last Post: 18th January 2012, 01:15 PM
  3. Partitioning Strategies
    By snakeyes30 in forum Installation, Upgrades and Live Media
    Replies: 1
    Last Post: 14th September 2006, 03:34 AM
  4. Hard Disk Partitioning Strategies
    By BandC in forum Installation, Upgrades and Live Media
    Replies: 10
    Last Post: 1st March 2006, 05:55 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •