Help with script for easily (and hopefully automatically) moving files around

[Thetargos]

Hi all!

I am trying to create a script that I can place under a cron tab for automating this task: I need to be able to copy a series of files periodically, here is the trick, I need to only copy those files from the source that are not in the destination based either on filename or preferrably by creation/access date. Since these are files used in a database (both locally and remotely, and the destination is an archive, part of another DB), filenames are cryptic and contained within equally cryptic named directories. So I need to figure out how to do this the most efficient way.

I thought of getting the dates on the source and destination of the directories/files and then copying only those "newer" to avoid overhead of copying files already present at the destination... Since some new files may be created within the directories of the source directory, I need to figure out a way to be able to also copy those files to the destination to keep them in sync.

For periodicity, I only need to run this once a day, for which I'll use cron. However, I'm debating on what the best way to handle file dates would be best, perl, bash or other?

Personally I'm more familiar with Bash, but I've only done some rather simple scripting with it.

[ikamusume]

Originally Posted by Thetargos

Hi all!

I am trying to create a script that I can place under a cron tab for automating this task: I need to be able to copy a series of files periodically, here is the trick, I need to only copy those files from the source that are not in the destination based either on filename or preferrably by creation/access date. Since these are files used in a database (both locally and remotely, and the destination is an archive, part of another DB), filenames are cryptic and contained within equally cryptic named directories. So I need to figure out how to do this the most efficient way.

This sounds like a job for rsync. You should check whether or not rsync solves the problem before writing a script, but I think it will.

[glennzo]

I have something similar on my server, run as a cron job. It uses rsync and updates files on a couple other Linux boxes, so if the master copy is newer than the copy on the target computers the master copy gets pushed to the target computers. I wrote this with the help of some Fedora Forum members.

PHP Code:

# Variables
include_file="/script/junk_yard/includes"                            # Script function include file
color_file="/script/junk_yard/scolors"                               # Script color include file
backup_file="/script/test_backup.sh"                                 # Backup script
target_path="/script"                                                # Target path
test_data="/script/the_test_file"

function push_files()
{
rsync -vv -u $backup_file rsync@$target_addr:$target_path/test_backup.sh
rsync -vv -u $include_file rsync@$target_addr:$target_path/includes
rsync -vv -u $color_file rsync@$target_addr:$target_path/scolors
rsync -vv -u $test_data rsync@$target_addr:$target_path/test_data
}

for i in 2 8 18                                                      # IP addresses
do                                                                   # script-test, goodoldbob, enterprise
   target_addr=10.0.0.$i
   push_files
done
logger "Push test script has run." 


So in this script there are three files that I want to push to three other computers if they are newer than the copies on the target computers. The script loops through three ip addresses. If those boxes are up and the files are newer they get copied.

I've made changes to the master file just to test the script. It works.

If something like this won't work for your purpose then hit Google. I've found several methods for getting the data/time from a file. These methods could be applied to your script.

[Thetargos]

Thanks glennzo!

The situation is a bit of a mess for me, though. The destination directory for these files is actually a locally mounted drive, which is also mounted over the network remotely as a network drive on a Windows machine. The peculiarity of this setup, (not my decision, though) is that this drive is actually an external Desktop drive attached via FireWire 800 to the server, and formatted as NTFS (supposedly to have the ability to attach it to some Windows machines, even when the drive seldom moves from the server, and when it did was to a Mac ).

Anyway the structure of the source directory is as follows:

$ pwd
/home/user/.appname/DATABASE.noindex/
$ ll
drwxrwxr-x.  140 user user 4096 sep  2 17:29 00eeff54
drwxrwxr-x. 2240 user user 4096 sep  4 21:41 02e5d2ff
drwxrwxr-x.  740 user user 4096 sep  9 19:40 12543e44
drwxrwxr-x.   64 user user 4096 sep 15 12:41 30dd5432
drwxrwxr-x.   70 user user 4096 sep 18 13:41 54a2bb48
drwxrwxr-x.  220 user user 4096 sep 21 09:42 cab21092
drwxrwxr-x. 1560 user user 4096 sep  5 07:42 d5704f20
drwxrwxr-x.  665 user user 4096 sep  9 15:29 ff003345
-- Many More --

I need to copy these to a directory on the NTFS drive:

$ pwd
/run/media/user/WinBack/appname/INCOMING.noindex

The basic idea is that the application we are using uses that DATABSE.noindex directory as the storage directory for the files indexed in its own DB (sqlite); The INCOMING.noidendex directory is scanned by the application to then index the files as they arrive and then moves them to the DATABASE.noindex as it does so.

So, I need to be able to copy the files of the local machine to the also local drive using the access/creation time... The destination is intended to be an archive of the original, despite the local files being deleted and created all the time, which is why the script must be run at least once a day, and why I wanted to make it a cron job. I know little of other programming functions other than Bash, and am still looking over my Bash book on how could I solve this... Indeed rsync seems to be what I need... I'll run some tests with my own external wallet drive and some files I've got and see how it goes.

[Thetargos]

Well, rsync works really good. I ended up writing a simple yet effecient script for my needs:

#!/bin/sh

DestPath="$BackupPath"
SrcPath="$LocalPath"

if [ -d "$DestPath" ] && [ -d "$DestPath/Logs" ]
then
      rsync -Cavz "$SrcPath" "$DestPath" > "$DestPath/Logs/Backup - $(date +'%a %b %d %Y')"
else
     mkdir -p "$DestPath/Logs"
     rsync -Cavz "$SrcPath" "$DestPath" > "$DestPath/Logs/Backup - $(date +'%a %b %d %Y')"
fi

So far it works fine for a cron job with a rudimentary logging of the job

[glennzo]

Great! Now all you may need to do is tweak it a little bit, maybe not.