FedoraForum.org - Fedora Support Forums and Community
Results 1 to 5 of 5
  1. #1
    Join Date
    Mar 2009
    Location
    Lake Hughes, CA
    Age
    42
    Posts
    14

    Example, bash scrip to check remote server up status via ssh and alert

    I have several large NAS's with mirrored backups. The problem was, as long as the main servers are up the backups may go unnoticed and could go down with out anyone knowing. So I came up with a little bash script to run via cron twice a day to check network status, ssh, and raid volume access. I cron rsync over ssh to backup the main servers so testing the ssh connectivity was important. If any any problems are found the main server will issue an email alert. I spent a few hours searching and many hours experimenting so hopefully posting this here can help someone else along.

    Code:
    #! /bin/bash
    #
    #Check network to mxnas2 from mxnas1 via ssh
    #Check file access to Raid volume 'mxnas2'
    #Email alert if exit status is not 0
    #
    /usr/bin/ssh -n 10.x.x.x test -f /mxnas2/.mxnas_up
    exstat=$?
    if [ $exstat -ne 0 ]; then
      if [ $exstat -eq 1 ]; then
      /bin/echo "The MXNAS2 server appears to be up but the /mxnas2/.mxnas2_up file can not be found $(date)" > /tmp/mxnas2_stat_report.log
      /bin/mail -s "WARNING-MXNAS2 file system check error" damon.stewart@x.com -c tvcvtrxsan@x.com < /tmp/mxnas2_stat_report.log -- -f "mxnas1@x.com"
        elif [ $exstat -eq 255 ]; then
        /bin/echo "Can not SSH to MXNAS2 $(date)" > /tmp/mxnas2_stat_report.log
        /bin/mail -s "WARNING-MXNAS2 network check error" damon.stewart@x.com -c tvcvtrxsan@x.com < /tmp/mxnas2_stat_report.log -- -f "mxnas1@x.com"
          else 
          /bin/echo "System status check to MXNAS2 with exit status code $exstat $(date)" > /tmp/mxnas2_stat_report.log
          /bin/mail -s "WARNING-MXNAS2 unknown system check error" damon.stewart@x.com -c tvcvtrxsan@x.com < /tmp/mxnas2_stat_report.log -- -f "mxnas1@x.com"
      fi
    fi
    #Uncomment the following line to report exit status for testing
    #echo "exit status to /mxnas2/ = $exstat"
    #
    #djs/js v1.2 2010-12-09
    Just as a note, Im using an rsa key between the main and backup servers to allow the ssh login for both the rsync and status check. I also configured sendmail MTA as a client and added our smtprelay to the submit.mc and submit.cf. Im sure there are many other ways, maybe better, but this works well.

  2. #2
    Join Date
    May 2009
    Posts
    32

    Re: Example, bash scrip to check remote server up status via ssh and alert

    Surely a much easier way would be

    Code:
    exec 3<> /dev/tcp/host/port
    For Example

    Code:
    [affix@fedora Desktop]$ exec 3<> /dev/tcp/127.0.0.1/22
    [affix@fedora Desktop]$ exec 3<> /dev/tcp/127.0.0.1/2222
    bash: connect: Connection refused
    bash: /dev/tcp/127.0.0.1/2222: Connection refused
    You will see that it doesn't return anything if the host port is open. If not the connection will be refused

  3. #3
    Join Date
    Mar 2009
    Location
    Lake Hughes, CA
    Age
    42
    Posts
    14

    Re: Example, bash scrip to check remote server up status via ssh and alert

    If all you wanted to do is check for a network to port 22, yes. I needed to verify the raid mount and email status on errors.

  4. #4
    Join Date
    Feb 2007
    Posts
    5

    Re: Example, bash scrip to check remote server up status via ssh and alert

    If you also need to to check the raid status and look at mail you could also use expect to automate the logins. But what you are doing works.

  5. #5
    Join Date
    Mar 2009
    Location
    Lake Hughes, CA
    Age
    42
    Posts
    14

    Re: Example, bash scrip to check remote server up status via ssh and alert

    I've cleaned it up and added to to it some since. One of these days I'll play with 'expect'.

    rsync will only run if ssh is available to the backup server and the destination raid volume is mounted. If no errors only a log is created. On any error either from the server up test or from rsync, I will get an email with the exit status code. If the error is from rsync I included an exit status legend.

    Code:
    [root@tvc-mxnas1 bin]# cat promo_stat_and_sync.sh 
    #!/bin/bash
    #
    #Check network to mxnas2 from mxnas1 via ssh
    #Check file access to Raid volume,'.mxnas_up'
    #rsync /mxnas1/ to /mxnas2/ over ssh
    #Email if exit status is not 0
    #
    ssh -n 10.102.63.32 test -f /mxnas2/.mxnas_up
    teststat=$?
    #
    error1="MXNAS1 to MXNAS2 Sync did not occur at $(date) do to the following error;"
    error2="The MXNAS2 server appears to be up but the /mxnas2/.mxnas2_up file can not be found"
    error3="No ssh path to MXNAS2"
    error4="System status check to MXNAS2 with unknown error, exit status code $teststat"
    subject1="WARNING-MXNAS2 System check error"
    subject2="ERROR-MXNAS1 to MXNAS2 RSYNC FAULT"
    rsync_exit_stat="rsync exit status code legend;
    0\tSuccess
    1\tSyntax or usage error
    2\tProtocol incompatibility
    3\tErrors selecting input/output files, dirs
    4\tRequested  action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.
    5\tError starting client-server protocol
    6\tDaemon unable to append to log-file
    10\tError in socket I/O
    11\tError in file I/O
    12\tError in rsync protocol data stream
    13\tErrors with program diagnostics
    14\tError in IPC code
    20\tReceived SIGUSR1 or SIGINT
    21\tSome error returned by waitpid()
    22\tError allocating core memory buffers
    23\tPartial transfer due to error
    24\tPartial transfer due to vanished source files
    25\tThe --max-delete limit stopped deletions
    30\tTimeout in data send/receive"
    #
    if [ $teststat -eq 0 ]; then
       rsync -a --delete /mxnas1/ 10.102.63.32:/mxnas2/
       syncstat=$?
       echo -e "MXNAS1 to MXNAS2, RSYNC last ran at $(date) with an exit status code of;\n/mxnas1/=$syncstat\n\n$rsync_exit_stat" > /var/log/mxnas2_stat_report.log
            if [ $syncstat -ne 0 ]; then
               mail -s "$subject2" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
            fi
       elif [ $teststat -eq 1 ]; then
            echo -e "$error1\n$error2" > /var/log/mxnas2_stat_report.log
            mail -s "$subject1" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
       elif [ $teststat -eq 255 ]; then
            echo -e "$error1\n$error3" > /var/log/mxnas2_stat_report.log
            mail -s "$subject1" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
       else
            echo -e "$error1\n$error4" > /var/log/mxnas2_stat_report.log
            mail -s "$subject1" damon.stewart@x.com -c tvcvtrxsan@x.com < /var/log/mxnas2_stat_report.log -- -f "mxnas1@x.com"
    fi
    #
    #djs v2.3 2010-12-29


    Here is an example email body with a rsync error;

    MXNAS1 to MXNAS2, RSYNC last ran at Thu Dec 30 04:31:17 PST 2010 with an exit status code of;
    /mxnas1/=24

    rsync exit status code legend;
    0 Success
    1 Syntax or usage error
    2 Protocol incompatibility
    3 Errors selecting input/output files, dirs
    4 Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.
    5 Error starting client-server protocol
    6 Daemon unable to append to log-file
    10 Error in socket I/O
    11 Error in file I/O
    12 Error in rsync protocol data stream
    13 Errors with program diagnostics
    14 Error in IPC code
    20 Received SIGUSR1 or SIGINT
    21 Some error returned by waitpid()
    22 Error allocating core memory buffers
    23 Partial transfer due to error
    24 Partial transfer due to vanished source files
    25 The --max-delete limit stopped deletions
    30 Timeout in data send/receive

Similar Threads

  1. kickstart config background and scrip
    By savagehobo in forum Fedora Spins & Remixes
    Replies: 1
    Last Post: 26th April 2010, 10:44 PM
  2. How do I check status of e2fsck?
    By wintermute000 in forum Hardware & Laptops
    Replies: 3
    Last Post: 21st December 2009, 09:30 AM
  3. Hardware RAID - Check Status?
    By Kilo in forum Using Fedora
    Replies: 5
    Last Post: 20th September 2006, 01:47 AM
  4. bash check for existance of dir
    By jim in forum Programming & Packaging
    Replies: 2
    Last Post: 24th January 2006, 04:21 AM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •