logging proc and mem usage

[dhav]

does anybody know a way to log the proc and mem usage? I'm basiclly looking for a way to log the type of info you get from top. I have a machine that's bogging down, I can see it with ganglia (http://tinyurl.com/7mxed screenshot), but I can't be around when it's happening to watch top...

thanks for any advice

[bytesniper]

theres probably a much better solution as this would be kinda cluncky, but you could write a really simple scipt that would cat /proc/meminfo & /proc/stat into a file thats stamped with what is logging and the date.

#!/bin/bash
# filename (in this) would be logstats
#
counter=0
touch ~/memstat && touch ~/sysstat
while [  $counter -lt $2 ]; do
 unset catstat && unset memstat && unset date
 sysstat=`cat /proc/stat`
 memstat=`cat /proc/meminfo`
 rawdate=`date` 
 fmtdate=`echo $rawdate | tr " " "-"`
 date=`echo $fmtdate1 | tr -d ":"`
 echo $date $sysstat >> ~/sysstat
 echo $date $memstat >> ~/memstat
 wait $1
done
exit 0

usage: logstats wait_time_between_logs($1) number_of_cycles($2)
example: logstats 20 300 (wait 20 between logs and repeat 300 times)

this will create 2 files in home dir and echo the contents of them every "$1" for "$2" times. each log entry will be dated/timed. i havent actually run this, but in theory i think it would work

if someone doesnt give a better option let me know and ill refine this

[dhav]

thanks for the quick reply and script! However, it doesn't quite get me towards my goal. What I REALLY need is a way to log the proc and mem of the applications. This way, when in the middle of the night, the machine starts bogging down, I can then go back and see which process was the offender. Ganglia does a great job of graphing proc/mem (and many other metrics) by the way, I highly reccomend playing around with it if you have several machines you monitor.

So, I kind of found a solution, top has a batch modes, running top -b will scroll info to the console, it will repeat nonstop unless you give it a -n option. so for instance if I want to run it once, I run top -b -n1 and get somthing like this:

top - 11:24:04 up 7 days, 22:02, 9 users, load average: 0.09, 0.05, 0.01
Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.1% us, 0.5% sy, 0.0% ni, 96.4% id, 1.9% wa, 0.0% hi, 0.0% si
Mem: 1555028k total, 1439284k used, 115744k free, 651764k buffers
Swap: 0k total, 0k used, 0k free, 95476k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4284 root 14 -1 305m 221m 7612 S 11.8 14.6 128:54.26 X
4388 root 15 0 52484 27m 8772 S 2.0 1.8 2:31.61 gnome-terminal
5289 root 15 0 3660 860 664 R 2.0 0.1 0:00.01 top
1 root 16 0 2788 628 540 S 0.0 0.0 0:01.11 init


So unless I can get somthing better, for now I'll just cron it up every five minutes, output it to a file and see if I can catch it that way. If I get the time, I'll see if I can't bang out a script to just grab the top 5 and only the important stuff and log it.

again, thanks!

[bytesniper]

i totally misunderstood what you needed, sorry you might try this though:

top -b -n1 | head -n10

that will output only the first 10 lines (system info and the first few running apps) so if you want to use cron you could

unset $date && unset $topdata
date=`date`
topdata=`top -b -n1 | head -n10`
echo "Top data for: $date" >> ~/topdata
echo $topdata >> ~/topdata
echo "------------------------------------------------------" >> ~/topdata
echo " "

hope this helps some