FC10: Added RAM, now getting Out Of Memory errors

[StickyC]

I've got an FC10 LAMP box that's relatively inactive (we use it for SVN, TWiki, and Bugzilla for a team of 2 people). We recently upgraded the RAM from 8gb to 32GB. The system will run fine for several hours using ~17mb for cache, then suddenly grab all the free RAM for cache. Even though the pattern seems odd (it doesn't seem to happen according to any schedule), my understanding is it's not out-of-the-norm.

Before:

vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 32725328 176724 153280    0    0     1     6   30   41  0  0 99  0  0

After:

vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0     68 460028  54268 32491640    0    0  9608     0 3443 3795  0  5 95  0  0

The trouble is, within an hour of grabbing all the RAM, we'll get a burst of "Out of memory: kill process xxx score xxx or a child. Killed process xxx" errors and the machine will die. It doesn't seem to be any specific process that's setting it off (I can scroll back in the console and see it's killed 3 or 4 before completely freaking out).

We did run Memtest86 for a dozen or so passes after upgrading the RAM without fail, so I'm assuming this is a memory management issue rather than actual bad RAM.

Any ideas what I can do to stave off these crashes (worst case, I could make a cron job that calls sync; echo 3 > /proc/sys/vm/drop_caches every 8 hours or so).

Linux version 2.6.27.5-117.fc10.i686.PAE (mockbuild@x86-7.fedora.phx.redhat.com) (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP Tue Nov 18 12:08:10 EST 2008

[marko]

I'd just go to koji and get a newer kernel in the FC10 series, see:

http://kojipkgs.fedoraproject.org/pa...117.fc10/i686/

You could get the PAE ones from there so you're still using PAE:
kernel-PAE-2.6.27.41-170.2.117.fc10.i686.rpm
kernel-PAE-devel-2.6.27.41-170.2.117.fc10.i686.rpm

then then download those two rpms and install:

yum localinstall --nogpgcheck kernel-PAE-2.6.27.41-170.2.117.fc10.i686.rpm kernel-PAE-devel-2.6.27.41-170.2.117.fc10.i686.rpm

This is relatively low risk since it just adds a new kernel and the one you already have would stay. Just make sure that installonly_limit in /etc/yum.conf is not so small as to push out a kernel you want to keep.
Reboot to the 2.6.27.41-170.2.117 and see if has the same problems. If ok, then edit the default in /boot/grub/grub.conf to point to this kernel, otherwise stay with the 'bad' one and look for another solution

[StickyC]

Thanks for the guidance, Marko. Unfortunately, the newer kernel did not prevent another crash. I'll have to look elsewhere for a solution.

[marko]

it might be you have something installed that has a huge memory leak

[leigh123linux]

Originally Posted by StickyC

I've got an FC10 LAMP box that's relatively inactive (we use it for SVN, TWiki, and Bugzilla for a team of 2 people). We recently upgraded the RAM from 8gb to 32GB. The system will run fine for several hours using ~17mb for cache, then suddenly grab all the free RAM for cache. Even though the pattern seems odd (it doesn't seem to happen according to any schedule), my understanding is it's not out-of-the-norm.

Before:

vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 32725328 176724 153280    0    0     1     6   30   41  0  0 99  0  0

After:

vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0     68 460028  54268 32491640    0    0  9608     0 3443 3795  0  5 95  0  0

The trouble is, within an hour of grabbing all the RAM, we'll get a burst of "Out of memory: kill process xxx score xxx or a child. Killed process xxx" errors and the machine will die. It doesn't seem to be any specific process that's setting it off (I can scroll back in the console and see it's killed 3 or 4 before completely freaking out).

We did run Memtest86 for a dozen or so passes after upgrading the RAM without fail, so I'm assuming this is a memory management issue rather than actual bad RAM.

Any ideas what I can do to stave off these crashes (worst case, I could make a cron job that calls sync; echo 3 > /proc/sys/vm/drop_caches every 8 hours or so).

Linux version 2.6.27.5-117.fc10.i686.PAE (mockbuild@x86-7.fedora.phx.redhat.com) (gcc version 4.3.2 20081105 (Red Hat 4.3.2-7) (GCC) ) #1 SMP Tue Nov 18 12:08:10 EST 2008

You should consider using a 64 bit OS as your programs are restricted to 4Gb each, PAE doesn't correct this ( I believe this is the cause of the memory error ).