Problems with freezing and kernel errors

[djey]

Hi,

i have a some questions, because i can not find out some solution for this, maybe here you will can help me.

We have problems with servers (Fedora 6/ x86_64) , they are freezing and we can't understand where is the problem. sometimes in the logs we get messages like this:

Aug 15 08:29:53 odin kernel: ntpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Aug 15 08:29:53 odin kernel:
Aug 15 08:29:53 odin kernel: Call Trace:
Aug 15 08:29:54 odin kernel: [<ffffffff8106e2d6>] out_of_memory+0x70/0x2ea
Aug 15 08:29:54 odin kernel: [<ffffffff8106ffc9>] __alloc_pages+0x24b/0x2d4
Aug 15 08:29:54 odin kernel: [<ffffffff8102d141>] default_wake_function+0x0/0xe
Aug 15 08:29:54 odin kernel: [<ffffffff81071873>] __do_page_cache_readahead+0xa1/0x1df
Aug 15 08:29:54 odin kernel: [<ffffffff8126be74>] __wait_on_bit_lock+0x5b/0x66
Aug 15 08:29:54 odin kernel: [<ffffffff880cabf1>] :dm_mod:dm_any_congested+0x38/0x3f
Aug 15 08:29:54 odin kernel: [<ffffffff8106d71b>] filemap_nopage+0x13f/0x2c1
Aug 15 08:29:54 odin kernel: [<ffffffff8107965f>] __handle_mm_fault+0x1fa/0xcfa
Aug 15 08:29:54 odin kernel: [<ffffffff8126ef8b>] do_page_fault+0x443/0x7b4
Aug 15 08:29:54 odin kernel: [<ffffffff81009972>] sys_rt_sigreturn+0x281/0x34f
Aug 15 08:29:55 odin kernel: [<ffffffff81009a08>] sys_rt_sigreturn+0x317/0x34f
Aug 15 08:29:55 odin kernel: [<ffffffff8126d4dd>] error_exit+0x0/0x84
Aug 15 08:29:55 odin kernel:
Aug 15 08:29:55 odin kernel: Mem-info:
Aug 15 08:29:55 odin kernel: Node 0 DMA per-cpu:
Aug 15 08:29:56 odin kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Aug 15 08:29:56 odin kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Aug 15 08:29:56 odin kernel: Node 0 DMA32 per-cpu:
Aug 15 08:29:56 odin kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 25 Cold: hi: 62, btch: 15 usd: 44
Aug 15 08:29:56 odin kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 114 Cold: hi: 62, btch: 15 usd: 50
Aug 15 08:29:56 odin kernel: Node 0 Normal per-cpu:
Aug 15 08:29:56 odin kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 19 Cold: hi: 62, btch: 15 usd: 61
Aug 15 08:29:57 odin kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 50 Cold: hi: 62, btch: 15 usd: 59
Aug 15 08:29:57 odin kernel: Active:453307 inactive:501457 dirty:0 writeback:0 unstable:0
Aug 15 08:29:57 odin kernel: free:5206 slab:10533 mapped:19 pagetables:30217 bounce:0
Aug 15 08:29:59 odin kernel: Node 0 DMA free:10780kB min:20kB low:24kB high:28kB active:0kB inactive:0kB present:10260kB pages_scanned:0 all_unreclaimable? yes
Aug 15 08:29:59 odin kernel: lowmem_reserve[]: 0 3511 4016
Aug 15 08:29:59 odin kernel: Node 0 DMA32 free:9080kB min:7080kB low:8848kB high:10620kB active:1538828kB inactive:1901380kB present:3595360kB pages_scanned:15126949 all_unrecla
imable? yes
Aug 15 08:29:59 odin kernel: lowmem_reserve[]: 0 0 505
Aug 15 08:29:59 odin kernel: Node 0 Normal free:964kB min:1016kB low:1268kB high:1524kB active:274400kB inactive:104448kB present:517120kB pages_scanned:2637130 all_unreclaimable? yes

We get this kind of messages. we tested for bad rams and rootkis, everything is ok. And the main problem is that when the server is frozen, it doesn't logged what is happening after restart, we can't see what is wrong.

What can we do in this situation, how to understand which aplication make the problems with out-of-memory?

There writes killed mysql or irqbalance, but sometimes i suppose this is not right, or maybe i'm wrong...

I hope that you will help me to solve these problems

Thanks in advanced!

[PhillyFloyd]

I think that is just output from magic sysrq -- those outputs are just dumps of system registers, cpu, and memory information ... if your box is about to crash it will dump this information

ntpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

There is the problem, ntp is reporting "oom" -- sounds like it ran away with the system RAM -- you can disable it, however if your system resources aren't tuned to what you are using the server for another proc may just jump up to the top and cause a system freeze ... you should check to make sure the system RAM is enough for what you are tasking on it

[redhat71]

it happens to me too, i can't say for sure but i think it's only with recent kernels
cause it can't be that all these apps are causing problems

[~]$ grep "invoked oom-killer" /var/log/messages | more
Oct 11 18:40:41 redhat71 kernel: ssh-agent invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:09 redhat71 kernel: xfce4-systemloa invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: gam_server invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: mysqld invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: pop3-login invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: amule invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: Xorg invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: scim-panel-gtk invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: httpd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 11 18:41:10 redhat71 kernel: xfce4-menu-plug invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:01:28 redhat71 kernel: pop3-login invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:02:28 redhat71 kernel: ssh-agent invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:03:10 redhat71 kernel: rpmq invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:03:13 redhat71 kernel: xfce4-netload-p invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:03:25 redhat71 kernel: xfce4-netload-p invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:05:52 redhat71 kernel: klogd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:06:59 redhat71 kernel: scim-launcher invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:08:19 redhat71 kernel: scim-launcher invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:11:49 redhat71 kernel: gconfd-2 invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:13:14 redhat71 kernel: scim-launcher invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:57:32 redhat71 kernel: amule invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:06 redhat71 kernel: giftd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:21 redhat71 kernel: mysqld invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:25 redhat71 kernel: hald-addon-stor invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:25 redhat71 kernel: mysqld invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:25 redhat71 kernel: hald-addon-stor invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:25 redhat71 kernel: dovecot invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:25 redhat71 kernel: scim-launcher invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:25 redhat71 kernel: scim-launcher invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Oct 13 16:58:25 redhat71 kernel: giftd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0

[djey]

Hi, thanks for the answers. We find a way to solve the problem. We understood that actually problem was with small swap and big RAM. When the system try to cache data from RAM to swap, the swap is not so big and then is called oom. We have 4 GB RAM and we have set 2 GB swap. Now we have 8 GB swap and we have not problems with oom. I don't know how is the situation with you, but you can think for something similar. I'm not sure that ntpd is the real problem. there is some other application that is reason for out-of memory. The point is to find the real reason about it. If i can help with something, i will be glad to do it.

Thanks again!
Regards, Ali Nebi!