100% CPU consumption with 3.6.2-4 kernel and Java

[thhal]

After updating my kernel to 3.6.2-4 a Java process frequently consumes 100% sys time of all CPU cores for 30 seconds to several minutes. Everything works normally in between. I've tried using different JVM's but that makes no difference. Reverting back to my previous kernel 3.5.6-1 makes the problem go away.

Has anyone else seen this?

[rocketraman]

Originally Posted by thhal

After updating my kernel to 3.6.2-4 a Java process frequently consumes 100% sys time of all CPU cores for 30 seconds to several minutes. Everything works normally in between. I've tried using different JVM's but that makes no difference. Reverting back to my previous kernel 3.5.6-1 makes the problem go away.

Has anyone else seen this?

Yup, I am seeing the same thing with a Maven build using JDK 1.7.0_07 on kernel 3.6.2-1. I used the Kernel ftrace mechanism, and it seems to have something to do with migrating pages... here are some results from the ftrace:

http://pastebin.com/DDfqgWm0

[marko]

Yup, I am seeing the same thing with a Maven build using JDK 1.7.0_07 on kernel 3.6.2-1

You can try JDK 1.7.0_09 , it's recently come out

http://www.oracle.com/technetwork/ja...ads/index.html

[thhal]

Switching JDK doesn't help much I'm afraid. I'm now testing Kernel 3.6.3-1 and JDK 1.7.0_09-idedtea and I'm still seeing this. I've even tested with JDK 1.6 to no avail.

At first I thought it was the infamous leap-second bug but the remedy for that (stopping ntpd, setting date, starting ntpd again) doesn't help.

[rocketraman]

Yup, I confirm that here as well -- JDK 1.7.0_09 doesn't help.

Something that *does* help (but only temporarily) is:

$ sudo sh -c 'sync ; echo 3 >/proc/sys/vm/drop_caches ; sync'

This stops the CPU going to 100% for a while, but the problem eventually comes back again.

[David Batson]

There have been numerous reports of high CPU use with kernel 3.6.x. Its killing notebook battery life. This is the first report I've read regarding Java.

[rocketraman]

Upgraded to kernel 3.6.5 on F16 today, and so far this problem seems to be fixed.

[Ihatewindows]

Try to give the process a very low priority. That's really all I can think of at the minute.

[rocketraman]

Originally Posted by Ihatewindows

Try to give the process a very low priority. That's really all I can think of at the minute.

FYI -- nice had no effect on this issue.

[dciarnie]

Originally Posted by rocketraman

FYI -- nice had no effect on this issue.

I thought it might have fixed the problem as well but it seems that the problem doesn't manifest itself until the computer has been running for a while.

A workaround that I have found that seems to work (until the problem gets properly fixed) is to turn transparent hugpage defragging off:

echo never > /sys/kernel/mm/transparent_hugepage/defrag

[rocketraman]

Originally Posted by dciarnie

I thought it might have fixed the problem as well but it seems that the problem doesn't manifest itself until the computer has been running for a while.

Yup, I am seeing this as well -- ignore my earlier message about kernel 3.6.5 fixing it.

Originally Posted by dciarnie

A workaround that I have found that seems to work (until the problem gets properly fixed) is to turn transparent hugpage defragging off:

echo never > /sys/kernel/mm/transparent_hugepage/defrag

This does seem to work.

[stevea]

Maybe related maybe not but I've seen Firefox freeze for ~30 seconds at a time on a few occassions w/ the recent kernel. It's not waiting on network. Won't scroll or respond to buttons or anything local.

[wangmaster]

I've seen reports of this from various co-workers but have not actually run into this problem myself til recently. Just now, I was having the issue, found this post and disabled transparent hugepage defrag and the problem appears to go away.

Is transparent hugepages the same as AnonHugePages?

$ cat /proc/meminfo | grep HugePages
AnonHugePages: 796672 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0

[marko]

Some of the tools that come with the JDK will show at least some information on what the threads are doing:

pgrep java ---> will show the PID(s) of the Java process(es)

jstack -m <pid> (mixed format)

or

jstack -l <pid> (long format)

passing the PID from pgrep java as the argument where I have "<pid>"
Another good tool would be jvisualvm which comes with the JDK also, you'd just run it and select the offending process from the "Local" list and review what it's doing.

[wangmaster]

If you look at the java threads, they aren't doing anything abnormal other than what they are normally doing. The fact that the transparent_hugepages/defrag workaround works, really does imply that this is a problem with hugepage defrag algorithm causing things to go out to lunch.