FC2 load issue on server running mysql-4.0.xx

[tirespeed]

I recently upgrading one of our slave mysql servers from FC1 to FC2. The only service running on this machine is mysql-4.0.20-server (precompiled binary from mysql.com.

The server is a Compaq DL380 Dual 2.4 Xeons with 5G of ram.

When on 2.4.22-1.2197.nptlsmp kernel, load average was approx .30 to .50

Fresh install of FC2 as server, installed all updates including kernel to 2.6.6-1.435-2.3smp.

Now the load fluctuates wildly from .90 to 1.95 and query times are really slow compared to FC1.

Has anyone else seen this issue?

Thanks

[rkl]

I'm using MySQL 4.0.18 and FC2 to drive a phpBB bulletin board without any load issues - Dell PowerEdge with 2 * 2.80Ghz Xeons. Load is quite low (0.1, 0.2), but the board isn't that busy (yet !). Certainly not fluctuating like you suggested.

What happens to the load average if you:

1. Shut down MySQL for a few mins (if you can...might not be possible if it's a busy live service).

or

2. "renice -n 4" on the various MySQL processes.

I'm presuming it is MySQL to blame here and not PHP/Apache/something else.

[tirespeed]

thanks for the reply.

It cannot be apache/php because they aren't on the box. Other than some monitoring tools, this box only runs mysql. I've verified the load (database wise) via mytop, which I use all of the time. It shows the same queries per second, same % select/insert/update/delete so I'm pretty sure it's something to do with the 2.6 smp kernel and hyperthreading when the server is under a load

This machine does slightly over 4million queries per day, but this didn't change since the upgrade, so again my issue seems to be 2.4 vs. 2.6 kernels (and more specifically, maybe just the smp hyperthreading portion of 2.6) on my hardware.

I can take this server out of the group of slaves for the live site, and stop mysqld and the load will return to zero.

I haven't tried renicing the mysqld processes and will try that tomorrow.

What I will also try is disabling hyperthreading in the bios to see if that has any effect.

Anyone else having load issues at all on FC2?

I was very interested in upgrading from FC1 to FC2 mainly due to the 2.6 kernels and according to several articles they show decent improvements in multi-cpu environments. Up til now, my experience has not show this on the server side.

[tirespeed]

I've done some further research. I installed a fresh copy of FC1, ran all updates, then compiled and installed a vanilla 2.6.7 kernel from kernel.org and the same machine runs at approx the same load avg as FC1 with a 2.4 kernel.

This leads me to believe there is an issue with either the stock FC2 2.6 kernels on SMP hyperthreading, or possibly a glibc library issue?

I will re-install FC2 on this machine, and compile a vanilla 2.6.7 kernel to check.

If there are any other directions I should be looking, please let me know.

Thanks again

[jeru]

It could just be the mysql build...... Did you try building your own rpm's?

[tirespeed]

I've finally run some more tests on this situation.

I loaded FC2 and all updates on another (same model) machine and it too had wild load variabtions and overall high load averages.

I compiled a vanilla 2.6.7 SMP kernel and rebooted. voila, problem is gone again.

I believe this narrows the issue down to a Fedora 2.6 kernel.

[WeNdeL]

tirespeed, after you recompiled the kernel, were you still using a mysqld from the binary rpm install?

[tirespeed]

I've never had an rpm install of mysql, just the pre-compiled versions from mysql.com, not max just standard. I haven't had much time to go much further, but it looks like it's one of the kernel patches that the Fedora teams applies is causing the load fluctuations. I say this because I've changed mysql versions, Fedora core 1 and 2 exhibit the same issue, but only when using the Fedora smp 2.6 kernels. With either core 1 or core 2 and a vanilla kernel 2.6 the problem is gone.