F10 Webserver goes down.

[LaKing]

total 80
-rw-r--r-- 1 root root 3492 2008-02-25 13:00 auth_mysql.conf
-rw-r--r-- 1 root root 874 2008-12-31 09:18 awstats.conf
-rw-r--r-- 1 root root 604 2008-08-11 18:17 BackupPC.conf
-rw-r--r-- 1 root root 475 2008-12-10 18:45 limitipconn.conf
-rw-r--r-- 1 root root 295 2005-12-07 17:45 manual.conf
-rw-r--r-- 1 root root 7380 2008-05-16 11:37 mod_bw.conf
-rw-r--r-- 1 root root 2159 2008-08-11 17:12 mod_cband.conf
-rw-r--r-- 1 root root 140 2008-08-29 04:34 mod_geoip.conf
-rw-r--r-- 1 root root 1796 2005-04-22 14:53 perl.conf
-rw-r--r-- 1 root root 560 2008-09-13 17:15 php.conf
-rw-r--r-- 1 root root 566 2005-12-05 18:26 proxy_ajp.conf
-rw-r--r-- 1 root root 1671 2008-10-07 16:00 python.conf
-rw-r--r-- 1 root root 392 2008-10-21 13:52 README
-rw-r--r-- 1 root root 334 2008-12-19 08:42 squid.conf
-rw-r--r-- 1 root root 96 2008-12-04 13:05 squirrelmail.conf
-rw-r--r-- 1 root root 9534 2008-07-15 15:44 ssl.conf
-rw-r--r-- 1 root root 299 2004-09-09 08:16 welcome.conf

#
# PHP is an HTML-embedded scripting language which attempts to make it
# easy for developers to write dynamically generated webpages.
#

LoadModule php5_module modules/libphp5.so

#
# Cause the PHP interpreter to handle files with a .php extension.
#
AddHandler php5-script .php
AddType text/html .php

#
# Add index.php to the list of files that will be served as directory
# indexes.
#
DirectoryIndex index.php

#
# Uncomment the following line to allow PHP to pretty-print .phps
# files as PHP source code:
#
#AddType application/x-httpd-php-source .phps

All those installation defaults, ..
Only CBand added, but for now disabled.

[LaKing]

epoll(4) - Linux man page
Name
epoll - I/O event notification facility

... Can this be a Hardware failure?

[marcrblevins]

I don't know if its a hardware failure.

If you don't mind, can you remove packages for:
mod_bw.i386 : Bandwidth Limiter For Apache
mod_cband.i386 : Bandwidth limiting for virtual hosts
mod_geoip.i386 : GeoIP module for the Apache HTTP Server

Just do one at a time to remove a package and check your Apache(service httpd restart).

[LaKing]

It didnt help.

[LaKing]

No solution found. ..
Reinstalled all software on new hardware ...

[LaKing]

After a few days of running without problems, i installed more packages.
The problem exist again, in the same way.
I suspect now performance monitoring programm bugs.

Those packages are suspected:
Feb 09 15:42:02 Installed: t1lib.x86_64 5.1.2-2.fc9
Feb 09 15:42:03 Installed: php-gd.x86_64 5.2.6-5
Feb 09 15:46:17 Installed: php-mbstring.x86_64 5.2.6-5
Feb 09 15:46:19 Installed: aspell.x86_64 12:0.60.6-3.fc10
Feb 09 15:46:20 Installed: aspell-en.x86_64 50:6.0-9.fc9
Feb 09 15:46:33 Installed: squirrelmail.noarch 1.4.17-2.fc10
Feb 09 15:50:16 Installed: awstats.noarch 6.9-1.fc10
Feb 09 15:52:11 Installed: GeoIP.x86_64 1.4.5-2.fc10
Feb 09 15:52:12 Installed: mod_bw.x86_64 0.8-1.fc10
Feb 09 15:52:12 Installed: mod_cband.x86_64 0.9.7.5-3.fc10
Feb 09 15:52:12 Installed: mod_geoip.x86_64 1.2.5-2.fc10
Feb 09 15:52:13 Installed: mod_limitipconn.x86_64 0.23-1.fc10
Feb 09 15:58:46 Installed: raptor.x86_64 1.4.18-1.fc10
Feb 09 15:58:47 Installed: ilmbase.x86_64 1.0.1-2.fc9
Feb 09 15:58:47 Installed: glib.x86_64 1:1.2.10-30.fc10
Feb 09 15:58:48 Installed: clucene-core.x86_64 0.9.21-1.fc10
Feb 09 15:58:49 Installed: strigi-libs.x86_64 0.5.11.1-2.fc10
Feb 09 15:58:51 Installed: gtk+.x86_64 1:1.2.10-66.fc10
Feb 09 15:58:52 Installed: OpenEXR-libs.x86_64 1.6.1-4.fc10
Feb 09 15:58:52 Installed: rasqal.x86_64 0.9.15-1.fc9
Feb 09 15:58:53 Installed: libmodplug.x86_64 1:0.8.4-3.fc9
Feb 09 15:58:54 Installed: libmpcdec.x86_64 1.2.6-4.fc9
Feb 09 15:58:54 Installed: postgresql-libs.x86_64 8.3.6-1.fc10
Feb 09 15:58:55 Installed: redland.x86_64 1.0.7-2.fc10
Feb 09 15:58:56 Installed: soprano.x86_64 2.1.1-1.fc10
Feb 09 15:58:57 Installed: perl-TermReadKey.x86_64 2.30-6.fc9
Feb 09 15:58:57 Installed: libacpi.x86_64 0.2-12.fc10
Feb 09 15:58:57 Installed: libXvMC.x86_64 1.0.4-5.fc10
Feb 09 15:58:59 Installed: kde-filesystem.noarch 4-23.fc10
Feb 09 15:59:02 Installed: xine-lib.x86_64 1.1.16.1-1.fc10
Feb 09 15:59:03 Installed: kdelibs-common.x86_64 6:4.1.4-2.fc10
Feb 09 15:59:28 Installed: oxygen-icon-theme.noarch 4.1.4-2.fc10
Feb 09 15:59:29 Installed: solar-backgrounds-common.noarch 0.92.0-1.fc10
Feb 09 15:59:30 Installed: yacpi.x86_64 3.0.1-11.fc10
Feb 09 15:59:30 Installed: bubblemon.x86_64 1.46-8.fc9
Feb 09 15:59:30 Installed: libpfm.x86_64 3.5-2.fc10
Feb 09 15:59:31 Installed: xrestop.x86_64 0.4-5.fc10
Feb 09 15:59:32 Installed: fspy.x86_64 0.1.0-3.fc10
Feb 09 15:59:32 Installed: latencytop.x86_64 0.4-2.fc10
Feb 09 15:59:34 Installed: gkrellm.x86_64 2.3.2-1.fc10
Feb 09 15:59:34 Installed: libatasmart.x86_64 0.1-1.fc10
Feb 09 15:59:35 Installed: solar-kde-theme.noarch 0.1.17-1.fc10
Feb 09 15:59:36 Installed: kde-settings.noarch 4.1-4.20081031svn.fc10
Feb 09 15:59:40 Installed: ibmonitor.noarch 1.4-1
Feb 09 15:59:40 Installed: phonon.x86_64 4.2.0-7.fc10
Feb 09 16:00:08 Installed: kdelibs.x86_64 6:4.1.4-2.fc10
Feb 09 16:00:08 Installed: ksysguardd.x86_64 4.1.4-1.fc10
Feb 09 16:00:09 Installed: phonon-backend-xine.x86_64 4.1.4-2.fc10

[djvaporz]

try disabling mod_python as it looks it needed to be reloaded.

[LaKing]

removed mod_phthon .. didnt help.
Massive crashing continiues

[root@r2 ~]# ll /etc/httpd/conf.d
total 52
-rw-r--r-- 1 root root 874 2008-12-31 09:18 awstats.conf
-rw-r--r-- 1 root root 295 2005-12-07 17:45 manual.conf
-rw-r--r-- 1 root root 1796 2005-04-22 14:53 perl.conf
-rw-r--r-- 1 root root 560 2008-09-13 17:15 php.conf
-rw-r--r-- 1 root root 566 2005-12-05 18:26 proxy_ajp.conf
-rw-r--r-- 1 root root 392 2008-10-21 13:52 README
-rw-r--r-- 1 root root 334 2008-12-19 08:42 squid.conf
-rw-r--r-- 1 root root 96 2008-12-04 13:05 squirrelmail.conf
-rw-r--r-- 1 root root 9534 2008-07-15 15:44 ssl.conf
-rw-r--r-- 1 root root 352 2004-09-09 16:22 webalizer.conf
-rw-r--r-- 1 root root 299 2004-09-09 08:16 welcome.conf

[root@r2 ~]# cat /etc/httpd/conf.d/php.conf
#
# PHP is an HTML-embedded scripting language which attempts to make it
# easy for developers to write dynamically generated webpages.
#

LoadModule php5_module modules/libphp5.so

#
# Cause the PHP interpreter to handle files with a .php extension.
#
AddHandler php5-script .php
AddType text/html .php

#
# Add index.php to the list of files that will be served as directory
# indexes.
#
DirectoryIndex index.php

#
# Uncomment the following line to allow PHP to pretty-print .phps
# files as PHP source code:
#
#AddType application/x-httpd-php-source .phps

[LaKing]

i added the PID numbers to the access logging.
... whenever a segfault appears, there is no entry with that PID in the access logs.

Tryed to disable apache sites in groups.
... There are no errors if there is no significant traffic. Couldnt locate any error sources with that method.

Now back at looking the gdb code.

[root@r2 init.d]# gdb /usr/sbin/httpd /dump/a/1
GNU gdb Fedora (6.8-29.fc10)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...

warning: Can't read pathname for load map: Input/output error.
Loaded symbols for /usr/sbin/httpd
Reading symbols from /lib64/libm-2.9.so...Reading symbols from /usr/lib/debug/lib64/libm-2.9.so.debug...done.
done.
Loaded symbols for /lib64/libm-2.9.so
Reading symbols from /lib64/libpcre.so.0.0.1...Reading symbols from /usr/lib/debug/lib64/libpcre.so.0.0.1.debug...done.
done.
...
...
...
Loaded symbols for /usr/lib64/php/modules/pdo_mysql.so
Reading symbols from /usr/lib64/php/modules/pdo_sqlite.so...Reading symbols from /usr/lib/debug/usr/lib64/php/modules/pdo_sqlite.so.debug...done.
done.
Loaded symbols for /usr/lib64/php/modules/pdo_sqlite.so
Reading symbols from /usr/lib64/libsqlite3.so.0.8.6...Reading symbols from /usr/lib/debug/usr/lib64/libsqlite3.so.0.8.6.debug...done.
done.
Loaded symbols for /usr/lib64/libsqlite3.so.0.8.6
Reading symbols from /usr/lib64/php/modules/zip.so...Reading symbols from /usr/lib/debug/usr/lib64/php/modules/zip.so.debug...done.
done.
Loaded symbols for /usr/lib64/php/modules/zip.so
Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
[New process 18770]
#0 apr_pollset_add (pollset=0x0, descriptor=0x7fffcfc125c0) at poll/unix/epoll.c:150
150 if (pollset->flags & APR_POLLSET_NOCOPY) {
(gdb)
(gdb) thread apply all bt full

Thread 1 (process 18770):
#0 apr_pollset_add (pollset=0x0, descriptor=0x7fffcfc125c0) at poll/unix/epoll.c:150
ev = {events = 1, data = {ptr = 0x0, fd = 0, u32 = 0, u64 = 0}}
ret = <value optimized out>
elem = <value optimized out>
rv = <value optimized out>
#1 0x00000000002df61e in child_main (child_num_arg=<value optimized out>)
at /usr/src/debug/httpd-2.2.10/server/mpm/prefork/prefork.c:532
pfd = {p = 0x0, desc_type = APR_POLL_SOCKET, reqevents = 1, rtnevents = 0, desc = {f = 0x7fbcc7b01b78,
s = 0x7fbcc7b01b78}, client_data = 0x7fbcc7b01b40}
ptrans = (apr_pool_t *) 0x7fbcb6b690e8
allocator = (apr_allocator_t *) 0x7fbcb6b66fe0
status = <value optimized out>
i = <value optimized out>
lr = (ap_listen_rec *) 0x7fbcc7b01b40
pollset = (apr_pollset_t *) 0x0
sbh = (ap_sb_handle_t *) 0x7fbcb6b67320
bucket_alloc = <value optimized out>
last_poll_idx = <value optimized out>
#2 0x00000000002dfa7a in make_child (s=0x7fbcc7b09e30, slot=129)
at /usr/src/debug/httpd-2.2.10/server/mpm/prefork/prefork.c:746
No locals.
#3 0x00000000002e05aa in perform_idle_server_maintenance () at /usr/src/debug/httpd-2.2.10/server/mpm/prefork/prefork.c:881
i = 2
idle_count = 0
free_length = 4
free_slots = {128, 129, 130, 131, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 63, 64, 65, 66, 67,
68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78}
last_non_dead = 2064
total_non_dead = -809426144
#4 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>)
at /usr/src/debug/httpd-2.2.10/server/mpm/prefork/prefork.c:1085
status = 0
pid = {pid = -1, in = 0x0, out = 0x7fbcba4b8670, err = 0x4f3140}
child_slot = <value optimized out>
exitwhy = APR_PROC_EXIT
processed_status = <value optimized out>
---Type <return> to continue, or q <return> to quit---

index = <value optimized out>
remaining_children_to_start = 0
rv = <value optimized out>
#5 0x00000000002b858d in main (argc=1, argv=0x7fffcfc12a18) at /usr/src/debug/httpd-2.2.10/server/main.c:740
c = 0 '\0'
configtestonly = <value optimized out>
confname = 0x2e2000 "conf/httpd.conf"
def_server_root = 0x2e2010 "/etc/httpd"
temp_error_log = 0x0
error = <value optimized out>
process = <value optimized out>
server_conf = (server_rec *) 0x7fbcc7b09e30
pglobal = (apr_pool_t *) 0x7fbcc7b006f8
pconf = (apr_pool_t *) 0x7fbcc7b02708
plog = (apr_pool_t *) 0x7fbcc7b34898
ptemp = (apr_pool_t *) 0x7fbcc7b3b8c8
pcommands = (apr_pool_t *) 0x7fbcc7b04718
opt = (apr_getopt_t *) 0x7fbcc7b04810
rv = <value optimized out>
mod = <value optimized out>
optarg = 0xef53 ""

[vallimar]

If the error is only happening under traffic, try removing mod_limitipconn.
Also, if you have any php accelerators, remove them as well.
Your output shows you crashing on the php zip module again, so remove that.
Remove any php modules you don't need/actively require.

[LaKing]

We deleted all files from the suspected host, and uploaded back only mp3's.
Seems, the error still exist, so i guess its not a php related problem.

We will have to move the file transfers to FTP protocoll ...

[LaKing]

The larg file downloads have been moved to the anonymus FTP area.
Since then, no error accoured. ...

.. problem solved. ...