1. Problem Description:
Our production apache22/mod_perl2 web server segfaults regularly. I
have tracked down this problem as much as I can, and could use some
help. This is running on FreeBSD; the same problem occurs on FreeBSD
7.0, 7.2, and 8.0. apache22 is installed from ports, with WITH_DEBUG
and WITH_EXECPTION_HOOK; mod_perl2 is installed from ports, my
attempts to compile it with debug flags appear so far to be
unsuccessful. (detailed version info below)
For no reason I can fathom, I can not seem to get apache to dump core
when it seg faults. I've set CoreDumpDirectory, I've set ulimit
coredumpsize, nothing. No matter; I was able to install
mod_whatkilledus and mod_backtrace.
mod_whatkilledus shows that the segfaults happen on a wide variety of
requests; most requests to our main web server are handled by
mod_perl, either with a ResponseHandler or a FilterHandler.
mod_backtrace shows that the seg fault is always happening in
modperl_perl_global_request_save:
[Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace backtrace for sig 11 (thread "pid" 6955)
[Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace main() is at 8063880
0x8085a26 <ap_run_fatal_exception+70> at /usr/local/sbin/httpd
0x808752e <ap_mpm_set_exception_hook+238> at /usr/local/sbin/httpd
0x8087569 <ap_mpm_set_exception_hook+297> at /usr/local/sbin/httpd
0xbfbfffb4
0x287a83c5 <modperl_perl_global_request_save+213> at /usr/local/libexec/apache22/mod_perl.so
0x2879982d <modperl_response_handler_cgi+141> at /usr/local/libexec/apache22/mod_perl.so
0x807bc66 <ap_run_handler+70> at /usr/local/sbin/httpd
...
So, any general pointers for what direction to go next would be
appreciated. I would like to have mod_backtrace tell me what in my
perl code is triggering the error, in which case I could work around
it. Should I continue to pursue recompiling mod_perl2 and possibly
perl itself with more debugging turned on?
(http://perl.apache.org/docs/1.0/guide/help.html)
2. Used Components and their Configuration:
*** mod_perl version 2.000004
*** using /usr/local/lib/perl5/site_perl/5.8.9/mach/Apache2/BuildConfig.pm
*** Makefile.PL options:
MP_APR_LIB => aprext
MP_APXS => /usr/local/sbin/apxs
MP_COMPAT_1X => 1
MP_GENERATE_XS => 1
MP_LIBNAME => mod_perl
MP_USE_DSO => 1
*** The httpd binary was not found
# I ran it by hand instead:
bsdtest# /usr/local/sbin/httpd -V
Server version: Apache/2.2.14 (FreeBSD)
Server built: Dec 30 2009 09:46:16
Server's Module Magic Number: 20051115:23
Server loaded: APR 1.3.8, APR-Util 1.3.9
Compiled using: APR 1.3.8, APR-Util 1.3.9
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_USE_FLOCK_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local"
-D SUEXEC_BIN="/usr/local/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="/var/log/httpd-error.log"
-D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
-D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
[Wed Dec 30 15:49:00 2009] [notice] Apache/2.2.14 (FreeBSD) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.8.9 configured -- resuming normal operations
*** (apr|apu)-config linking info
-L/usr/local/lib -laprutil-1 -lexpat -liconv -L/usr/local/lib
-L/usr/local/lib -lapr-1 -lcrypt -pthread
*** /usr/local/bin/perl -V
Summary of my perl5 (revision 5 version 8 subversion 9) configuration:
Platform:
osname=freebsd, osvers=8.0-release, archname=i386-freebsd-64int
uname='freebsd freebsd.org 8.0-release freebsd 8.0-release #0: fri sep 18 03:01:27 pdt 2009 kris@(protected) '
config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.8.9/mach -Dprivlib=/usr/local/lib/perl5/5.8.9 -Dman3dir=/usr/local/lib/perl5/5.8.9/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.8.9/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.8.9 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.8.9/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -Doptimize=-O2 -pipe -fno-strict-aliasing -Ud_dosuid -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint'
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=define use64bitall=undef uselongdouble=undef
usemymalloc=y, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include',
optimize='-O2 -pipe -fno-strict-aliasing',
cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='4.2.1 20070719 [FreeBSD]', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -Wl,-E -L/usr/local/lib'
libpth=/usr/lib /usr/local/lib
libs=-lm -lcrypt -lutil
perllibs=-lm -lcrypt -lutil
libc=, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-R/usr/local/lib/perl5/5.8.9/mach/CORE'
cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: MYMALLOC PERL_MALLOC_WRAP USE_64_BIT_INT
USE_FAST_STDIO USE_LARGE_FILES USE_PERLIO
Locally applied patches:
defined-or
Built under freebsd
Compiled at Sep 18 2009 10:03:02
%ENV:
PERL_LWP_USE_HTTP_10="1"
@INC:
/usr/local/lib/perl5/5.8.9/BSDPAN
/usr/local/lib/perl5/site_perl/5.8.9/mach
/usr/local/lib/perl5/site_perl/5.8.9
/usr/local/lib/perl5/5.8.9/mach
/usr/local/lib/perl5/5.8.9
.
*** Packages of interest status:
Apache2 : -
Apache2::Request : 2.12
CGI : 3.42
ExtUtils::MakeMaker: 6.48, undef
LWP : 5.834
mod_perl : -
mod_perl2 : 2.000004
3. This is the core dump trace: (if you get a core dump):
[CORE TRACE COMES HERE]
This report was generated by /usr/local/bin/mp2bug on Wed Dec 30 20:46:25 2009 GMT.
- Alex Aminoff
BaseSpace.net
National Bureau of Economic Research (nber.org)
