Description

Frequently encountered problems (warnings and fatal errors) and their troubleshooting.

Building and Installation

Cannot find -lgdbm / libgdbm.so.3: open failed: No such file or directory

Please see: Missing or Misconfigured libgdbm.so.

Also it seems that on Solaris this exact issue doesn't show up at compile time, but at run time, so you may see the errors like:

  .../mod_perl-1.99_17/blib/arch/auto/APR/APR.so' for module APR:
  ld.so.1: /usr/local/ActivePerl-5.8/bin/perl: fatal:
  libgdbm.so.3: open failed: No such file or directory at
  ...5.8.3/sun4-solaris-thread-multi/DynaLoader.pm line 229.

the solution is the same, make sure that you have the libgdbm shared library and it's properly symlinked.

Configuration and Startup

Can't locate TestFilter/in_str_consume.pm in @INC...

Sometimes you get a problem of perl not being able to locate a certain Perl module. This can happen in the mod_perl test suite or in the normal mod_perl setup. One of the possible reasons is a low limit on the number of files that can be opened by a single process. To check whether this is the problem run the process under strace(1) or an equivalent utility.

For example on OpenBSD 3.5 the default setting for a maximum number of files opened by a single process seems to be 64, so when you try to run the mod_perl test suite, which opens a few hundreds of files, you will have a problem. e.g. the test suite may fail as:

  [Wed Aug 25 09:49:40 2004] [info] 26 Apache2:: modules loaded
  [Wed Aug 25 09:49:40 2004] [info] 7 APR:: modules loaded
  [Wed Aug 25 09:49:40 2004] [info] base server + 20 vhosts ready 
  to run tests
  [Wed Aug 25 09:49:40 2004] [error] Can't locate 
  TestFilter/in_str_consume.pm in @INC (@INC contains: ...

Running the system calls tracing program (ktrace(1) on OpenBSD, strace(1) on Linux):

  % sudo ktrace -d /usr/local/apache/bin/httpd -d /tmp/mod_perl-2.0/t \
    -f /tmp/mod_perl-2.0/t/conf/httpd.conf -DAPACHE2 -X

looking at the ktrace dump reveals:

 16641 httpd    NAMI "/tmp/mod_perl-2.0/t/lib/TestFilter/in_str_consume.pmc"
 16641 httpd    RET   stat -1 errno 2 No such file or directory
 16641 httpd    CALL  open(0x3cdae100,0,0)
 16641 httpd    RET   open -1 errno 24 Too many open files

It's clear that Perl can't load TestFilter/in_str_consume.pm because it can't open the file.

This problem can be resolved by increasing the open file limit to 128 (or higher):

 $ ulimit -n 128

"mod_perl.c" is not compatible with this version of Apache (found 20020628, need 20020903)

That error message means that mod_perl was built against Apache released on or post-20020628, but you are trying to load it against one released on or post-20020903. You will see the same error message for any other Apache module -- this is an error coming from Apache, not mod_perl.

Apache bumps up a special magic number every time it does a binary incompatible change, and then it makes sure that all modules that it loads were compiled against the same compatibility generation (which may include only one or quite a few Apache releases).

You may encounter this situation when you upgrade to a newer Apache, without rebuilding mod_perl. Or when you have several versions of Apache installed on the same system. Or when you install prepackaged binary versions which aren't coming from the source and aren't made against the same Apache version.

The solution is to have mod_perl built against the same Apache installed on your system. So either build from source or contact your binary version supplier and get a proper package(s) from them.

Server Hanging at the Startup

First you need to figure out where it hangs. strace(1) or an equivalent utility can be used to discover which call the server hangs on. You need to start the process in the single server mode so you will have only one process to monitor.

For example if the server hangs during 'make test', you should run:

  % cd modperl-2.0
  % strace /path/to/httpd -d t -f t/conf/httpd.conf \
    -DAPACHE2 -DONE_PROCESS -DNO_DETATCH

(and may be -DPERL_USEITHREADS if it was in the original output of make test.)

If the trace ends with:

  open("/dev/random", O_RDONLY)           = 3
  read(3, <unfinished ...>

then you have a problem with your OS, as /dev/random doesn't have enough entropy to give the required random data, and therefore it hangs. This may happen in apr_uuid_get() C call or Perl APR::UUID->new.

The solution in this case is either to fix the problem with your OS, so that

  % perl -le 'open I, "/dev/random"; read I, $d, 10; print $d'

will print some random data and not block. Or you can use an even simpler test:

  % cat /dev/random

which should print some random data and not block.

If you can't fix the OS problem, you can rebuild Apache 2.0 with --with-devrandom=/dev/urandom - however, that is not secure for certain needs. Alternatively setup EGD and rebuild Apache 2.0 with --with-egd. Apache 2.1/apr-1.1 will have a self-contained PRNG generator built-in, which won't rely on /dev/random.

(28)No space left on device

httpd-2.0 is not very helpful at telling which device has run out of precious space. Most of the time when you get an error like:

  (28)No space left on device:
  mod_rewrite: could not create rewrite_log_lock

it means that your system have run out of semaphore arrays. Sometimes it's full with legitimate semaphores at other times it's because some application has leaked semaphores and haven't cleaned them up during the shutdown (which is usually the case when an application segfaults).

Use the relevant application to list the ipc facilities usage. On most Unix platforms this is usually an ipcs(1) utility. For example linux to list the semaphore arrays you should execute:

  % ipcs -s
  ------ Semaphore Arrays --------
  key        semid      owner      perms      nsems
  0x00000000 2686976    stas      600        1
  0x00000000 2719745    stas      600        1
  0x00000000 2752514    stas      600        1

Next you have to figure out what are the dead ones and remove them. For example to remove the semid 2719745 execute:

  % ipcrm -s 2719745

Instead of manually removing each (and sometimes there can be many of them), and if you know that none of listed the semaphores is really used (all leaked), you can try to remove them all:

  % ipcs -s | perl -ane '`ipcrm -s $F[1]`'

httpd-2.0 seems to use the key 0x00000000 for its semaphores on Linux, so to remove only those that match that key you can use:

  % ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'

Notice that on other platforms the output of ipcs -s might be different, so you may need to apply a different Perl one-liner.

Segmentation Fault when Using DBI

Update DBI to at least version 1.31.

<Perl> directive missing closing '>'

See the Apache2::PerlSections manpage.

'Invalid per-unknown PerlOption: ParseHeaders' on HP-UX 11 for PA-RISC

When building mod_perl 2.0 on HP-UX 11 for PA-RISC architecture, using the HP ANSI C compiler, please make sure you have installed patches PHSS_29484 and PHSS_29485. Once installed the issue should go away.

Shutdown and Restart

Issues happening during server shutdown and restart, or during specific interpreter shutdown at runtime with threaded mpm.

Subroutines in <perl> sections under threaded mpm

If you have defined a subroutine inside a <perl> section, under threaded mpm (or under perl with enabled ithreads which spawn its own ithreads), like so:

  <Perl>
    sub foo {}
  </Perl>

At the server shutdown, or when any interpreter quits you will see the following error in the error_log:

  Attempt to free temp prematurely: SV 0x91b8e74,
  Perl interpreter: 0x8547698 during global destruction.
  Scalars leaked: 1

This is a bug in Perl and as of Perl 5.8.4 it's not resolved. For more information see: