I have an update on this story.

I set up a shell script that would notice this condition (empty pages returned) and alert me immediately via "wall" within 10 seconds.

So I caught this in progress, before my 5 minute restarter would intervene, and experimented quickly.

As I mentioned before, I have perlbal running in front on port 80, with apache listening on localhost port 10080 and serving perlbal.

Since we discussed perlbal, I tried to access http://localhost.algebra.com:10080/ instead of www.algebra.com. (localhost.algebra.com resolves to 127.0.0.1 and is an alias for www.algebra.com).

Results were materially different: instead of 100% failures that I had with perlbal, I would only fail one out of many times.

The failure became intermittent instead of constant.

I believe that I have an explanation: only one worker happens to be messed up. When I access apache directly, I would connect to a random worker, so my failure would be intermittent. However, perlbal was set (in perlbal.conf) to maintain persistent connection, so it would mostly bang on one worker instead of randomly hitting them.

As a result, with perlbal, if that worker goes bad, then the whole website is consistently not working.

In other words, perlbal is not the issue, as such, but it exacerbated the problem due to the way I set it up.

I changed perlbal.conf to set persist_backend = off, which, I hope, will have an effect of perlbal creating a separate connection for every request. Should not have much effect, speed wise, since everything runs on localhost, I hope.

My new perlbal.conf is included:

REATE POOL dynamic
  pool dynamic add 127.0.0.1:10080
 
CREATE SERVICE balancer
  SET listen           = 0.0.0.0:80
  SET role             = reverse_proxy
  SET pool             = dynamic
  SET persist_client  = on
  SET persist_backend = off
  SET verify_backend  = on
ENABLE balancer
 
# always good to keep an internal management port open:
CREATE SERVICE mgmt
  SET role   = management
  SET listen = 127.0.0.1:60000
ENABLE mgmt