Hi there, we're having a strange issue and wondering if anyone might
have any suggestions. We have a cluster of apache machines and are
experiencing a problem where the load average and other hardware
metrics are well below normal ranges, but the response times are
exceedingly high (even for tiny pieces of static content transferred
over localhost). We've been running Apache in production for several
years now without any issues, and this problem just started yesterday.
This likely means the problem isn't directly caused by Apache but is
affecting it nonetheless. If we take an httpd server out of the pool
via our load balancer, the response times go back to normal. Here's an
example of the behavior I'm describing.

[root@(protected)
--2010-08-31 15:49:43-- http://apc.rhl060.in.xxxxxxx.com/foo.html
Resolving apc.rhl060.in.xxxxxxx.com... 127.0.0.1
Connecting to apc.rhl060.in.xxxxxxx.com|127.0.0.1|:80... failed:
Connection timed out.
Retrying.

--2010-08-31 15:52:53-- (try: 2) http://apc.rhl060.in.xxxxxxx.com/foo.html
Connecting to apc.rhl060.in.xxxxxxx.com|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4 [text/html]
Saving to: `foo.html.72'

2010-08-31 15:53:14 (558 KB/s) - `foo.html.72' saved [4/4]

real  3m31.004s
user  0m0.000s
sys  0m0.003s

For the problem illustrated above, the loopback has no iptables rules
applied. Running `netstat -tapn | grep 80 | wc -l' shows 11299
connections to port 80. We're setup to use prefork with the following
configuration.

<IfModule prefork.c>
StartServers     100
MinSpareServers   100
MaxSpareServers  200
ServerLimit     1000
MaxClients     1000
MaxRequestsPerChild 100000
</IfModule>

# uptime
16:06:47 up 14 days, 17:47, 5 users, load average: 2.45, 2.54, 2.74

Checking iostat shows very little disk utilization.

The server is running CentOS 5.5 with kernel version
2.6.18-194.8.1.el5. This particular server has 24 cores and 50G of
RAM. A number of our servers are experiencing the same issue, and they
all have similar hardware and software specs.

I'm pretty stumped here. Any help would be highly appreciated.

-Travis

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@(protected)
 "  from the digest: users-digest-unsubscribe@(protected)
For additional commands, e-mail: users-help@(protected)