Bloated Processes won't die

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Bloated Processes won't die

Preston Podaril
Running Apache 1.3.33 (compiled with mod_ssl, uses mod_gzip and
mod_auth_mysql also), php 4.3.10, phpa 1.3.3r2, mysql 4.0.18 on remote
server.  the problem exists on the servers not actually starting apache
with SSL, just /etc/init.d/apache start.  These are running in Solaris 8
on sparc hardware.

I have apache processes that will bloat up to over 100 MB, I have seen
as high as 125 MB, or RES memory according to top.  Even such a large
number that I see between 50M and 80 MB is ridiculous, also it seems
strange to me that the server would be using so much swap when it has
1/2 a GB of real memory free...:

load averages:  1.18,  0.83,  0.83
                10:35:54
68 processes:  67 sleeping, 1 on cpu
CPU states: 62.5% idle, 23.2% user, 14.4% kernel,  0.0% iowait,  0.0% swap
Memory: 2048M real, 547M free, 1293M swap in use, 2198M swap free

    PID USERNAME THR PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
  20354 nobody     3  31    0  221M  101M sleep    2:27  3.71% libhttpd.ep
  20408 nobody     3  58    0  155M   38M sleep    0:24  0.00% libhttpd.ep
  20318 nobody     3  60    0  155M   36M sleep    0:25  0.00% libhttpd.ep

I performed an lsof -p 20354 to see what all it had open and seen this:

COMMAND     PID   USER   FD   TYPE        DEVICE SIZE/OFF   NODE NAME
libhttpd. 20354 nobody  cwd   VDIR         136,0     1024      2 /
libhttpd. 20354 nobody  txt   VREG         253,1    30676 349796
/usr/local/etc/apache-1.3.33/libexec/libhttpd.ep
libhttpd. 20354 nobody  txt   VREG         136,0    44892  98711
/usr/lib/nss_files.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    24536  98707
/usr/lib/nss_dns.so.1
libhttpd. 20354 nobody  txt   VREG         253,1   163532 349808
/usr/local/etc/apache-1.3.33/libexec/php_accelerator.so
libhttpd. 20354 nobody  txt   VREG         136,0   382600  98706
/usr/lib/libresolv.so.2
libhttpd. 20354 nobody  txt   VREG         253,1  1305976 445372
/usr/local/ssl/lib/libcrypto.so.0.9.7
libhttpd. 20354 nobody  txt   VREG         253,1   232068 445376
/usr/local/ssl/lib/libssl.so.0.9.7
libhttpd. 20354 nobody  txt   VREG         253,1 10581944 349801
/usr/local/etc/apache-1.3.33/libexec/libphp4.so
libhttpd. 20354 nobody  txt   VREG         136,0   225712  98703
/usr/lib/libldap.so.4
libhttpd. 20354 nobody  txt   VREG         136,0    46936  98469
/usr/lib/libaio.so.1
libhttpd. 20354 nobody  txt   VREG         253,1   760864 390565
/usr/local/lib/libgcc_s.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    42184  98472
/usr/lib/libgen.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    16020  98459
/usr/lib/libcrypt_i.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    41816  98677
/usr/lib/librt.so.1
libhttpd. 20354 nobody  txt   VREG         253,1    78980 389700
/usr/local/lib/libz.so.1.2.1
libhttpd. 20354 nobody  txt   VREG         136,0   102424  98536
/usr/lib/libm.so.1
libhttpd. 20354 nobody  txt   VREG         253,1   161544 406192
/usr/local/etc/mysql/lib/mysql/libmysqlclient.so.12.0.0
libhttpd. 20354 nobody  txt   VREG         253,1    70520 349799
/usr/local/etc/apache-1.3.33/libexec/mod_auth_mysql.so
libhttpd. 20354 nobody  txt   VREG         253,1   109228 349800
/usr/local/etc/apache-1.3.33/libexec/mod_gzip.so
libhttpd. 20354 nobody  txt   VREG         136,0   191996  98723
/usr/lib/libthread.so.1
libhttpd. 20354 nobody  txt   VREG         136,0  1146168  98718
/usr/lib/libc.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    17096 155652
/usr/platform/sun4u/lib/libc_psr.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    24968  98487
/usr/lib/libmp.so.2
libhttpd. 20354 nobody  txt   VREG         136,0   898580  98721
/usr/lib/libnsl.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    38900  98722
/usr/lib/libpthread.so.1
libhttpd. 20354 nobody  txt   VREG         136,0    70864  98657
/usr/lib/libsocket.so.1
libhttpd. 20354 nobody  txt   VREG         253,1  1005004 349797
/usr/local/etc/apache-1.3.33/libexec/libhttpd.so
libhttpd. 20354 nobody  txt   VREG         136,0     5296  98495
/usr/lib/libdl.so.1
libhttpd. 20354 nobody  txt   VREG         136,0   227520  98479
/usr/lib/ld.so.1
libhttpd. 20354 nobody    0r  VCHR          13,2      0t0 172107
/devices/pseudo/mm@0:null
libhttpd. 20354 nobody    1w  VCHR          13,2      0t0 172107
/devices/pseudo/mm@0:null
libhttpd. 20354 nobody    2w  VREG         136,3    14979  44904
/export/apache/logs/error_log
libhttpd. 20354 nobody    3u  VREG         136,3        0  44822
/export/apache/logs/httpd.mm.22204.sem
libhttpd. 20354 nobody   15w  VREG         136,3    14979  44904
/export/apache/logs/error_log
libhttpd. 20354 nobody   16u  IPv4 0x30007e22550      0t0    TCP *:80
(LISTEN)
libhttpd. 20354 nobody   17w  VREG         136,3  1409414  44959
/export/apache/logs/access_log
libhttpd. 20354 nobody   18w  VREG         136,3        0  44960 /export
(/dev/dsk/c0t0d0s3)

I have tried trussing the process also to see if I could notice
anything, but I am not even sure what to look for in there.

These larger processes sometimes will go into a perpetual sleep also,
and will never close, but never get anymore cpu time.  It happens some
times but not always.    They have to be killed manually with a kill -9.
  A lot of times on those particular process and lsof shows an
Established tcp connection to the mysql server.

My httpd.conf is setup like this to start:

ServerType standalone
ServerRoot "/usr/local/etc/apache"
LockFile /export/apache/logs/httpd.lock
PidFile /export/apache/logs/httpd.pid
ScoreBoardFile /export/apache/logs/httpd.scoreboard
Timeout 300
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 5
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 150
MaxRequestsPerChild 100
LoadModule gzip_module        libexec/mod_gzip.so
LoadModule mysql_auth_module  libexec/mod_auth_mysql.so
LoadModule php4_module        libexec/libphp4.so
<IfDefine SSL>
LoadModule ssl_module         libexec/libssl.so
</IfDefine>

I am at a loss right now of where to go next, if anyone has any insight
or ideas on what I can do to track this down, it would be much appreciated.

Preston Podaril
[hidden email]
Network Administrator
New Digital Group

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: [hidden email]
   "   from the digest: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Bloated Processes won't die

Preston Podaril
Preston Podaril wrote:

> Running Apache 1.3.33 (compiled with mod_ssl, uses mod_gzip and
> mod_auth_mysql also), php 4.3.10, phpa 1.3.3r2, mysql 4.0.18 on remote
> server.  the problem exists on the servers not actually starting apache
> with SSL, just /etc/init.d/apache start.  These are running in Solaris 8
> on sparc hardware.
>
> I have apache processes that will bloat up to over 100 MB, I have seen
> as high as 125 MB, or RES memory according to top.  Even such a large
> number that I see between 50M and 80 MB is ridiculous, also it seems
> strange to me that the server would be using so much swap when it has
> 1/2 a GB of real memory free...:
>
> load averages:  1.18,  0.83,  0.83                10:35:54
> 68 processes:  67 sleeping, 1 on cpu
> CPU states: 62.5% idle, 23.2% user, 14.4% kernel,  0.0% iowait,  0.0% swap
> Memory: 2048M real, 547M free, 1293M swap in use, 2198M swap free
>
>    PID USERNAME THR PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
>  20354 nobody     3  31    0  221M  101M sleep    2:27  3.71% libhttpd.ep
>  20408 nobody     3  58    0  155M   38M sleep    0:24  0.00% libhttpd.ep
>  20318 nobody     3  60    0  155M   36M sleep    0:25  0.00% libhttpd.ep
>
> I performed an lsof -p 20354 to see what all it had open and seen this:
>
> COMMAND     PID   USER   FD   TYPE        DEVICE SIZE/OFF   NODE NAME
> libhttpd. 20354 nobody  cwd   VDIR         136,0     1024      2 /
> libhttpd. 20354 nobody  txt   VREG         253,1    30676 349796
> /usr/local/etc/apache-1.3.33/libexec/libhttpd.ep
> libhttpd. 20354 nobody  txt   VREG         136,0    44892  98711
> /usr/lib/nss_files.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    24536  98707
> /usr/lib/nss_dns.so.1
> libhttpd. 20354 nobody  txt   VREG         253,1   163532 349808
> /usr/local/etc/apache-1.3.33/libexec/php_accelerator.so
> libhttpd. 20354 nobody  txt   VREG         136,0   382600  98706
> /usr/lib/libresolv.so.2
> libhttpd. 20354 nobody  txt   VREG         253,1  1305976 445372
> /usr/local/ssl/lib/libcrypto.so.0.9.7
> libhttpd. 20354 nobody  txt   VREG         253,1   232068 445376
> /usr/local/ssl/lib/libssl.so.0.9.7
> libhttpd. 20354 nobody  txt   VREG         253,1 10581944 349801
> /usr/local/etc/apache-1.3.33/libexec/libphp4.so
> libhttpd. 20354 nobody  txt   VREG         136,0   225712  98703
> /usr/lib/libldap.so.4
> libhttpd. 20354 nobody  txt   VREG         136,0    46936  98469
> /usr/lib/libaio.so.1
> libhttpd. 20354 nobody  txt   VREG         253,1   760864 390565
> /usr/local/lib/libgcc_s.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    42184  98472
> /usr/lib/libgen.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    16020  98459
> /usr/lib/libcrypt_i.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    41816  98677
> /usr/lib/librt.so.1
> libhttpd. 20354 nobody  txt   VREG         253,1    78980 389700
> /usr/local/lib/libz.so.1.2.1
> libhttpd. 20354 nobody  txt   VREG         136,0   102424  98536
> /usr/lib/libm.so.1
> libhttpd. 20354 nobody  txt   VREG         253,1   161544 406192
> /usr/local/etc/mysql/lib/mysql/libmysqlclient.so.12.0.0
> libhttpd. 20354 nobody  txt   VREG         253,1    70520 349799
> /usr/local/etc/apache-1.3.33/libexec/mod_auth_mysql.so
> libhttpd. 20354 nobody  txt   VREG         253,1   109228 349800
> /usr/local/etc/apache-1.3.33/libexec/mod_gzip.so
> libhttpd. 20354 nobody  txt   VREG         136,0   191996  98723
> /usr/lib/libthread.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0  1146168  98718
> /usr/lib/libc.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    17096 155652
> /usr/platform/sun4u/lib/libc_psr.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    24968  98487
> /usr/lib/libmp.so.2
> libhttpd. 20354 nobody  txt   VREG         136,0   898580  98721
> /usr/lib/libnsl.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    38900  98722
> /usr/lib/libpthread.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0    70864  98657
> /usr/lib/libsocket.so.1
> libhttpd. 20354 nobody  txt   VREG         253,1  1005004 349797
> /usr/local/etc/apache-1.3.33/libexec/libhttpd.so
> libhttpd. 20354 nobody  txt   VREG         136,0     5296  98495
> /usr/lib/libdl.so.1
> libhttpd. 20354 nobody  txt   VREG         136,0   227520  98479
> /usr/lib/ld.so.1
> libhttpd. 20354 nobody    0r  VCHR          13,2      0t0 172107
> /devices/pseudo/mm@0:null
> libhttpd. 20354 nobody    1w  VCHR          13,2      0t0 172107
> /devices/pseudo/mm@0:null
> libhttpd. 20354 nobody    2w  VREG         136,3    14979  44904
> /export/apache/logs/error_log
> libhttpd. 20354 nobody    3u  VREG         136,3        0  44822
> /export/apache/logs/httpd.mm.22204.sem
> libhttpd. 20354 nobody   15w  VREG         136,3    14979  44904
> /export/apache/logs/error_log
> libhttpd. 20354 nobody   16u  IPv4 0x30007e22550      0t0    TCP *:80
> (LISTEN)
> libhttpd. 20354 nobody   17w  VREG         136,3  1409414  44959
> /export/apache/logs/access_log
> libhttpd. 20354 nobody   18w  VREG         136,3        0  44960 /export
> (/dev/dsk/c0t0d0s3)
>
> I have tried trussing the process also to see if I could notice
> anything, but I am not even sure what to look for in there.
>
> These larger processes sometimes will go into a perpetual sleep also,
> and will never close, but never get anymore cpu time.  It happens some
> times but not always.    They have to be killed manually with a kill -9.
>  A lot of times on those particular process and lsof shows an
> Established tcp connection to the mysql server.
>
> My httpd.conf is setup like this to start:
>
> ServerType standalone
> ServerRoot "/usr/local/etc/apache"
> LockFile /export/apache/logs/httpd.lock
> PidFile /export/apache/logs/httpd.pid
> ScoreBoardFile /export/apache/logs/httpd.scoreboard
> Timeout 300
> KeepAlive On
> MaxKeepAliveRequests 1000
> KeepAliveTimeout 5
> MinSpareServers 10
> MaxSpareServers 20
> StartServers 20
> MaxClients 150
> MaxRequestsPerChild 100
> LoadModule gzip_module        libexec/mod_gzip.so
> LoadModule mysql_auth_module  libexec/mod_auth_mysql.so
> LoadModule php4_module        libexec/libphp4.so
> <IfDefine SSL>
> LoadModule ssl_module         libexec/libssl.so
> </IfDefine>
>
> I am at a loss right now of where to go next, if anyone has any insight
> or ideas on what I can do to track this down, it would be much appreciated.
>
> Preston Podaril
> [hidden email]
> Network Administrator
> New Digital Group
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: [hidden email]
>   "   from the digest: [hidden email]
> For additional commands, e-mail: [hidden email]
>

I also compiled in and we are using OSSP mm shared memory allocation
1.30 if this helps anyone come up with any ideas.  Or some tools I can
use to dig into this apparent Memory leak.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: [hidden email]
   "   from the digest: [hidden email]
For additional commands, e-mail: [hidden email]