[Bug 52865] Crash by segmentation fault in mod_authn_core in Apache-2.4.1

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[Bug 52865] Crash by segmentation fault in mod_authn_core in Apache-2.4.1

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=52865

janani <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Version|2.4.1                       |2.4.43

--- Comment #8 from janani <[hidden email]> ---
Apache crash was first noticed in 2.4.41 .Error stack looked  similar to the
bug which was fixed in 2.4.43 but still the issue was seen in 2.4.43.We made
system level memory setting changes even after that we could see that the issue
was happening .

# vi rc.local
   # (Add the lines below to the end. Replace eth0, eth1 with the actual names)
   #          sysctl -w net.core.rmem_max=16777216
   #          sysctl -w net.core.wmem_max=16777216
   #          sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
   #          sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"
   #          sysctl -w net.ipv4.tcp_fin_timeout=10
   #          ifconfig eth0 txqueuelen 10000
   #          ifconfig eth2 txqueuelen 10000

   # reboot
14) Configure new ulimit settings
   # pbrun su-root
   # cd /etc/security
   # vi limits.conf
   # (Add these lines to the end)
   #          root             hard    nofile          16384
   #          root             soft    nofile          16384
   #          apache           hard    nofile          16384
   #          apache           soft    nofile          16384
   # :wq
   # reboot

We suspected  that the load on the server is more which may be due increase in
the number of application users . We increased the apache  mpm values and since
we are using worker settings which deals with increase load on the server with
respect to the increased connection .

/apps/hrs/tt/bin/ ./apachectl -V |grep MPM
Server MPM:     worker

             Server limit set was  300 .We are increasing the value to 600.

             Start servers from 200 to 500  and Max client to 15000
             Minspare and Maxspare thread values are changed as per the
recommendation(75 and 250) from the apache documentation.




    ServerLimit          600
    StartServers         500
    MaxClients           15000
    MinSpareThreads      75 (recommended value)
    MaxSpareThreads      250 (recommended value)
    ThreadsPerChild      25
    MaxRequestsPerChild   0


But the issue was still seen with below exception post implementing the change

I was researching on this issue  again and fortunately got the source code for
apache .

Please find the source code link for apache.

people.apache.org/~igalic/checks/httpd/2012-09-14-1/report-69Y1He.html

After the below exception Server gets hung and cannot serve any request.

[Fri Aug 07 16:12:12.684173 2020] [mpm_event:debug] [pid 19580:tid
139740527363840] event.c(1810): Too many open connections (25), not accepting
new conns in this process

This exception is seen from the time we enabled debugs .

[Fri Aug 07 16:12:12.889710 2020] [mpm_event:debug] [pid 19609:tid
139740820498176] event.c(2314): AH02471: start_threads: Using epoll (wakeable)
[Fri Aug 07 16:12:12.889934 2020] [mpm_event:debug] [pid 19610:tid
139740820498176] event.c(2314): AH02471: start_threads: Using epoll (wakeable)

Lines in the source code referring to the exception

1805                 else if (connections_above_limit()) {
1806                     disable_listensocks();
1807                     ap_log_error(APLOG_MARK, APLOG_DEBUG, 0,
ap_server_conf,
1808                                  "Too many open connections (%u), "
1809                                  "not accepting new conns in this
process",
1810                                  apr_atomic_read32(&connection_count));
1811                     ap_log_error(APLOG_MARK, APLOG_TRACE1, 0,
ap_server_conf,
1812                                  "Idle workers: %u",

This %u refers to the threads per child value in mpm which we is the default
recommended value but looks like that request on server is high where there is
lack of this parameter . Increased the thread per child to  50  and made other
parameter changes to match the value .but still the  issue was seen .

From

    ServerLimit          600
    StartServers         500
    MaxClients           15000
    MinSpareThreads      75
    MaxSpareThreads      250
    ThreadsPerChild      25

New value :

ServerLimit          300
    StartServers         200
    MaxClients           15000
    MinSpareThreads      75
    MaxSpareThreads      250
    ThreadsPerChild      50

we were getting other exception post the changes as below

AH00486: server seems busy, (you may need to increase StartServers,
ThreadsPerChild or Min/MaxSpareThreads), spawning 8 children, there are around
54 idle threads, 6 active children, and 6 children that are shutting down.

Made the threads per child to 40 and min and max spare s 25 and 75 respectively
with the semaphore changes at the system  level but still the issue is seen

Changes made on the server level

   # 17) Increase semaphore limits
   # pbrun su-root
   # vi /etc/sysctl.conf
   # Add this line:
   #    # Add additional semaphores for Channel Secure and mod_rewrite
   #    kernel.sem = 4096 512000 1600 9000
   # :wq
   # sysctl -p /etc/sysctl.conf


 What I notice is webagent fails to initialise all the time and the server
loses the connection ( no connections hit from F5 or the server does not take u
the connection which  like very low count of user when netstat performed say 25
and then the apacheURL goes down for the individual apache server though apache
process is up .

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]