Reverse proxy server - SIGSEGV

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Reverse proxy server - SIGSEGV

Don Poitras
Hello. Recently, we ran a stress test (using Loadrunner) against our backend server using Apache as a reverse-proxy server. We've been running this way for some years. It's not clear when the last time (or if ever) this stress test was run, but we noticed that we were getting SegVs in various locations (mainly in mod_proxy). We're running version 2.4.39 of httpd and version 1.7.0 of APR. We're also using worker mpm and if we switch to the event or prefork mpms, we don't get any errors. We were able to reproduce the problem at 2.4.38 and 2.4.34 and didn't try any other versions.

Looking at the core files, we could see corrupted memory chains and sometimes we see a pointer to thread_mutex_cleanup(). Since this pointer is only referenced in the code as an argument to apr_pool_cleanup_register(), this led us to believe that we were seeing a case where two threads were accessing the same pool concurrently. We rebuilt httpd using --enable-pool-concurrency-check and the SegVs turned into aborts emitted by the pool concurrency check.

Looking at these new core files, it could be seen that two threads were concurrently accessing the same worker proxy connection pool. While proxy_util.c has one mutex call around the call to apr_sockaddr_info_get() (where the proxy connection pool is used), there were other calls in this module that needed the same protection.

I see a couple of outstanding bugs listed where this might be the fix:

We have a fix that works and would like to submit this to the bug database. Let me know if that seems reasonable, or if there are any questions that we can answer here before taking that step. Thanks.

Don Poitras - Host R&D    SAS Institute Inc.            SAS Campus Drive
mailto:[hidden email]     (919)531-5637  Fax:677-4444   Cary, NC 27513

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]