[Bug 63687] New: High Memory usage after upgrade to 2.4.41

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Bug 63687] New: High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

            Bug ID: 63687
           Summary: High Memory usage after upgrade to 2.4.41
           Product: Apache httpd-2
           Version: 2.4.41
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: Core
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

Hello,

after we upgraded 7 of our systems to apr-1.7.0 (from 1.6.5), pcre-8.43 (from
8.42) and apache 2.4.41 (from 2.4.39) we have a hugh memory problem - you can
see this in the screenshot attached. Upgrade was performed on 20 Aug.

We do not changed anything in the config-files.
After rolling back to the old versions, everything is fine.

Any advice?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

nitop <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--- Comment #1 from nitop <[hidden email]> ---
Created attachment 36732
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=36732&action=edit
High Memory usage

High Memory usage after upgrading

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #2 from nitop <[hidden email]> ---
Additional details:
All systems are running with Debian and Kernel "4.4.186".

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #3 from nitop <[hidden email]> ---
Created attachment 36733
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=36733&action=edit
apache2.conf

httpd/apache2 config

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

Curtis Wilson <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[hidden email]

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #4 from Joe Orton <[hidden email]> ---
You've made three changes at the same time which increases the difficult in
diagnosing this.  Can you try httpd 2.4.41 on the old APR/PCRE versions, and
see if that also has the same memory problem?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #5 from Curtis Wilson <[hidden email]> ---
I have run into the same issue on my servers, they are all Centos 6 and are
running cPanel. After the updates I found that I was getting constant issues
with memory use from apache in the worker mpm, most times when I would get to
them all httpd processes were using 8-12% memory, and causing the boxes to
overcommit.

They updated to
Apache 2.4.41 (8/22/2019)
APR 1.7 (07/03/2019)
pcre is at 7.8-7

Apache is being obtained from the cPanel easyapache repository.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #6 from Curtis Wilson <[hidden email]> ---
Also to note, until I can figure out the cause as to why this is happening, I
have had to downgrade them all to Apache 2.4.39.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #7 from Ruediger Pluem <[hidden email]> ---
(In reply to Curtis Wilson from comment #5)

> I have run into the same issue on my servers, they are all Centos 6 and are
> running cPanel. After the updates I found that I was getting constant issues
> with memory use from apache in the worker mpm, most times when I would get
> to them all httpd processes were using 8-12% memory, and causing the boxes
> to overcommit.
>
> They updated to
> Apache 2.4.41 (8/22/2019)
> APR 1.7 (07/03/2019)
> pcre is at 7.8-7

So only httpd and APR where updated, correct? pcre remained unchanged?

Like with the other reporter, can you just update Apache to isolate the
component that causes this?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #8 from Ruediger Pluem <[hidden email]> ---
(In reply to nitop from comment #3)
> Created attachment 36733 [details]
> apache2.conf
>
> httpd/apache2 config

The given configuration does not show me which MPM you are using. It may be
configured in
/etc/apache2/mods-enabled/*.load
/etc/apache2/mods-enabled/*.conf
Which MPM do you use?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #9 from nitop <[hidden email]> ---
Hello,

"Which MPM do you use?"

-> We use worker.

I've now just updated apache2 to 2.4.41 - so far no problems with Memory BUT, I
saw this:

# apache2 -V
Server version: Apache/2.4.41 (Unix)
Server loaded:  APR 1.7.0, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1

Why is this different?
We've compiled APR 1.6.5 and NOT 1.7.0.

Thank you!

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #10 from Ruediger Pluem <[hidden email]> ---
(In reply to nitop from comment #9)

> Hello,
>
> "Which MPM do you use?"
>
> -> We use worker.
>
> I've now just updated apache2 to 2.4.41 - so far no problems with Memory
> BUT, I saw this:
>
> # apache2 -V
> Server version: Apache/2.4.41 (Unix)
> Server loaded:  APR 1.7.0, APR-UTIL 1.6.1
> Compiled using: APR 1.6.5, APR-UTIL 1.6.1
>
> Why is this different?
> We've compiled APR 1.6.5 and NOT 1.7.0.

This is because you complied it against 1.6.5, but when started the httpd
process finds 1.7.0 first and hence loads it.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #11 from nitop <[hidden email]> ---
It's definitly Apache 2.4.41. I've updated only Apache2 this morning at ~9:00am
- without APR or PCRE:
Please see attached Memory-Graph (mem_usage_ONLY_Apache2_4_41.PNG)

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #12 from nitop <[hidden email]> ---
Created attachment 36743
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=36743&action=edit
Apache2.4.41 Memory Usage without APR- and pcre-update

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #13 from Ruediger Pluem <[hidden email]> ---
Have you compiled your Apache with debugging symbols?
Are you able to attach to such a memory consuming process with gdb?
If this is the case it would be helpful if you could use the following .gdbinit
for your gdb session
http://svn.apache.org/viewvc/httpd/httpd/trunk/.gdbinit?revision=1866078&view=co
Once you attached to such a memory consuming process with gdb using the above
.gdbinit please execute the following command on gdb side and report back the
output:
dump_all_pools

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #14 from nitop <[hidden email]> ---
@RuedigerPluem
"Have you compiled your Apache with debugging symbols?"

No. Is this necessary to go on with gdb?

"Are you able to attach to such a memory consuming process with gdb?"

I've not tried it yet. How should I do this?
I can not use your .gdbinit, it outputs some errors - we have an old debian
here and gdb 7.0.1.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #15 from Ruediger Pluem <[hidden email]> ---
(In reply to nitop from comment #14)
> @RuedigerPluem
> "Have you compiled your Apache with debugging symbols?"
>
> No. Is this necessary to go on with gdb?

Yes, you need the debugging symbols to extract the proper information from the
process.

>
> "Are you able to attach to such a memory consuming process with gdb?"
>
> I've not tried it yet. How should I do this?

http://httpd.apache.org/dev/debugging.html#backtrace

> I can not use your .gdbinit, it outputs some errors - we have an old debian
> here and gdb 7.0.1.

This is bad. The lowest version I tested the .gdbinit with was with 7.2. What
are the error messages?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #16 from nitop <[hidden email]> ---
@RuedigerPluem

Here are the errors with your ".gdbinit":
gdb apache2 16606
Reading symbols from /usr/sbin/apache2...done.
Attaching to program: /usr/sbin/apache2, process 16606

warning: no loadable sections found in added symbol-file system-supplied DSO at
0x7fffc77fa000
0x00007fc035f1b303 in ?? ()
Traceback (most recent call last):
  File "<string>", line 78, in <module>
  File "<string>", line 8, in __init__
AttributeError: 'module' object has no attribute 'COMMAND_USER'
/usr/local/src/mni/.gdbinit:548: Error in sourced command file:
Error while executing Python code.

"Yes, you need the debugging symbols to extract the proper information from the
process."

-> Did you mean the flag "--enable-maintainer-mode"?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #17 from Ruediger Pluem <[hidden email]> ---
(In reply to nitop from comment #16)

> @RuedigerPluem
>
> Here are the errors with your ".gdbinit":
> gdb apache2 16606
> Reading symbols from /usr/sbin/apache2...done.
> Attaching to program: /usr/sbin/apache2, process 16606
>
> warning: no loadable sections found in added symbol-file system-supplied DSO
> at 0x7fffc77fa000
> 0x00007fc035f1b303 in ?? ()
> Traceback (most recent call last):
>   File "<string>", line 78, in <module>
>   File "<string>", line 8, in __init__
> AttributeError: 'module' object has no attribute 'COMMAND_USER'
> /usr/local/src/mni/.gdbinit:548: Error in sourced command file:
> Error while executing Python code.

I cannot fix this. You would need to use a higher version of gdb in this case.
Unfortunately the Python code in .gdbinit is essential for debugging your
issue.

>
> "Yes, you need the debugging symbols to extract the proper information from
> the process."
>
> -> Did you mean the flag "--enable-maintainer-mode"?

That would be one way. Another less strict one is to

export CFLAGS="-Wall -O2 -g"

before you run the configure script.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63687] High Memory usage after upgrade to 2.4.41

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63687

--- Comment #18 from nitop <[hidden email]> ---
I am now running gdb 7.2 and tried it again:

/usr/local/bin/gdb apache2 29505
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/apache2...done.
Attaching to program: /usr/sbin/apache2, process 29505
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /usr/lib/libaprutil-1.so.0...done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /usr/lib/libapr-1.so.0...done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_files.so.2
....
....
warning: no loadable sections found in added symbol-file system-supplied DSO at
0x7ffd92a92000
0x00007f7bcee85303 in select () from /lib/libc.so.6
.gdbinit:548: Error in sourced command file:
Python scripting is not supported in this copy of GDB.
(gdb) dump_all_pools
Undefined command: "dump_pool_and_children".  Try "help".

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12