[Bug 64452] New: modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[Bug 64452] New: modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

            Bug ID: 64452
           Summary: modproxy.tmp Files filling up /tmp after upgrade to
                    2.4.43
           Product: Apache httpd-2
           Version: 2.4.43
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: mod_proxy_http
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

After upgrading from Apache httpd 2.4.41 to 2.4.43 (RHEL 7, self compiled) we
are encountering problems with /tmp partition filling up rapidly because of
many "modproxy.tmp.*" files lingering.

The problem is encountered using ProxyPass with http as proxy target and doing
POST requests (payload size doesn't really matter, but it is really nice
posting an ISO image for testing, filling up /tmp).

We can not enable proxy-sendchunked because the application we are proxying to
has problems with it as soon as we enable the flag. This was the recommendation
of this article: https://access.redhat.com/solutions/3217891

Using systemd with PrivateTmp enabled the files are cleaned up on restart of
the unit. If using "legacy" sysv init Scripts we need to stop httpd, clean up
/tmp and start up again as httpd still holds a file handle open to every
modproxy.tmp.* file.

This leads to gigabytes of /tmp usage after some days..

My guess would be that this change is the source of the problem as it was the
only related change in a while:
https://github.com/apache/httpd/commit/f7d35dc166732bff0e55d1509202d76d53f8c270#diff-037f73ad4a026d77ac9c31a007388308

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #1 from Yann Ylavic <[hidden email]> ---
Any particular message in error_log file (like "child pid xxx exit signal..")?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #2 from Bernhard Friedreich <[hidden email]> ---
No nothing in error log. Only as disk was already full:

[Tue May 12 07:17:10.399683 2020] [proxy_http:error] [pid 31956:tid
139971851650816] (20014)Internal error (specific information not available):
[client 192.168.1.47:35042] AH01089: search for temporary directory failed,
referer https://ourcooldomain.com/contextpath
"000000000000000000000000097511ac-7cd4-5eba3156-bc7f0700-918e3e0ddcf9"

[Tue May 12 07:38:48.303085 2020] [proxy_http:error] [pid 31956:tid
139971960755968] (28)No space left on device: [client 192.168.1.47:34974]
AH01091: write to temporary file /tmp/modproxy.tmp.E9nczB failed, referer
https://ourcooldomain.com/contextpath
"000000000000000000000000097511ac-7cd4-5eba3668-c2ffd700-0709374c64de"

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #3 from Yann Ylavic <[hidden email]> ---
Did you also look at the main error log? I don't know if RHEL uses a single
error log file or one per vhost (plus the main one), so I ask..

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #4 from Bernhard Friedreich <[hidden email]> ---
This is from the "main" errorlog. As already stated we are using a self
compiled Apache. There is nothing related in the vhost error logs..

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #5 from Yann Ylavic <[hidden email]> ---
I can only see a reason for the file to not be cleaned up: another request
cleanup that runs before and fails (killing the cleanup chain).

Which modules are in use?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #6 from Yann Ylavic <[hidden email]> ---
Also, since you can reproduce easily (I can't), can you run a debugger with
some instructions?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #7 from Bernhard Friedreich <[hidden email]> ---
Here are the modules loaded (apachectl -M):
Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 logio_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 headers_module (shared)
 unique_id_module (shared)
 proxy_module (shared)
 proxy_connect_module (shared)
 proxy_http_module (shared)
 proxy_ajp_module (shared)
 ssl_module (shared)
 vhost_alias_module (shared)
 rewrite_module (shared)
 unixd_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_groupfile_module (shared)
 dir_module (shared)
 authz_user_module (shared)
 setenvif_module (shared)
 alias_module (shared)
 log_config_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 mime_module (shared)
 socache_shmcb_module (shared)
 socache_dbm_module (shared)
 slotmem_shm_module (shared)
 auth_basic_module (shared)
 include_module (shared)
 env_module (shared)
 status_module (shared)
 mpm_event_module (shared)
 jk_module (shared)
 sm_module (shared)

-

Yes what shall I do to debug the issue? Shall I enable some higher LogLevel
too?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #8 from Yann Ylavic <[hidden email]> ---
Created attachment 37252
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=37252&action=edit
gdb procedure

Here are the gdb steps for me to reach the cleanup point, does it work the same
for you?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #9 from Yann Ylavic <[hidden email]> ---
Comment on attachment 37252
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=37252
gdb procedure

>(gdb) run
>Starting program: /path/to/httpd -f /path/to/httpd.conf -X
>[...]
># here httpd is waiting for input, run the from the browser or curl terminal
^typo:
# here httpd is waiting for input, send the request from the browser or curl
from another terminal

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #10 from Bernhard Friedreich <[hidden email]> ---
Is it also possible to attach to a running httpd? I don't have root permission
on the server so I can only start apache using sudo with our systemd unit. Is
it sufficient to connect to the parent pid or do I need to connect to the right
worker pid?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #11 from Ruediger Pluem <[hidden email]> ---
(In reply to Yann Ylavic from comment #5)
> I can only see a reason for the file to not be cleaned up: another request
> cleanup that runs before and fails (killing the cleanup chain).

Why should a failing cleanup kill the cleanup chain? If the cleanups are run by
apr_pool_clear or apr_pool_destroy no return code is checked and the cleanup
functions in the chain are all executed regardless of their return value.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #12 from Yann Ylavic <[hidden email]> ---
(In reply to Bernhard Friedreich from comment #10)
> Is it also possible to attach to a running httpd? I don't have root
> permission on the server so I can only start apache using sudo with our
> systemd unit. Is it sufficient to connect to the parent pid or do I need to
> connect to the right worker pid?

You could attach to a worker pid, but I wouldn't recommend that on a production
server anyway, it's going to catch all the requests passing through which will
be hard to debug and disturb your production.

Can you run "strace" on a worker pid? Something like "strace -p <pid>
2>/tmp/strace.out", but it may require sudo too..
If it works, let it run a little while to see if
unlink("/tmp/modproxy.tmp.xxx") gets called (and succeeds).

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #13 from Yann Ylavic <[hidden email]> ---
(In reply to Ruediger Pluem from comment #11)
> Why should a failing cleanup kill the cleanup chain? If the cleanups are run
> by apr_pool_clear or apr_pool_destroy no return code is checked and the
> cleanup functions in the chain are all executed regardless of their return
> value.

Yeah, I don't know why I thought that..

I can't see any reason for that file stay then, it should be cleaned up with
the request (i.e. always), and there seems to be no child crash.

(In reply to Bernhard Friedreich from comment #0)
>
> Using systemd with PrivateTmp enabled the files are cleaned up on restart of
> the unit. If using "legacy" sysv init Scripts we need to stop httpd, clean
> up /tmp and start up again as httpd still holds a file handle open to every
> modproxy.tmp.* file.

Is there something special on your system that prevent opened file to be
removed? It shouldn't be an issue on unix systems usually.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #14 from Bernhard Friedreich <[hidden email]> ---
(In reply to Yann Ylavic from comment #13)

> Is there something special on your system that prevent opened file to be
> removed? It shouldn't be an issue on unix systems usually.

Between updating from 2.4.41 and 2.4.43 nothing changed on those servers so I
guess it has nothing to do wit anything special on our servers..


(In reply to Yann Ylavic from comment #12)
> You could attach to a worker pid, but I wouldn't recommend that on a
> production server anyway, it's going to catch all the requests passing
> through which will be hard to debug and disturb your production.
>

I can already reproduce the problem on one of our test environments so I should
be able to do this.. worst case I can even also get root there.


(In reply to Yann Ylavic from comment #12)
> Can you run "strace" on a worker pid? Something like "strace -p <pid>
> 2>/tmp/strace.out", but it may require sudo too..
> If it works, let it run a little while to see if
> unlink("/tmp/modproxy.tmp.xxx") gets called (and succeeds).

I'll try this later but I guess it has nothing to do with unlink not working
and more to do that those files simply weren't created <= 2.4.41..

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #15 from Bernhard Friedreich <[hidden email]> ---
I can now reproduce the problem in my local vm with a more or less stock
centos7 and our self compiled apache.

Running gdb the interesting thing is that file_cleanup is reached for apr-tmp
but never for modproxy.tmp. Sadly there seems to be a mismatch in line numbers
to source - even if the same versions (apr 1.7.0 and apr-util 1.6.1) are used..

Breakpoint 1, ap_proxy_http_prefetch (url=<optimized out>, uri=0x7fffbc010d18,
req=<optimized out>) at mod_proxy_http.c:807
807             rv = spool_reqbody_cl(req, &bytes);
(gdb)
Continuing.

Breakpoint 2, apr_file_mktemp (fp=fp@entry=0x7fffe4a63958,
template=0x7fffbc00ed80 "/tmp/apr-tmp.XXXXXX", flags=flags@entry=0,
p=p@entry=0x7fffbc009f18) at file_io/unix/mktemp.c:177
177     {
(gdb) c
Continuing.

Breakpoint 3, apr_unix_file_cleanup (thefile=0x7fffbc00ed98) at
file_io/unix/open.c:80
80          rv = file_cleanup(file, 0);
(gdb) c
Continuing.

Breakpoint 2, apr_file_mktemp (fp=fp@entry=0x7fffe4a63b30,
template=0x7fffbc00ee50 "/tmp/modproxy.tmp.XXXXXX", flags=flags@entry=0,
p=p@entry=0x7fffbc009f18) at file_io/unix/mktemp.c:177
177     {
(gdb) c
Continuing.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #16 from Bernhard Friedreich <[hidden email]> ---
I've also tried via strace using those arguments:

strace -o /root/process_dump -ff /path/to/apache/bin/httpd -f
/path/to/apache/conf/httpd.conf -X

The only unlinks I could find where those:
[root@devbox1 ~]# grep -Hrn "unlink" process_dump*
process_dump.12069:5441:unlink("/path/to/apachelogs/jk-runtime-status.12069.lock")
= 0
process_dump.12069:5442:unlink("/path/to/apachelogs/jk-runtime-status.12069") =
0
process_dump.12135:156:unlink("/tmp/apr-tmp.gYvqb8")           = 0

This is the mode using which modproxy.tmp File was opened
open("/tmp/modproxy.tmp.My4PnF", O_RDWR|O_CREAT|O_EXCL, 0600) = 23

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #17 from Bernhard Friedreich <[hidden email]> ---
I've just reconfirmed in my local vm that the modproxy.tmp files are cleaned up
using httpd 2.4.41 but not with 2.4.43.

Only differences:
httpd: 2.4.41
mod_jk: 1.2.46

vs

httpd: 2.4.43
mod_jk: 1.2.48

Common:
APR: 1.7.0
APR-UTIL: 1.6.1

As the problem happens using ProxyPass on http and has nothing to do with
mod_jk this has to be some bug in the new 2.4.43 version..

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

--- Comment #18 from Bernhard Friedreich <[hidden email]> ---
Thats how our httpd is compiled:

./configure \
  --prefix=/opt \
  --enable-so \
  --disable-userdir \
  --enable-cache=shared \
  --enable-cgi=shared \
  --enable-expires=shared \
  --enable-headers=shared \
  --enable-logio=shared \
  --enable-mem-cache=shared \
  --enable-mime-magic=shared \
  --enable-nonportable-atomics=yes \
  --enable-proxy=shared \
  --enable-proxy-http=shared \
  --enable-rewrite=shared \
  --enable-ssl=shared \
  --enable-unique-id=shared \
  --enable-usertrack=shared \
  --enable-vhost-alias=shared \
  --enable-mpms-shared='event worker' \
  --with-included-apr \
  --with-included-apr-util \
  --with-ssl=/opt/openssl-1.1.1f

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64452] modproxy.tmp Files filling up /tmp after upgrade to 2.4.43

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64452

Bernhard Friedreich <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #19 from Bernhard Friedreich <[hidden email]> ---
Looks like this bug can be closed.

The problems seems to occur from a behavior change in 2.4.43 which is
problematic for the CA SSO WebAgent module.
As soon as I enable the mod_sm module modproxy.tmp files are created (and never
deleted). With the module disabled those files are never even created. Tried it
using a fedora iso image for something really big => no modproxy.tmp file.

Thanks for your help and sorry for wasting your time..

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

123