[Bug 63430] New: proxy client certificates not found despite being configured

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Bug 63430] New: proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

            Bug ID: 63430
           Summary: proxy client certificates not found despite being
                    configured
           Product: Apache httpd-2
           Version: 2.4.39
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: mod_ssl
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

Apache 2.4.39 is failing requests going to an HTTPS proxy backend with errors
like so, indicating a client certificate was not configured, however, we know
that it was configured.

[Tue May 14 09:49:03.378930 2019] [ssl:warn] [pid 1674555:tid 140693875197696]
AH02268: Proxy client certificate callback: (dw25136:443) downstream server
wanted client certificate but none are configured


These log lines, a hour or so earlier, tell us that a client certificate was
configured (in fact, the same one was used in three 3 contexts, hence the
triple message)


[Tue May 14 08:05:16.787346 2019] [ssl:debug] [pid 1670484:tid 140697080997632]
ssl_engine_init.c(1582): AH02207: loaded 1 client certs for SSL proxy
[Tue May 14 08:05:16.787558 2019] [ssl:debug] [pid 1670484:tid 140697080997632]
ssl_engine_init.c(1582): AH02207: loaded 1 client certs for SSL proxy
[Tue May 14 08:05:16.788403 2019] [ssl:debug] [pid 1670484:tid 140697080997632]
ssl_engine_init.c(1582): AH02207: loaded 1 client certs for SSL proxy

I have selectively pulled out log lines to make the point and to minimize the
amount of scrubbing I have to do.

This looks like new behavior in 2.4.39 to me, but I haven't yet demonstrated
that.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #1 from Ruediger Pluem <[hidden email]> ---
The version was the same (2.4.39) for both log lines / blocks and you did not
restart in between?
So the server was just running?
What is your configuration?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #2 from Ruediger Pluem <[hidden email]> ---
With which version does your setup work?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #3 from [hidden email] ---
technically, there was a restart between those sets of lines, but not between
these, I have inserted the "resuming operation" and "mod_ssl" lines from the
global log as well for comparison

[Tue May 14 09:27:15.212161 2019] [ssl:info] [pid 781991:tid 140222231095040]
AH01876: mod_ssl/2.4.39 compiled against Server: Apache/2.4.39, Library:
OpenSSL/1.0.2r
[Tue May 14 09:27:15.252246 2019] [ssl:debug] [pid 781991:tid 140222231095040]
ssl_engine_init.c(1582): AH02207: loaded 1 client certs for SSL proxy
[Tue May 14 09:27:15.252487 2019] [ssl:debug] [pid 781991:tid 140222231095040]
ssl_engine_init.c(1582): AH02207: loaded 1 client certs for SSL proxy
[Tue May 14 09:27:15.253510 2019] [ssl:debug] [pid 781991:tid 140222231095040]
ssl_engine_init.c(1582): AH02207: loaded 1 client certs for SSL proxy
[Tue May 14 09:27:29.269928 2019] [mpm_event:notice] [pid 781991:tid
140222231095040] AH00489: Apache/2.4.39 (Unix) OpenSSL/1.0.2r mod_fcgid/2.3.9
mod_auth_kerb/5.4 mod_qos/11.62 mod_jk/1.2.46 configured -- resuming normal
operations
[Tue May 14 09:37:43.553029 2019] [ssl:warn] [pid 799222:tid 140218148460288]
AH02268: Proxy client certificate callback: (dw25136:11719) downstream server
wanted client certificate but none are configured

The configuration is pretty big, but the relevant configuration just involves

<Proxy balancer://balancer2>
SSLProxyMachineCertificateFile  /path/to/cert.pem
</Proxy>

and a later proxypass line.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #4 from Ruediger Pluem <[hidden email]> ---
(In reply to mark from comment #3)

> technically, there was a restart between those sets of lines, but not
> between these, I have inserted the "resuming operation" and "mod_ssl" lines
> from the global log as well for comparison
>
> [Tue May 14 09:27:15.212161 2019] [ssl:info] [pid 781991:tid
> 140222231095040] AH01876: mod_ssl/2.4.39 compiled against Server:
> Apache/2.4.39, Library: OpenSSL/1.0.2r
> [Tue May 14 09:27:15.252246 2019] [ssl:debug] [pid 781991:tid
> 140222231095040] ssl_engine_init.c(1582): AH02207: loaded 1 client certs for
> SSL proxy
> [Tue May 14 09:27:15.252487 2019] [ssl:debug] [pid 781991:tid
> 140222231095040] ssl_engine_init.c(1582): AH02207: loaded 1 client certs for
> SSL proxy
> [Tue May 14 09:27:15.253510 2019] [ssl:debug] [pid 781991:tid
> 140222231095040] ssl_engine_init.c(1582): AH02207: loaded 1 client certs for
> SSL proxy
> [Tue May 14 09:27:29.269928 2019] [mpm_event:notice] [pid 781991:tid
> 140222231095040] AH00489: Apache/2.4.39 (Unix) OpenSSL/1.0.2r
> mod_fcgid/2.3.9 mod_auth_kerb/5.4 mod_qos/11.62 mod_jk/1.2.46 configured --
> resuming normal operations
> [Tue May 14 09:37:43.553029 2019] [ssl:warn] [pid 799222:tid
> 140218148460288] AH02268: Proxy client certificate callback: (dw25136:11719)
> downstream server wanted client certificate but none are configured
>
> The configuration is pretty big, but the relevant configuration just involves
>
> <Proxy balancer://balancer2>
> SSLProxyMachineCertificateFile  /path/to/cert.pem
> </Proxy>
>
> and a later proxypass line.

What is the ProxyPass line?
Which URL triggers the AH02268?
The AH01876 appears 3 times. So you have 2 further Proxy sections that contain
SSLProxyMachineCertificateFile?
It did work with 2.4.x?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #5 from [hidden email] ---
this is the proxypass line:

 ProxyPass /cca/messages balancer://balancer2/cca/messages

Here's the full Proxy block with a bit of scrubbing

#
# BalancerConfiguration 2
#
<Proxy balancer://balancer2>
 SSLProxyMachineCertificateFile /vhosts/somevhost/somepath/client.pem
 BalancerMember https://some.backend.corp.com:443 retry=5 timeout=120
 ProxySet stickysession=JSESSIONID|jsessionid
 ProxySet scolonpathdelim=On
 ProxySet lbmethod=byrequests
 ProxySet forcerecovery=On
</Proxy>

You're correct, we have three proxy blocks + corresponding ProxyPass
definitions for this VirtualHost.

Here's the access log line for that failed request.

10.10.10.10 - some_remote_user [14/May/2019:09:37:43 +0200] "HEAD
/cca/messages?q=read:false HTTP/1.1" 500 - "-" "-"


I have not yet verified it, but my understanding is that this did work for
Apache 2.4.38 at least. You may wish to wait until I can verify this myself,
but I believe to be true so far.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #6 from [hidden email] ---
We have confirmed a configuration nearly identical to this one does work for
version 2.4.38

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #7 from Ruediger Pluem <[hidden email]> ---
Created attachment 36585
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=36585&action=edit
Possible fix

Does the attached patch fix your problem?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #8 from [hidden email] ---
(In reply to Ruediger Pluem from comment #7)
> Created attachment 36585 [details]
> Possible fix
>
> Does the attached patch fix your problem?

Assuming I patched it in correctly, it looks like that patch has not changed
the outcome. I do now have a fast loop for building and testing, so you're
welcome to feed us small updates to the patch as frequently as you like and I
can feedback within an hour or less.

Is there any diagnostics that can be dumped?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #9 from [hidden email] ---
(In reply to mark from comment #6)
> We have confirmed a configuration nearly identical to this one does work for
> version 2.4.38

And to confirm, now that i can flip back and forth between a 2.4.38 and 2.4.39
build within seconds, this works fine with 2.4.38. Definitely a regression, I'm
afraid.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #10 from Rainer Jung <[hidden email]> ---
I can reproduce with 2.4.39, not with 2.4.38.

It has to do with using SSL config in a <Proxy> container section.

Small repro setup:

- using one web server with two VHosts.
  - One VHost http, one https
- Loading mod_ssl, mod_socache_shmcb, mod_proxy, mod_proxy_http
- config:

Listen 9980
<VirtualHost *:9980>

  SSLProxyEngine on
  SSLProxyVerify none
  SSLProxyCheckPeerName off
  SSLProxyCheckPeerExpire off

  ProxyPass / https://localhost:9943/

  # 2.4.39 is olny broken when SSLProxyMachineCertificateFile
  # is in this <Proxy> container.
  # 2.4.38 works.
  <Proxy https://localhost:9943>
    SSLProxyMachineCertificateFile conf/client.pem
  </Proxy>

</VirtualHost>

# Default SSL settings
SSLCipherSuite HIGH:MEDIUM:!MD5:!RC4:!3DES
SSLProxyCipherSuite HIGH:MEDIUM:!MD5:!RC4:!3DES
SSLHonorCipherOrder on
SSLProtocol all -SSLv3
SSLProxyProtocol all -SSLv3
SSLPassPhraseDialog  builtin
SSLSessionCache        "shmcb:/path/to/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300

Listen 9943
<VirtualHost _default_:9943>
  DocumentRoot "/path/to/my/htdocs/virt"
  ServerName www.example.com
  ServerAdmin [hidden email]
  SSLEngine on
  SSLCertificateFile "conf/ssl.crt/server.crt"
  SSLCertificateKeyFile "conf/ssl.key/server.key"
  SSLCACertificateFile "conf/client.crt"
  SSLVerifyClient require
  SSLVerifyDepth  0
</VirtualHost>

- run test using

  curl http://localhost:9980/

results in

HTTP/1.1 502 Proxy Error

Diffing proxy and ssl trace8 log lines in error.log shows as the first delta:

< ssl_engine_kernel.c(1899): [client ::1:9943] AH02277: Proxy client
certificate callback: (...:9980) AH02279: found acceptable cert, sending
[subject: CERTDETAILS / issuer: ISSUERDETAILS / serial: 848692B2649501A5 /
notbefore: May 15 12:10:22 2019 GMT / notafter: May 14 12:10:22 2022 GMT]


> [Wed May 15 14:38:56.282961 2019] [ssl:warn] [pid 14962] - AH02268: Proxy client certificate callback: (abies-12.kippdata.de:9980) downstream server wanted client certificate
but none are configured

The OP had suspected patch r1855918 from PR 63256 as a possible root cause.
Will rebuild 2.4.39 without that patch and recheck.

Regards,

Rainer

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

Rainer Jung <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|proxy client certificates   |proxy client certificates
                   |not found despite being     |not found despite being
                   |configured                  |configured in <proxy>
                   |                            |section

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #11 from Rainer Jung <[hidden email]> ---
One aditional info: using 2.4.38 with mod_proxy and mod_proxy_http from 2.4.39
still works, using 2.4.39 with mod_ssl from 2.4.38 also works. So it is very
likely that the few code changes between 2.4.38 and 2.4.39 in mod_ssl contains
the root cause.

Those changes are:

- r1853197 - very unlikely (FIPS mode handling)
- r1855380 - no (log level change)
- r1855572 - no (code comment)
- r1855917 - unlikely (happens after PHA failure)
- r1855918 - possible and suspected (cleanup per-request SSL configuration for
recycled proxy conns)

I have to note, that I can see the problem even for the initial connection, not
only for recycled ones.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #12 from Rainer Jung <[hidden email]> ---
The culprit really is r1855918.

Originally in ssl_init_connection_ctx() when sslconn was not null, the function
returned early without any changing anything.

The new version always resets sslconn->dc.

By adding some debug logging I can see, that ssl_init_connection_ctx() is run
twice when doing a single curl test request. First it is called from mod_proxy
via ap_proxy_ssl_engine and sets sslconn->dc from per_dir_config. But then it
is called again frm hook_pre_connection and overwrites sslconn->dc from
c->base_server->lookup_defaults, which has no client cert configured.

Not sure how to proceed best. Yann or Rüdiger? Any hints?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #13 from Yann Ylavic <[hidden email]> ---
Created attachment 36589
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=36589&action=edit
Preserve sslconn->dc besides ssl_engine_set() calls

How about this patch?

I think Rüdiger's one is needed for the connection reuse case too. If so, it
probably should be extended to other proxy modules (all the calls to
ap_proxy_connection_create[_ex]() actually), but first let's see if both
patches fix the proxy_http case..

Thanks Rainer and Rüdiger for looking into this anymay.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #14 from Rainer Jung <[hidden email]> ---
Thanks Yann and Rüdiger, both patches together look promising. At least my
simple setup no longer fails when applying on top of 2.4.39.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #15 from [hidden email] ---
I can confirm that in my test environment, using both patches together, the
client certificate is now presented for the proxy backend.



[Thu May 16 10:11:33.353528 2019] [ssl:debug] [pid 2150015:tid 140552047036160]
ssl_engine_kernel.c(1943): AH02267: Proxy client certificate callback:
(dw00050:5000) entered
[Thu May 16 10:11:33.353587 2019] [ssl:debug] [pid 2150015:tid 140552047036160]
ssl_engine_kernel.c(1902): [remote 10.244.4.101:443] AH02277: Proxy client
certificate callback: (dw00050:5000) AH02279: found
acceptable cert, sending [subject:
CN=backend.proxy.corp.com:CCA,O=BB,DC=corp,DC=com / issuer: CN=BB Server TEST
CA 13,OU=PKI,O=BB,C=UN / serial: 03C1E7FF37C922B6
F81075AC / notbefore: Apr 12 06:20:51 2019 GMT / notafter: Oct 12 17:00:00 2019
GMT]
[Thu May 16 10:11:33.353778 2019] [ssl:trace4] [pid 2150015:tid
140552047036160] ssl_engine_io.c(2214): [remote 10.244.4.101:443] OpenSSL:
write 4530/4530 bytes to BIO#7fd4bc035f80 [mem: 7fd4bc046a53]

On this basis, we will move this build to the large-scale non-production
environments.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #16 from Rainer Jung <[hidden email]> ---
Applied to trunk as r1859371. Has to adjust slightly because of differences
between trunk and 2.4.x.

I also proposed backporting r1818726 from trunk to 2.4.x, because that makes
the backport of r1859371 more straightforward (and r1818726 should not be a
problem for 2.4). As soon as r1818726 gets applied to 2.4, I will also suggest
the backport of r1859371.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

[hidden email] changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #17 from [hidden email] ---
Using httpd 2.4.39 with these two patches applied, end users have confirmed
expected behaviours for proxy client certificates are restored and we have no
reports of new issues.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63430] proxy client certificates not found despite being configured in <proxy> section

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63430

--- Comment #18 from [hidden email] ---
Thanks for the fast turnaround on the fixes.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12