TLS Client Hello not responded by mod_ssl

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

TLS Client Hello not responded by mod_ssl

Thierry Moreau
Dear Apache enthusiasts ...

My application is a very simple https-only apache (2.4.43) server with
mod_ssl (openssl 1.1.1g) in Linux (crux distribution 3.5).

Using log levels up to trace8, I am unable to get anything useful past

[...] [ssl:info] [...] [...] AH01964: Connection to child 64 established
(server admin-rachis.connotech-internal.com:443)

The browser-client eventually times out with a "connected to ..." status
in the meantime. The openssl s_client utility supports the hypothesis
that the ServerHello preparation and/or transmission is blocked somehow
in the mod_ssl logic:

openssl s_client -connect admin-rachis.connotech-internal.com:443 -debug

I first get this ClientHello dump:

CONNECTED(00000003)
write to 0x969df0 [0x969e70] (307 bytes => 307 (0x133))
0000 - 16 03 01 01 2e 01 00 01-2a 03 03 37 46 cb ff c9   ........*..7F...
0010 - bc ae 33 6b 32 0f 51 e7-e3 9a 15 e6 67 1b 24 be   ..3k2.Q.....g.$.
0020 - e8 ba e9 b1 5a 56 13 d7-bf 82 71 00 00 94 c0 30   ....ZV....q....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a3 00 9f 00 6b   .,.(.$.........k
0040 - 00 6a 00 39 00 38 00 88-00 87 c0 32 c0 2e c0 2a   .j.9.8.....2...*
0050 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f   .&.......=.5.../
0060 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a2 00 9e 00 67   .+.'.#.........g
0070 - 00 40 00 33 00 32 00 9a-00 99 00 45 00 44 c0 31   .@.3.2.....E.D.1
0080 - c0 2d c0 29 c0 25 c0 0e-c0 04 00 9c 00 3c 00 2f   .-.).%.......<./
0090 - 00 96 00 41 00 07 c0 11-c0 07 c0 0c c0 02 00 05   ...A............
00a0 - 00 04 c0 12 c0 08 00 16-00 13 c0 0d c0 03 00 0a   ................
00b0 - 00 15 00 12 00 09 00 14-00 11 00 08 00 06 00 03   ................
00c0 - 00 ff 01 00 00 6d 00 0b-00 04 03 00 01 02 00 0a   .....m..........
00d0 - 00 34 00 32 00 0e 00 0d-00 19 00 0b 00 0c 00 18   .4.2............
00e0 - 00 09 00 0a 00 16 00 17-00 08 00 06 00 07 00 14   ................
00f0 - 00 15 00 04 00 05 00 12-00 13 00 01 00 02 00 03   ................
0100 - 00 0f 00 10 00 11 00 23-00 00 00 0d 00 20 00 1e   .......#..... ..
0110 - 06 01 06 02 06 03 05 01-05 02 05 03 04 01 04 02   ................
0120 - 04 03 03 01 03 02 03 03-02 01 02 02 02 03 00 0f   ................
0130 - 00 01 01                                          ...

and then nothing ... until I stop the httpd, then openssl s_client exits
after reporting an un-initialized SSL connection:

read from 0x969df0 [0x96f3d0] (7 bytes => -1 (0xFFFFFFFFFFFFFFFF))
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 307 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

==================

That's it for the difficulty I face. I am quite familiar with the TLS
concepts, I did not encounter any configuration where the mod_ssl logic
would complain about something wrong or noticeable in the ClientHello
message. I mean the suspected mod_ssl "freeze" is the systematic outcome
irrespective of the typical SSL/TLS configuration variants. E.g. same
symptoms irrespective of enabling port 80/http with Require all denied
or granted, or moving the CA certificate from SSLCertificateFile to
SSLCACertificateFile.

Except for one [ssl:warn] which I cleared with an SSLSessionCache
configuration line.

[...] [ssl:warn] [...] AH01873: Init: Session Cache is not configured
[hint: SSLSessionCache]

(knowing what breaks the limited success may be useful for troubleshooting).

==================

Concluding:

I Include below (a) the apache configuration file (comment lines
removed), (b) a portion of the apache log, and build-time configuration
for (c) apache and (d) openssl.

My next step is to look at the mod_ssl source code, starting where the
AH01964 log mesage is emitted.

Obviously any hint would be very much appreciated. If I solve the issue,
I should report to the mailing list for the record.

In any event, thanks for this wonderful package!

- Thierry Moreau

================== (a)
ServerRoot "/usr"
Mutex pthread default
LoadModule mpm_event_module lib/apache/mod_mpm_event.so
LoadModule authn_core_module lib/apache/mod_authn_core.so
LoadModule authz_host_module lib/apache/mod_authz_host.so
LoadModule authz_groupfile_module lib/apache/mod_authz_groupfile.so
LoadModule authz_user_module lib/apache/mod_authz_user.so
LoadModule authz_core_module lib/apache/mod_authz_core.so
LoadModule access_compat_module lib/apache/mod_access_compat.so
LoadModule auth_basic_module lib/apache/mod_auth_basic.so
LoadModule socache_shmcb_module lib/apache/mod_socache_shmcb.so
LoadModule reqtimeout_module lib/apache/mod_reqtimeout.so
LoadModule filter_module lib/apache/mod_filter.so
LoadModule mime_module lib/apache/mod_mime.so
LoadModule log_config_module lib/apache/mod_log_config.so
LoadModule env_module lib/apache/mod_env.so
LoadModule headers_module lib/apache/mod_headers.so
LoadModule setenvif_module lib/apache/mod_setenvif.so
LoadModule version_module lib/apache/mod_version.so
LoadModule ssl_module lib/apache/mod_ssl.so
LoadModule unixd_module lib/apache/mod_unixd.so
LoadModule status_module lib/apache/mod_status.so
LoadModule autoindex_module lib/apache/mod_autoindex.so
<IfModule unixd_module>
User www
Group www
</IfModule>
ServerAdmin [hidden email]
Listen 192.168.1.218:443
ServerName admin-rachis.connotech-internal.com
SSLSessionCache shmcb:/var/run/apache-ssl-cycl-buf(512000)
Mutex pthread ssl-cache
SSLRandomSeed startup "file:/dev/urandom" 512
SSLRandomSeed connect "file:/dev/urandom"
<Directory />
     AllowOverride none
     Require all denied
</Directory>
DocumentRoot "/var/www/htdocs"
<VirtualHost *:443>
ServerName admin-rachis.connotech-internal.com
SSLEngine on
SSLCertificateFile "/home/tmoreau/serv_ca/server_files/website.501.pem"
SSLCertificateKeyFile
"/home/tmoreau/serv_ca/server_files/web-server-key.500.pem"
SSLVerifyClient none
SSLCACertificateFile "/etc/apache/server-ca-ss-plus-aixcm-ca-ss.pem"
SSLCARevocationCheck none
SSLOCSPEnable off
SSLSessionTickets off
<Directory "/var/www/htdocs">
     Options +Indexes -FollowSymLinks -ExecCGI -Includes -MultiViews
     AllowOverride None
     SSLRequireSSL
     Require ssl
</Directory>
</VirtualHost>
ErrorLog "/var/log/apache/error_log"
LogLevel info
LogLevel ssl_module:trace7
<IfModule log_config_module>
     LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\"" combined
     LogFormat "%h %l %u %t \"%r\" %>s %b" common
     <IfModule logio_module>
       LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\" %I %O" combinedio
     </IfModule>
     CustomLog "/var/log/apache/access_log" common
</IfModule>
<IfModule alias_module>
     ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
<IfModule cgid_module>
</IfModule>
<Directory "/var/www/cgi-bin">
     AllowOverride None
     Options None
     Require all granted
</Directory>
<IfModule headers_module>
     RequestHeader unset Proxy early
</IfModule>
<IfModule mime_module>
     TypesConfig /etc/apache/mime.types
     AddType application/x-compress .Z
     AddType application/x-gzip .gz .tgz
</IfModule>
================== (b)
[Tue Jul 14 01:10:50.902502 2020] [mpm_event:notice] [pid 28423:tid
139850552182656] AH00489: Apache/2.4.43 (Unix) OpenSSL/1.1.1g configured
-- resuming normal operations
[Tue Jul 14 01:10:50.902512 2020] [mpm_event:info] [pid 28423:tid
139850552182656] AH00490: Server built: May 29 2020 14:50:50
[Tue Jul 14 01:10:50.902526 2020] [core:notice] [pid 28423:tid
139850552182656] AH00094: Command line: 'httpd'
[Tue Jul 14 01:11:28.150888 2020] [ssl:info] [pid 28425:tid
139850538366720] [client 192.168.1.208:35287] AH01964: Connection to
child 64 established (server admin-rachis.connotech-internal.com:443)
[Tue Jul 14 01:11:52.688801 2020] [core:warn] [pid 28423:tid
139850552182656] AH00045: child process 28425 still did not exit,
sending a SIGTERM
[Tue Jul 14 01:11:54.693089 2020] [core:warn] [pid 28423:tid
139850552182656] AH00045: child process 28425 still did not exit,
sending a SIGTERM
[Tue Jul 14 01:11:56.699786 2020] [core:warn] [pid 28423:tid
139850552182656] AH00045: child process 28425 still did not exit,
sending a SIGTERM
[Tue Jul 14 01:11:58.706450 2020] [core:error] [pid 28423:tid
139850552182656] AH00046: child process 28425 still did not exit,
sending a SIGKILL
[Tue Jul 14 01:11:59.709127 2020] [core:info] [pid 28423:tid
139850552182656] AH00096: removed PID file /var/run/apache/httpd.pid
(pid=28423)
[Tue Jul 14 01:11:59.709140 2020] [mpm_event:notice] [pid 28423:tid
139850552182656] AH00491: caught SIGTERM, shutting down
================== (c)
    cat $SRC/crux.layout >> config.layout
    sed -ri '/^(User|Group)/s/daemon/www/' docs/conf/httpd.conf.in

    ./configure --enable-layout=CRUX \
                --with-apr=/usr \
                --with-apr-util=/usr \
                --with-pcre=/usr \
                --enable-so \
                --enable-modules=all \
                --enable-mods-shared=all \
                --enable-mpms-shared=all

httpd -V reports:

Server version: Apache/2.4.43 (Unix)
Server built:   May 29 2020 14:50:50
Server's Module Magic Number: 20120211:92
Server loaded:  APR 1.7.0, APR-UTIL 1.6.1
Compiled using: APR 1.7.0, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     event
   threaded:     yes (fixed thread count)
     forked:     yes (variable process count)
Server compiled with....
  -D APR_HAS_SENDFILE
  -D APR_HAS_MMAP
  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
  -D APR_USE_PROC_PTHREAD_SERIALIZE
  -D APR_USE_PTHREAD_SERIALIZE
  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
  -D APR_HAS_OTHER_CHILD
  -D AP_HAVE_RELIABLE_PIPED_LOGS
  -D DYNAMIC_MODULE_LIMIT=256
  -D HTTPD_ROOT="/usr"
  -D SUEXEC_BIN="/usr/bin/suexec"
  -D DEFAULT_PIDLOG="/var/run/apache/httpd.pid"
  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
  -D DEFAULT_ERRORLOG="logs/error_log"
  -D AP_TYPES_CONFIG_FILE="/etc/apache/mime.types"
  -D SERVER_CONFIG_FILE="/etc/apache/httpd.conf"

#   CRUX layout
<Layout CRUX>
     prefix:          /usr
     exec_prefix:     ${prefix}
     bindir:          ${prefix}/bin
     sbindir:         ${prefix}/sbin
     libdir:          ${prefix}/lib
     libexecdir:      ${prefix}/lib/apache
     installbuilddir: ${prefix}/lib/apache/build
     mandir:          ${prefix}/share/man
     sysconfdir:      /etc/apache
     datadir:         /var/www
     iconsdir:        ${datadir}/icons
     htdocsdir:       ${datadir}/htdocs
     manualdir:       ${htdocsdir}/manual
     cgidir:          ${datadir}/cgi-bin
     errordir:        ${datadir}/error
     includedir:      ${prefix}/include/apache
     localstatedir:   /var
     runtimedir:      ${localstatedir}/run/apache
     logfiledir:      ${localstatedir}/log/apache
     proxycachedir:   ${localstatedir}/cache/apache
</Layout>
================== (d)
     ./config --prefix=/usr \
              --libdir=lib \
              --openssldir=/etc/ssl \
              shared \
              enable-ec_nistp_64_gcc_128
================== -the end-

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TLS Client Hello not responded by mod_ssl

Rainer Canavan-2
> Dear Apache enthusiasts ...
>
> My application is a very simple https-only apache (2.4.43) server with
> mod_ssl (openssl 1.1.1g) in Linux (crux distribution 3.5).

[...]

>     ./configure --enable-layout=CRUX \
>                 --with-apr=/usr \
>                 --with-apr-util=/usr \
>                 --with-pcre=/usr \
>                 --enable-so \
>                 --enable-modules=all \
>                 --enable-mods-shared=all \
>                 --enable-mpms-shared=all

[...]

> ================== (d)
>      ./config --prefix=/usr \
>               --libdir=lib \
>               --openssldir=/etc/ssl \
>               shared \
>               enable-ec_nistp_64_gcc_128
> ================== -the end-

It appears that you're trying to use a custom openssl installation
to build your httpd, but at a casual glance, I haven't seen anything
that would actually make your httpd use that openssl installation.

Make sure that only the correct openssl headers are included during the
build, and that the LD_LIBRARY_PATH, LD_RUN_PATH or preferrably
DT_RUNPATH or DT_RPATH are set so that the matching libraries are
loaded and used (typically using -Wl,-R,/<path>). Also, make sure that
no other modules or libraries  are - possibly indirectly - linked against other
versions of openssl and load those during runtime. Use ldd against all
binaries involved to make sure.

If you want to dig deeper, I'd recommend re-compiling with debug infos (-g),
running with mpm_prefork for simplicity, attaching one httpd process that's
stuck in the ssl handshake and getting a full backtrace (bt full).

rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: TLS Client Hello not responded by mod_ssl

Thierry Moreau
Solved!

The configuration line:

SSLRandomSeed connect "file:/dev/urandom"

was missing a (not-always-)optional byte count as in

SSLRandomSeed connect "file:/dev/urandom" 512

and the mod_ssl software logic was attempting to read to end of file in
function ssl_rand_seed(...).


Thanks to Rainer for feedback, supporting the idea to look at running
code. I did recompile with a few calls to ap_log_error.

The seemingly "custom" installation was not exactly custom, being part
of the Crux source-based Linux distribution.

- Thierry Moreau


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]