Client Cert TLS issue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Client Cert TLS issue

George Stanchev-2
Hi,

I would need some help with tracking an issue with TC 8.5.47 (windows x64, java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple client tries to connect to the server, the server hangs on SSL handshake until either the client times out on read or the server times out (if I set the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I have enabled the client side SSL trace and everything goes well until ECDH key exchange (for brevity I have enabled only one TLS suite "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if requested. The cacerts we use is whatever comes with Azul's Java distro which has ~150 entries + the custom cert for testing. The issue comes from how the connector deals with trusted certs because if I reduce the cacerts to contain only the test certificate, the request is being served without an issue. Also if I remove the tcnative-1.dll from TC there is no issue either.

Perhaps I am missing something. Any help is appreciated.

George

[1]

<Connector
            port="8443" SSLEnabled="true" maxHttpHeaderSize="8192" maxThreads="150" acceptCount="100" enableLookups="false" disableUploadTimeout="true"
            scheme="https" secure="true" clientAuth="true" sslProtocol="TLS" sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2" protocol="org.apache.coyote.http11.Http11NioProtocol"
            keystoreType="JKS" keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit" keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
            ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_SHA_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_SHA_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_256_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256, TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA256, TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384" />

Reply | Threaded
Open this post in threaded view
|

Re: Client Cert TLS issue

markt
On 15/10/2019 22:15, George Stanchev wrote:
> Hi,
>
> I would need some help with tracking an issue with TC 8.5.47 (windows x64, java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple client tries to connect to the server, the server hangs on SSL handshake until either the client times out on read or the server times out (if I set the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I have enabled the client side SSL trace and everything goes well until ECDH key exchange (for brevity I have enabled only one TLS suite "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if requested. The cacerts we use is whatever comes with Azul's Java distro which has ~150 entries + the custom cert for testing. The issue comes from how the connector deals with trusted certs because if I reduce the cacerts to contain only the test certificate, the request is being served without an issue. Also if I remove the tcnative-1.dll from TC there is no issue either.
>
> Perhaps I am missing something. Any help is appreciated.

This sounds like it is repeatable and that you have a system you can
test with. On that basis here are a few things to try:

1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS
handshake is hanging.

2. Try a binary search to try and determine if the issue is the number
of certificates in the trust store or is caused by a specific certificate.

It sounds like there might be an issue converting one or more of the
trusted certs in the trust store to a format OpenSSL can work with.

Mark


>
> George
>
> [1]
>
> <Connector
>             port="8443" SSLEnabled="true" maxHttpHeaderSize="8192" maxThreads="150" acceptCount="100" enableLookups="false" disableUploadTimeout="true"
>             scheme="https" secure="true" clientAuth="true" sslProtocol="TLS" sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2" protocol="org.apache.coyote.http11.Http11NioProtocol"
>             keystoreType="JKS" keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit" keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
>             ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_SHA_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_SHA_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_256_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256, TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA256, TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384" />
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: Client Cert TLS issue

George Stanchev-2

On 15/10/2019 22:15, George Stanchev wrote:
> Hi,
>
> I would need some help with tracking an issue with TC 8.5.47 (windows x64, java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple client tries to connect to the server, the server hangs on SSL handshake until either the client times out on read or the server times out (if I set the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I have enabled the client side SSL trace and everything goes well until ECDH key exchange (for brevity I have enabled only one TLS suite "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if requested. The cacerts we use is whatever comes with Azul's Java distro which has ~150 entries + the custom cert for testing. The issue comes from how the connector deals with trusted certs because if I reduce the cacerts to contain only the test certificate, the request is being served without an issue. Also if I remove the tcnative-1.dll from TC there is no issue either.
>
> Perhaps I am missing something. Any help is appreciated.

<Mark wrote>
This sounds like it is repeatable and that you have a system you can test with. On that basis here are a few things to try:

1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS handshake is hanging.

2. Try a binary search to try and determine if the issue is the number of certificates in the trust store or is caused by a specific certificate.

It sounds like there might be an issue converting one or more of the trusted certs in the trust store to a format OpenSSL can work with.

</Mark wrote>

So the thread dumps didn't prove to be very useful - at least I couldn't see anything. I attached one of them  (dump3.txt). But the trial and error in the binary searched proved it is not any certificate but a magic number of certificates in cacerts - 125 hang and 124 work. I don't know if it is a memory issue or buffer size issue - I tried removing and adding random certs around the 124/125 boundary and which one is out and in doesn't make a difference. I realize that a cert is rather large blob so it still could be memory issue with the size of the cert doesn't matter that much. In 8.5.47-src\java\org\apache\tomcat\util\net\openssl\OpenSSLContext.java#init(kms, tms, sr) we pass the DER-encoded certs to OpenSSL:

                // Pass along the DER encoded certificates of the accepted client
                // certificate issuers, so that their subjects can be presented
                // by the server during the handshake to allow the client choosing
                // an acceptable certificate
                for (X509Certificate caCert : x509TrustManager.getAcceptedIssuers()) {
                    SSLContext.addClientCACertificateRaw(ctx, caCert.getEncoded());
                    if (log.isDebugEnabled())
                        log.debug(sm.getString("openssl.addedClientCaCert", caCert.toString()));
                }

And this is not where it hangs. I stepped through the code through the handshaker but still haven't been able to figure out the hang point as I am not familiar with the details of that portion of TC code . I've attached two cacerts that you can plug in as truststore to try out the issue.

George



>
> George
>
> [1]
>
> <Connector
>             port="8443" SSLEnabled="true" maxHttpHeaderSize="8192" maxThreads="150" acceptCount="100" enableLookups="false" disableUploadTimeout="true"
>             scheme="https" secure="true" clientAuth="true" sslProtocol="TLS" sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2" protocol="org.apache.coyote.http11.Http11NioProtocol"
>             keystoreType="JKS" keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit" keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
>             ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
> TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
> TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
> TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256,
> TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256,
> TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA,
> TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA,
> TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_GCM_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8,
> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_CCM_8,
> TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_256_SHA,
> TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8,
> TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
> TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256,
> TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384,
> TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
> TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384" />
>
>
B‹KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB•È[œÝXœØÜšX™KK[XZ[
ˆ\Ù\œË][œÝXœØÜšX™PÛXØ]
˜\XÚK›Ü™ÃB‘›ÜˆY][Û˜[ÛÛ[X[™ËK[XZ[
ˆ\Ù\œËZ[ÛXØ]
˜\XÚK›Ü™ÃBƒ


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

dump3.txt (119K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Client Cert TLS issue

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

George,

On 10/16/19 12:55, George Stanchev wrote:

>
> On 15/10/2019 22:15, George Stanchev wrote:
>> Hi,
>>
>> I would need some help with tracking an issue with TC 8.5.47
>> (windows x64, java: azul 1.8.0_222) configured with [1] and
>> tcnative-1.dll. When a simple client tries to connect to the
>> server, the server hangs on SSL handshake until either the client
>> times out on read or the server times out (if I set the
>> HttpsURLConnection#setConnectTimeout(0) and
>> ...#setReadTimeout(0)). I have enabled the client side SSL trace
>> and everything goes well until ECDH key exchange (for brevity I
>> have enabled only one TLS suite
>> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug
>> logs if requested. The cacerts we use is whatever comes with
>> Azul's Java distro which has ~150 entries + the custom cert for
>> testing. The issue comes from how the connector deals with
>> trusted certs because if I reduce the cacerts to contain only the
>> test certificate, the request is being served without an issue.
>> Also if I remove the tcnative-1.dll from TC there is no issue
>> either.
>>
>> Perhaps I am missing something. Any help is appreciated.
>
> <Mark wrote> This sounds like it is repeatable and that you have a
> system you can test with. On that basis here are a few things to
> try:
>
> 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS
> handshake is hanging.
>
> 2. Try a binary search to try and determine if the issue is the
> number of certificates in the trust store or is caused by a
> specific certificate.
>
> It sounds like there might be an issue converting one or more of
> the trusted certs in the trust store to a format OpenSSL can work
> with.
>
> </Mark wrote>
>
> So the thread dumps didn't prove to be very useful - at least I
> couldn't see anything.

Me, either.

Can you try again using:

pollerThreadCount="1"
and
acceptorThreadCount="1"

on your <Connector> to see if that changes anything?

Do you notice a spike in CPU when the connection hangs?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl2nbIYACgkQHPApP6U8
pFguXQ//cFd9lOb665cZRC68AB/0p869DBPRN9Cie8EALUgHicYSdEiRCWAUofuJ
MZx8/Sds3mrB965nmfcNLWRQBBkDqVRrN39yqOMumAZeohZq9phUbGT8chsdHD8C
tI8T96bEvKIlOtHoWK9qIYoEKZQ0nKotc0Rz49xZXTNmHmIred4nmR5fCQG4R5qD
xoqXOo5wui3aN7y9VBk+sWBQh0HGpOXvGAHaQ2NbSKE6VTitiiEM92n6Dkz60wnG
QbK2KrflW5E36NdVwNvFkqR/H3WVCrABBJ6puGHAL3nmlhg/n+MTpyqd7nkg1WSU
j1U+hqxN4EHPlTcBUtaeb6DriwaQUIHNMH3h0J8H/UyfvoIRVPA570LF5Cycj7oK
zlVgVmsZIZjIzt+qP6xzKkvhXzPLpemIOheDOZO4opgPdHIXGPAI9XVzwrARxMfv
KeqyA16XrU6pM7GKvkDEnSDiMye/pPGbq/U3mnYdlRs4Lwn9PvmnzBasSXbrsg6i
qeU1v6lSWPx18/9Qq1Qyfjxfgu3SkBpvHypwdv3MNMBk6Y2Gp/pg917FyqfNvoxX
l0TaIYYf5xL6bHsxj1uopUoCnl4KxaTAaQ3qYg4+hFO3nzDNTB1k+Xu+pbuePr3U
CyaIdmM3sqN7fNIwbfQ1slXrckavl+z/ZZmTZn2zIfa1yeMmP4s=
=vkfG
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Client Cert TLS issue

George Stanchev-2
Hi Chris,

This didn't make a difference and no difference in CPU utilization was observed. The lockup happens before server request the client cert so you can drop my big fat cacerts in a TC instance create a client-cert connector, in our case we set the " javax.net.ssl.trustStore" to explicitly point to cacerts and hit it with curl -k https://localhost:8443/

George


-----Original Message-----
From: Christopher Schultz <[hidden email]>
Sent: Wednesday, October 16, 2019 1:16 PM
To: [hidden email]
Subject: Re: Client Cert TLS issue

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

George,

On 10/16/19 12:55, George Stanchev wrote:

>
> On 15/10/2019 22:15, George Stanchev wrote:
>> Hi,
>>
>> I would need some help with tracking an issue with TC 8.5.47 (windows
>> x64, java: azul 1.8.0_222) configured with [1] and tcnative-1.dll.
>> When a simple client tries to connect to the server, the server hangs
>> on SSL handshake until either the client times out on read or the
>> server times out (if I set the
>> HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I
>> have enabled the client side SSL trace and everything goes well until
>> ECDH key exchange (for brevity I have enabled only one TLS suite
>> "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs
>> if requested. The cacerts we use is whatever comes with Azul's Java
>> distro which has ~150 entries + the custom cert for testing. The
>> issue comes from how the connector deals with trusted certs because
>> if I reduce the cacerts to contain only the test certificate, the
>> request is being served without an issue.
>> Also if I remove the tcnative-1.dll from TC there is no issue either.
>>
>> Perhaps I am missing something. Any help is appreciated.
>
> <Mark wrote> This sounds like it is repeatable and that you have a
> system you can test with. On that basis here are a few things to
> try:
>
> 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS
> handshake is hanging.
>
> 2. Try a binary search to try and determine if the issue is the number
> of certificates in the trust store or is caused by a specific
> certificate.
>
> It sounds like there might be an issue converting one or more of the
> trusted certs in the trust store to a format OpenSSL can work with.
>
> </Mark wrote>
>
> So the thread dumps didn't prove to be very useful - at least I
> couldn't see anything.

Me, either.

Can you try again using:

pollerThreadCount="1"
and
acceptorThreadCount="1"

on your <Connector> to see if that changes anything?

Do you notice a spike in CPU when the connection hangs?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl2nbIYACgkQHPApP6U8
pFguXQ//cFd9lOb665cZRC68AB/0p869DBPRN9Cie8EALUgHicYSdEiRCWAUofuJ
MZx8/Sds3mrB965nmfcNLWRQBBkDqVRrN39yqOMumAZeohZq9phUbGT8chsdHD8C
tI8T96bEvKIlOtHoWK9qIYoEKZQ0nKotc0Rz49xZXTNmHmIred4nmR5fCQG4R5qD
xoqXOo5wui3aN7y9VBk+sWBQh0HGpOXvGAHaQ2NbSKE6VTitiiEM92n6Dkz60wnG
QbK2KrflW5E36NdVwNvFkqR/H3WVCrABBJ6puGHAL3nmlhg/n+MTpyqd7nkg1WSU
j1U+hqxN4EHPlTcBUtaeb6DriwaQUIHNMH3h0J8H/UyfvoIRVPA570LF5Cycj7oK
zlVgVmsZIZjIzt+qP6xzKkvhXzPLpemIOheDOZO4opgPdHIXGPAI9XVzwrARxMfv
KeqyA16XrU6pM7GKvkDEnSDiMye/pPGbq/U3mnYdlRs4Lwn9PvmnzBasSXbrsg6i
qeU1v6lSWPx18/9Qq1Qyfjxfgu3SkBpvHypwdv3MNMBk6Y2Gp/pg917FyqfNvoxX
l0TaIYYf5xL6bHsxj1uopUoCnl4KxaTAaQ3qYg4+hFO3nzDNTB1k+Xu+pbuePr3U
CyaIdmM3sqN7fNIwbfQ1slXrckavl+z/ZZmTZn2zIfa1yeMmP4s=
=vkfG
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Client Cert TLS issue

markt
In reply to this post by George Stanchev-2
Just a note to say I haven't forgotten this. I hope to look at this this
week unless someone beats me to it.

Mark


On 16/10/2019 17:55, George Stanchev wrote:

>
> On 15/10/2019 22:15, George Stanchev wrote:
>> Hi,
>>
>> I would need some help with tracking an issue with TC 8.5.47 (windows x64, java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple client tries to connect to the server, the server hangs on SSL handshake until either the client times out on read or the server times out (if I set the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I have enabled the client side SSL trace and everything goes well until ECDH key exchange (for brevity I have enabled only one TLS suite "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if requested. The cacerts we use is whatever comes with Azul's Java distro which has ~150 entries + the custom cert for testing. The issue comes from how the connector deals with trusted certs because if I reduce the cacerts to contain only the test certificate, the request is being served without an issue. Also if I remove the tcnative-1.dll from TC there is no issue either.
>>
>> Perhaps I am missing something. Any help is appreciated.
>
> <Mark wrote>
> This sounds like it is repeatable and that you have a system you can test with. On that basis here are a few things to try:
>
> 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS handshake is hanging.
>
> 2. Try a binary search to try and determine if the issue is the number of certificates in the trust store or is caused by a specific certificate.
>
> It sounds like there might be an issue converting one or more of the trusted certs in the trust store to a format OpenSSL can work with.
>
> </Mark wrote>
>
> So the thread dumps didn't prove to be very useful - at least I couldn't see anything. I attached one of them  (dump3.txt). But the trial and error in the binary searched proved it is not any certificate but a magic number of certificates in cacerts - 125 hang and 124 work. I don't know if it is a memory issue or buffer size issue - I tried removing and adding random certs around the 124/125 boundary and which one is out and in doesn't make a difference. I realize that a cert is rather large blob so it still could be memory issue with the size of the cert doesn't matter that much. In 8.5.47-src\java\org\apache\tomcat\util\net\openssl\OpenSSLContext.java#init(kms, tms, sr) we pass the DER-encoded certs to OpenSSL:
>
>                 // Pass along the DER encoded certificates of the accepted client
>                 // certificate issuers, so that their subjects can be presented
>                 // by the server during the handshake to allow the client choosing
>                 // an acceptable certificate
>                 for (X509Certificate caCert : x509TrustManager.getAcceptedIssuers()) {
>                     SSLContext.addClientCACertificateRaw(ctx, caCert.getEncoded());
>                     if (log.isDebugEnabled())
>                         log.debug(sm.getString("openssl.addedClientCaCert", caCert.toString()));
>                 }
>
> And this is not where it hangs. I stepped through the code through the handshaker but still haven't been able to figure out the hang point as I am not familiar with the details of that portion of TC code . I've attached two cacerts that you can plug in as truststore to try out the issue.
>
> George
>
>
>
>>
>> George
>>
>> [1]
>>
>> <Connector
>>             port="8443" SSLEnabled="true" maxHttpHeaderSize="8192" maxThreads="150" acceptCount="100" enableLookups="false" disableUploadTimeout="true"
>>             scheme="https" secure="true" clientAuth="true" sslProtocol="TLS" sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2" protocol="org.apache.coyote.http11.Http11NioProtocol"
>>             keystoreType="JKS" keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit" keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
>>             ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
>> TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
>> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
>> TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
>> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
>> TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
>> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA,
>> TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA,
>> TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8,
>> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_CCM_8,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_256_SHA,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8,
>> TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
>> TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256,
>> TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384,
>> TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384" />
>>
>>
>
> B‹KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB•È[œÝXœØÜšX™KK[XZ[
> ˆ\Ù\œË][œÝXœØÜšX™PÛXØ]
> ˜\XÚK›Ü™ÃB‘›ÜˆY][Û˜[ÛÛ[X[™ËK[XZ[
> ˆ\Ù\œËZ[ÛXØ]
> ˜\XÚK›Ü™ÃBƒ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Client Cert TLS issue

George Stanchev-2
Thanks Mark,

I was going to try to build tcnative with debug this weekend to try to help out but life begged to disagree...i will try to step in that code next week as well...

Appreciate it!

George

-----Original Message-----
From: Mark Thomas <[hidden email]>
Sent: Sunday, October 20, 2019 2:38 PM
To: [hidden email]
Subject: Re: Client Cert TLS issue

Just a note to say I haven't forgotten this. I hope to look at this this week unless someone beats me to it.

Mark


On 16/10/2019 17:55, George Stanchev wrote:

>
> On 15/10/2019 22:15, George Stanchev wrote:
>> Hi,
>>
>> I would need some help with tracking an issue with TC 8.5.47 (windows x64, java: azul 1.8.0_222) configured with [1] and tcnative-1.dll. When a simple client tries to connect to the server, the server hangs on SSL handshake until either the client times out on read or the server times out (if I set the HttpsURLConnection#setConnectTimeout(0) and ...#setReadTimeout(0)). I have enabled the client side SSL trace and everything goes well until ECDH key exchange (for brevity I have enabled only one TLS suite "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"). I can provide the debug logs if requested. The cacerts we use is whatever comes with Azul's Java distro which has ~150 entries + the custom cert for testing. The issue comes from how the connector deals with trusted certs because if I reduce the cacerts to contain only the test certificate, the request is being served without an issue. Also if I remove the tcnative-1.dll from TC there is no issue either.
>>
>> Perhaps I am missing something. Any help is appreciated.
>
> <Mark wrote>
> This sounds like it is repeatable and that you have a system you can test with. On that basis here are a few things to try:
>
> 1. Take a 3 thread dumps ~5s apart of the Tomcat process when TLS handshake is hanging.
>
> 2. Try a binary search to try and determine if the issue is the number of certificates in the trust store or is caused by a specific certificate.
>
> It sounds like there might be an issue converting one or more of the trusted certs in the trust store to a format OpenSSL can work with.
>
> </Mark wrote>
>
> So the thread dumps didn't prove to be very useful - at least I couldn't see anything. I attached one of them  (dump3.txt). But the trial and error in the binary searched proved it is not any certificate but a magic number of certificates in cacerts - 125 hang and 124 work. I don't know if it is a memory issue or buffer size issue - I tried removing and adding random certs around the 124/125 boundary and which one is out and in doesn't make a difference. I realize that a cert is rather large blob so it still could be memory issue with the size of the cert doesn't matter that much. In 8.5.47-src\java\org\apache\tomcat\util\net\openssl\OpenSSLContext.java#init(kms, tms, sr) we pass the DER-encoded certs to OpenSSL:
>
>                 // Pass along the DER encoded certificates of the accepted client
>                 // certificate issuers, so that their subjects can be presented
>                 // by the server during the handshake to allow the client choosing
>                 // an acceptable certificate
>                 for (X509Certificate caCert : x509TrustManager.getAcceptedIssuers()) {
>                     SSLContext.addClientCACertificateRaw(ctx, caCert.getEncoded());
>                     if (log.isDebugEnabled())
>                         log.debug(sm.getString("openssl.addedClientCaCert", caCert.toString()));
>                 }
>
> And this is not where it hangs. I stepped through the code through the handshaker but still haven't been able to figure out the hang point as I am not familiar with the details of that portion of TC code . I've attached two cacerts that you can plug in as truststore to try out the issue.
>
> George
>
>
>
>>
>> George
>>
>> [1]
>>
>> <Connector
>>             port="8443" SSLEnabled="true" maxHttpHeaderSize="8192" maxThreads="150" acceptCount="100" enableLookups="false" disableUploadTimeout="true"
>>             scheme="https" secure="true" clientAuth="true" sslProtocol="TLS" sslEnabledProtocols="+TLSv1 +TLSv1.1 +TLSv1.2" protocol="org.apache.coyote.http11.Http11NioProtocol"
>>             keystoreType="JKS" keystoreFile="${server.conf.dir}/serena.keystore" keystorePass="changeit" keyAlias="jboss" URIEncoding="UTF-8" useServerCipherSuitesOrder="true"
>>             ciphers="TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384,
>> TLS_CHACHA20_POLY1305_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
>> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
>> TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
>> TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
>> TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
>> TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
>> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256,
>> TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA256,
>> TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA,
>> TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA,
>> TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_GCM_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_SHA_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8,
>> TLS_ECDHE_ECDSA_WITH_AES_256_GCM_CCM_8,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_256_SHA,
>> TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8,
>> TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
>> TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256,
>> TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384,
>> TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256,
>> TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384" />
>>
>>
>
> B‹KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
> KCB•È[œÝXœØÜšX™KK[XZ[
> ˆ\Ù\œË][œÝXœØÜšX™PÛXØ]
> ˜\XÚK›Ü™ÃB‘›ÜˆY][Û˜[ÛÛ[X[™ËK[XZ[
> ˆ\Ù\œËZ[ÛXØ]
> ˜\XÚK›Ü™ÃBƒ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Client Cert TLS issue

markt
In reply to this post by George Stanchev-2
On 16/10/2019 18:55, George Stanchev wrote:

> And this is not where it hangs. I stepped through the code through the handshaker but still haven't been able to figure out the hang point as I am not familiar with the details of that portion of TC code . I've attached two cacerts that you can plug in as truststore to try out the issue.

George,

The mailing list strips most attachments. Please open a BZ issue for
this. One, that means it won't get forgotten and two you will be able to
attach those cacert files so we can easily reproduce the issue you are
seeing.

Thanks,

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Client Cert TLS issue

George Stanchev-2
Thanks Mark, will do!

-----Original Message-----
From: Mark Thomas <[hidden email]>
Sent: Thursday, October 31, 2019 3:04 PM
To: Tomcat Users List <[hidden email]>; George Stanchev <[hidden email]>
Subject: Re: Client Cert TLS issue

On 16/10/2019 18:55, George Stanchev wrote:

> And this is not where it hangs. I stepped through the code through the handshaker but still haven't been able to figure out the hang point as I am not familiar with the details of that portion of TC code . I've attached two cacerts that you can plug in as truststore to try out the issue.

George,

The mailing list strips most attachments. Please open a BZ issue for this. One, that means it won't get forgotten and two you will be able to attach those cacert files so we can easily reproduce the issue you are seeing.

Thanks,

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]