NullPointerExceptions from Coyote over SSL

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

NullPointerExceptions from Coyote over SSL

Peter Robbins
Hi there,

Versions: Tomcat 8.5.3, JDK 1.8 + JCE, Bouncy Castle 1.48, Ubuntu 14.04 & 16.04,Windows 2012 R2

I’m running into an issue where we are getting NullPointerExceptions from the Coyote connector in a Tomcat web application.

This is an existing, stable web application that was recently upgraded to Tomcat 8.5.3. When we repeatedly send requests over HTTPS eventually (after 30 or so rapid requests) the web app becomes largely unresponsive and the catalina log fills up with errors like this:

-------- BEGIN LOG --------

18-Jul-2016 10:47:30.066 WARNING [Tomcat-7] org.apache.tomcat.util.net.AbstractEndpoint.countDownConnection Incorrect connection count, multiple socket.close called on the same socket.

18-Jul-2016 10:47:36.707 INFO [Tomcat-6] org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
 java.lang.IllegalStateException: Unexpected state: headers already parsed. Buffer not recycled?
        at org.apache.coyote.http11.Http11InputBuffer.parseHeaders(Http11InputBuffer.java:587)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1010)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

18-Jul-2016 10:47:36.815 SEVERE [Tomcat-12] org.apache.coyote.http11.Http11Processor.service Error processing request
 java.lang.NullPointerException
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:389)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1110)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

18-Jul-2016 10:47:36.816 SEVERE [Tomcat-12] org.apache.coyote.http11.Http11Processor.endRequest Error finishing response
 java.lang.NullPointerException
        at org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:533)
        at org.apache.coyote.http11.Http11OutputBuffer.endRequest(Http11OutputBuffer.java:318)
        at org.apache.coyote.http11.Http11Processor.endRequest(Http11Processor.java:1794)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1149)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

18-Jul-2016 10:56:54.739 INFO [https-jsse-nio-8443-ClientPoller-1] org.apache.catalina.connector.CoyoteAdapter.checkRecycled Encountered a non-recycled request and recycled it forcedly.
 org.apache.catalina.connector.CoyoteAdapter$RecycleRequiredException
        at org.apache.catalina.connector.CoyoteAdapter.checkRecycled(CoyoteAdapter.java:494)
        at org.apache.coyote.http11.Http11Processor.recycle(Http11Processor.java:1840)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:955)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.release(AbstractProtocol.java:976)
        at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:730)
        at org.apache.tomcat.util.net.NioEndpoint.close(NioEndpoint.java:505)
        at org.apache.tomcat.util.net.NioEndpoint.access$600(NioEndpoint.java:69)
        at org.apache.tomcat.util.net.NioEndpoint$Poller.processSendfile(NioEndpoint.java:964)
        at org.apache.tomcat.util.net.NioEndpoint$Poller.processKey(NioEndpoint.java:844)
        at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:826)
        at java.lang.Thread.run(Thread.java:745)

18-Jul-2016 10:59:13.195 SEVERE [Tomcat-23] org.apache.coyote.http11.Http11Processor.endRequest Error finishing response
 java.lang.IllegalArgumentException
        at java.nio.Buffer.position(Buffer.java:244)
        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:343)
        at sun.nio.ch.IOUtil.write(IOUtil.java:60)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:143)
        at org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:632)
        at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
        at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1241)
        at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:428)
        at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:418)
        at org.apache.coyote.http11.Http11OutputBuffer.flushBuffer(Http11OutputBuffer.java:533)
        at org.apache.coyote.http11.Http11OutputBuffer.endRequest(Http11OutputBuffer.java:318)
        at org.apache.coyote.http11.Http11Processor.endRequest(Http11Processor.java:1794)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:1149)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:785)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1425)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

-------- END LOG --------

We’ve looked for the usual suspects like referencing an old ServletRequest or ServletResponse and things like that, but I’m skeptical it’d be something like that simply because the web application works without issue over HTTP and continues to work without issue at the same time that we’re seeing this over HTTPS. Requests over port 8080 go without failure, but over 8443 we see these errors in the server log and on the client side things like net::ERR_CONTENT_LENGTH_MISMATCH and net::ERR_SSL_PROTOCOL_ERROR (from Chrome).

Particularly due to the “Incorrect connection count” warning, it seems like Tomcat may be recycling a request or response object twice or attempting to operate on a recycled object.

I realize this isn’t exact reproduction steps, but I’m really starting to run out of places to look for a root cause, much less a solution to this. Does anyone have any idea where I should look? I’ve included the server.xml below.

-------- BEGIN server.xml --------
<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
--><!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 --><Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <!-- Security listener. Documentation at /docs/config/listeners.html
  <Listener className="org.apache.catalina.security.SecurityListener" />
  -->
  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container",
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <Executor name="tomcatThreadPool" namePrefix="Tomcat-" />

    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
    -->
    <Connector URIEncoding="UTF-8" port="8080" executor="tomcatThreadPool" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    -->
    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
         This connector uses the NIO implementation with the JSSE engine. When
         using the JSSE engine, the JSSE configuration attributes must be used.
    -->
    <!--
    <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true">
        <SSLHostConfig>
            <Certificate certificateKeystoreFile="conf/localhost-rsa.jks"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>
    -->
    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2
         This connector uses the APR/native implementation. When using the
         APR/native implementation or the OpenSSL engine with NIO or NIO2 then
         the OpenSSL configuration attributes must be used.
    -->
    <!--
    <Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="150" SSLEnabled="true" >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
        <SSLHostConfig>
            <Certificate certificateKeyFile="conf/localhost-rsa-key.pem"
                         certificateFile="conf/localhost-rsa-cert.pem"
                         certificateChainFile="conf/localhost-rsa-chain.pem"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>
    -->
    <Connector URIEncoding="UTF-8" server="Apache" port="8443" executor="tomcatThreadPool" SSLEnabled="true" maxPostSize="-1" scheme="https" protocol="HTTP/1.1" secure="true" clientAuth="false" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2,TLSv1.1,TLSv1" keystoreFile="/usr/secret" keystorePass="1234" ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA"></Connector>

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector URIEncoding="UTF-8" port="8009" protocol="AJP/1.3" redirectPort="8443" />


    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

      <!-- Use the LockOutRealm to prevent attempts to guess user passwords
           via a brute-force attack -->
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <!-- This Realm uses the UserDatabase configured in the global JNDI
             resources under the key "UserDatabase".  Any edits
             that are performed against this UserDatabase are immediately
             available for use by the Realm.  -->
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase" />
      </Realm>

      <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true">

        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common" -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" resolveHosts="false"/>
        -->

      </Host>
    </Engine>
  </Service>
</Server>
-------- END server.xml --------



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

remm
2016-07-19 23:51 GMT+02:00 Peter Robbins <[hidden email]>:

> Hi there,
>
> JCE, Bouncy Castle 1.48
>
> Maybe try without that first.

Rémy
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

Peter Robbins
In reply to this post by Peter Robbins
Without JCE or BC? Both are pretty critical for core functionality and didn't cause any issues until 8.5.3 entered the mix. Any known issues there I should be aware of?

Peter

On Jul 19, 2016 6:24 PM, R?my Maucherat <[hidden email]> wrote:
2016-07-19 23:51 GMT+02:00 Peter Robbins <[hidden email]>:

> Hi there,
>
> JCE, Bouncy Castle 1.48
>
> Maybe try without that first.

R?my

Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

remm
2016-07-20 2:54 GMT+02:00 Peter Robbins <[hidden email]>:

> Without JCE or BC? Both are pretty critical for core functionality and
> didn't cause any issues until 8.5.3 entered the mix. Any known issues there
> I should be aware of?
>

You still need to test something. You don't describe anything out of the
ordinary other than that and we don't get issues at all with 8.5, so an
investigation has to start somewhere.

Why is BC critical for core functionality ? JSSE doesn't work for you ? Is
tomcat-native installed [OpenSSL could be used if that is the case] ?

Rémy
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

Peter Robbins
In reply to this post by Peter Robbins
Ok I'll see if I can dig BC out of the application and have it actually start up to try to see if that's the case.

You're saying there are known compatibility issues with Tomcat NIO https if you register another j2ee security provider? The errors we're seeing don't seem crypto related apart from only appearing over https.

On Jul 20, 2016 1:56 AM, R?my Maucherat <[hidden email]> wrote:
2016-07-20 2:54 GMT+02:00 Peter Robbins <[hidden email]>:

> Without JCE or BC? Both are pretty critical for core functionality and
> didn't cause any issues until 8.5.3 entered the mix. Any known issues there
> I should be aware of?
>

You still need to test something. You don't describe anything out of the
ordinary other than that and we don't get issues at all with 8.5, so an
investigation has to start somewhere.

Why is BC critical for core functionality ? JSSE doesn't work for you ? Is
tomcat-native installed [OpenSSL could be used if that is the case] ?

R?my

Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

remm
2016-07-20 13:59 GMT+02:00 Peter Robbins <[hidden email]>:

> Ok I'll see if I can dig BC out of the application and have it actually
> start up to try to see if that's the case.
>
> You're saying there are known compatibility issues with Tomcat NIO https
> if you register another j2ee security provider?


No, but we're not testing that kind of configuration at all. Since there's
OpenSSL support, it seems less important now as well.


> The errors we're seeing don't seem crypto related apart from only
> appearing over https.
>

Rémy
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

Peter Robbins
That’s good to know. In the short term I think we’re going to revert back to the 8.0.x branch and see if we can find put together a more isolated repro war to try to nail this thing down. Thanks for your help!

Peter

On 7/20/16, 7:40 AM, "Rémy Maucherat" <[hidden email]> wrote:

2016-07-20 13:59 GMT+02:00 Peter Robbins <[hidden email]>:

> Ok I'll see if I can dig BC out of the application and have it actually
> start up to try to see if that's the case.
>
> You're saying there are known compatibility issues with Tomcat NIO https
> if you register another j2ee security provider?


No, but we're not testing that kind of configuration at all. Since there's
OpenSSL support, it seems less important now as well.


> The errors we're seeing don't seem crypto related apart from only
> appearing over https.
>

Rémy



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

Peter Robbins
Just to update, we were able to work around this by changing our server.xml connector config from:

    protocol="HTTP/1.1"
to:
    protocol="org.apache.coyote.http11.Http11Nio2Protocol" sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"

Somewhere deep within Http11NioProtocol there is a bug that is fixed in Http11Nio2Protocol. Unfortunately, we don’t have the bandwidth to try to isolate it further, though I will update if anything else is uncovered.

Thanks,
Peter

On 7/20/16, 11:13 AM, "Peter Robbins" <[hidden email]> wrote:

That’s good to know. In the short term I think we’re going to revert back to the 8.0.x branch and see if we can find put together a more isolated repro war to try to nail this thing down. Thanks for your help!

Peter

On 7/20/16, 7:40 AM, "Rémy Maucherat" <[hidden email]> wrote:

2016-07-20 13:59 GMT+02:00 Peter Robbins <[hidden email]>:

> Ok I'll see if I can dig BC out of the application and have it actually
> start up to try to see if that's the case.
>
> You're saying there are known compatibility issues with Tomcat NIO https
> if you register another j2ee security provider?


No, but we're not testing that kind of configuration at all. Since there's
OpenSSL support, it seems less important now as well.


> The errors we're seeing don't seem crypto related apart from only
> appearing over https.
>

Rémy





---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

remm
2016-07-22 22:16 GMT+02:00 Peter Robbins <[hidden email]>:

> Just to update, we were able to work around this by changing our
> server.xml connector config from:
>
>     protocol="HTTP/1.1"
> to:
>     protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
>
> Somewhere deep within Http11NioProtocol there is a bug that is fixed in
> Http11Nio2Protocol. Unfortunately, we don’t have the bandwidth to try to
> isolate it further, though I will update if anything else is uncovered.
>
> You are potentially changing two things at the same time here. You
were/are using boutycastle. If you also have tomcat-native installed,
Tomcat would try to use OpenSSL with JSSE. I don't have any idea how that
interacts with boutycastle, se we're probably not supporting it (it is
never tested, and now we provide OpenSSL over which we have some control
and basically does the same thing in a better way).

Rémy
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

Peter Robbins
> If you also have tomcat-native installed…
No tomcat-native in any environment I saw, but I’ll make sure we check on that config.

We’re not knowingly plugging Bouncy Castle into the Tomcat SSL mix at all. We only use it in application logic after registering it with Security.addProvider() in a context listener. We then only ever access the BouncyCastle Provider by getting it by name, so not too sure what it would have to do with the SSL implementation.

We didn’t add any configuration to specify any value for sslImplementationName previously, so it should have just been using org.apache.tomcat.util.net.jsse.JSSEImplementation anyway. Being a JCE implementation, Bouncy Castle Doesn’t provide an SSL implementation, so I’m not sure how that could get mixed in at all.

I wish I could add more that we found, but at this point I’m just updating the list so that maybe someone else can work around the same thing we have. Thanks for the help!

Peter

On 7/25/16, 3:29 PM, "Rémy Maucherat" <[hidden email]> wrote:

>You are potentially changing two things at the same time here. You
>were/are using boutycastle. If you also have tomcat-native installed,
>Tomcat would try to use OpenSSL with JSSE. I don't have any idea how that
>interacts with boutycastle, se we're probably not supporting it (it is
>never tested, and now we provide OpenSSL over which we have some control
>and basically does the same thing in a better way).


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: NullPointerExceptions from Coyote over SSL

remm
2016-07-25 23:25 GMT+02:00 Peter Robbins <[hidden email]>:

> > If you also have tomcat-native installed…
> No tomcat-native in any environment I saw, but I’ll make sure we check on
> that config.
>
> We’re not knowingly plugging Bouncy Castle into the Tomcat SSL mix at all.
> We only use it in application logic after registering it with
> Security.addProvider() in a context listener. We then only ever access the
> BouncyCastle Provider by getting it by name, so not too sure what it would
> have to do with the SSL implementation.
>
> We didn’t add any configuration to specify any value for
> sslImplementationName previously, so it should have just been using
> org.apache.tomcat.util.net.jsse.JSSEImplementation anyway. Being a JCE
> implementation, Bouncy Castle Doesn’t provide an SSL implementation, so I’m
> not sure how that could get mixed in at all.
>
> I wish I could add more that we found, but at this point I’m just updating
> the list so that maybe someone else can work around the same thing we have.
> Thanks for the help!
>

Ok. If you're not using OpenSSL through tomcat-native, you don't need to
add
sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
then, it is the fallback.

Rémy


>
> Peter
>
> On 7/25/16, 3:29 PM, "Rémy Maucherat" <[hidden email]> wrote:
>
> >You are potentially changing two things at the same time here. You
> >were/are using boutycastle. If you also have tomcat-native installed,
> >Tomcat would try to use OpenSSL with JSSE. I don't have any idea how that
> >interacts with boutycastle, se we're probably not supporting it (it is
> >never tested, and now we provide OpenSSL over which we have some control
> >and basically does the same thing in a better way).
>
>
Reply | Threaded
Open this post in threaded view
|

RE: NullPointerExceptions from Coyote over SSL

George Stanchev
In reply to this post by Peter Robbins
Peter,

Depending at which slot you plug in BC in the Security context it might or it might not get used depending on the cipher suites used by you SSL connection. JSSE will ask Java for crypto implementation from the list of JCE providers and if your BC is high on the list, it will get used. It definitely can affect your SSL if you're using JSSE+JCE...

George

-----Original Message-----
From: Peter Robbins [mailto:[hidden email]]
Sent: Monday, July 25, 2016 3:25 PM
To: Tomcat Users List <[hidden email]>
Subject: Re: NullPointerExceptions from Coyote over SSL

> If you also have tomcat-native installed…
No tomcat-native in any environment I saw, but I’ll make sure we check on that config.

We’re not knowingly plugging Bouncy Castle into the Tomcat SSL mix at all. We only use it in application logic after registering it with Security.addProvider() in a context listener. We then only ever access the BouncyCastle Provider by getting it by name, so not too sure what it would have to do with the SSL implementation.

We didn’t add any configuration to specify any value for sslImplementationName previously, so it should have just been using org.apache.tomcat.util.net.jsse.JSSEImplementation anyway. Being a JCE implementation, Bouncy Castle Doesn’t provide an SSL implementation, so I’m not sure how that could get mixed in at all.

I wish I could add more that we found, but at this point I’m just updating the list so that maybe someone else can work around the same thing we have. Thanks for the help!

Peter

On 7/25/16, 3:29 PM, "Rémy Maucherat" <[hidden email]> wrote:

>You are potentially changing two things at the same time here. You
>were/are using boutycastle. If you also have tomcat-native installed,
>Tomcat would try to use OpenSSL with JSSE. I don't have any idea how
>that interacts with boutycastle, se we're probably not supporting it
>(it is never tested, and now we provide OpenSSL over which we have some
>control and basically does the same thing in a better way).


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]