Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Adam Feder
Hi --

While testing 9.0.8 in staging, we found that when a client connects to
our Tomcat's TLS connector and doesn't send any data (not even to do
the handshake), the connection is allowed to stay open indefinitely.

We first saw this problem on Debian.  We have been able to reproduce
it using the "Docker Official Image" for Tomcat.

Here is a recipe to reproduce it.  First, it shows the expected
behavior (which we get with the NIO handler) and then the unexpected
behavior (which we get with the NIO2 handler).

  (1) Run a tomcat docker container:
      docker run -it tomcat:9.0.8 /bin/bash

  (2) Install telnet
      apt update
      apt install telnet

  (3) Generate an RSA certificate
      keytool -genkey -alias tomcat -keyalg RSA
      (set password to 'password' and then take the defaults)

  (4) Edit /usr/local/tomcat/conf/server.xml
      uncomment the SSL/TLS HTTP/1.1 Connector for port 8443 (around line
88)
      set certificateKeystoreFile="/root/.keystore"
      set certificateKeystorePassword="password"

  (5) Start tomcat:
      /usr/local/tomcat/bin/startup.sh

  (6) Connect to the SSL/TLS port:
      telnet 127.0.0.1 8443

  (7) Note that the connection is broken in 60 seconds (that's the default
connectionTimeout).

  (8) Stop tomcat:
      /usr/local/tomcat/bin/shutdown.sh

  (9) Edit /usr/local/tomcat/conf/server.xml to use NIO2 instead.
      set protocol="org.apache.coyote.http11.Http11Nio2Protocol" (note the
*2*!)

 (10) Start tomcat:
      /usr/local/tomcat/bin/startup.sh

 (11) Connect to the SSL/TLS port:
      telnet 127.0.0.1 8443

 (12) The server never appears to disconnect the connection.

Note that if the client goes through the handshake, the server will
hang up on it as expected.

 (13) Make sure openssl is installed
      apt install openssl

 (14) openssl s_client -connect localhost:8443

 (15) Note that it disconnects in about a minute.

Using docker (18.03.1-ce-mac65 (24312)) on a mid-2012 Mac Book Pro
Retina (macOs 10.13.4), I've tried this with 9.0.8 and 8.5.31.  On
similar laptops and OSes we've also tried 9.0.7.

I've run it a little my Intellij IDEA with breakpoints in the code to
watch the Acceptor accept the connection and I don't see any timers
being set during the handshake phase, but i could have easily missed
them.

Is this behavior expected?  Is there a configuration parameter we're
missing?

thanks,
ab



-----------------------------------------------------------------------------------
| /usr/local/tomcat/conf/server.xml from tomcat:9.0.8n with edits described
above
-----------------------------------------------------------------------------------

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <!-- Security listener. Documentation at /docs/config/listeners.html
  <Listener className="org.apache.catalina.security.SecurityListener" />
  -->
  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container",
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">

    <!--The connectors can use a shared executor, you can define one or
more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
        maxThreads="150" minSpareThreads="4"/>
    -->


    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />
    -->
    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443
         This connector uses the NIO implementation. The default
         SSLImplementation will depend on the presence of the APR/native
         library and the useOpenSSL attribute of the
         AprLifecycleListener.
         Either JSSE or OpenSSL style configuration may be used regardless
of
         the SSLImplementation selected. JSSE style configuration is used
below.
    -->
    <!-- -->
    <Connector port="8443"
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
               maxThreads="150" SSLEnabled="true">
        <SSLHostConfig>
          <Certificate certificateKeystoreFile="/root/.keystore"
       certificateKeystorePassword="password"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>
    <!-- -->
    <!-- Define a SSL/TLS HTTP/1.1 Connector on port 8443 with HTTP/2
         This connector uses the APR/native implementation which always uses
         OpenSSL for TLS.
         Either JSSE or OpenSSL style configuration may be used. OpenSSL
style
         configuration is used below.
    -->
    <!--
    <Connector port="8443"
protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="150" SSLEnabled="true" >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
/>
        <SSLHostConfig>
            <Certificate certificateKeyFile="conf/localhost-rsa-key.pem"
                         certificateFile="conf/localhost-rsa-cert.pem"
                         certificateChainFile="conf/localhost-rsa-chain.pem"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />


    <!-- An Engine represents the entry point (within Catalina) that
processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes
them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->

      <!-- Use the LockOutRealm to prevent attempts to guess user passwords
           via a brute-force attack -->
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <!-- This Realm uses the UserDatabase configured in the global JNDI
             resources under the key "UserDatabase".  Any edits
             that are performed against this UserDatabase are immediately
             available for use by the Realm.  -->
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase"/>
      </Realm>

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

        <!-- SingleSignOn valve, share authentication between web
applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html
             Note: The pattern used is equivalent to using pattern="common"
-->
        <Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

      </Host>
    </Engine>
  </Service>
</Server>
Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Rémy Maucherat
On Wed, May 30, 2018 at 5:49 AM Adam Feder <[hidden email]> wrote:

> I've run it a little my Intellij IDEA with breakpoints in the code to
> watch the Acceptor accept the connection and I don't see any timers
> being set during the handshake phase, but i could have easily missed
> them.
>
> Is this behavior expected?  Is there a configuration parameter we're
> missing?
>

I have now added the timeout during handshake.

Rémy
Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Adam Feder
On Wed, May 30, 2018 at 1:34 AM, Rémy Maucherat <[hidden email]> wrote:

>
> I have now added the timeout during handshake.
>

Thanks for the quick fix, Rémy!

I haven't figured out what the release cycle is for 9.0.x releases.  Do you
know when the fix will be available in an official release?

thanks!
ab
Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Rémy Maucherat
On Wed, May 30, 2018 at 6:24 PM Adam Feder <[hidden email]> wrote:

> On Wed, May 30, 2018 at 1:34 AM, Rémy Maucherat <[hidden email]> wrote:
>
> >
> > I have now added the timeout during handshake.
> >
>
> Thanks for the quick fix, Rémy!
>
> I haven't figured out what the release cycle is for 9.0.x releases.  Do you
> know when the fix will be available in an official release?
>

It will be included in 9.0.9.

Rémy
Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Adam,

On 5/30/18 12:33 PM, Rémy Maucherat wrote:

> On Wed, May 30, 2018 at 6:24 PM Adam Feder <[hidden email]>
> wrote:
>
>> On Wed, May 30, 2018 at 1:34 AM, Rémy Maucherat <[hidden email]>
>> wrote:
>>
>>>
>>> I have now added the timeout during handshake.
>>>
>>
>> Thanks for the quick fix, Rémy!
>>
>> I haven't figured out what the release cycle is for 9.0.x
>> releases.  Do you know when the fix will be available in an
>> official release?
>>
>
> It will be included in 9.0.9.

More specifically, the current releases are usually released about
once per month give or take a few weeks. You shouldn't have to wait
very long, or you could patch your own build if you'd like.

Looks like r1832519 is what you are looking.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlsO9NsACgkQHPApP6U8
pFgRFRAAqjB/TpapkGZEMEB3Ol8UKTmwVO8ykmu08xZwnJ4RWRXLj29JF2I0FxXw
RZ4YgEf/AQp0DzHnUf0f5HLCtkFDU5VYX9YbcpSWO1j4Mbc+N5ZdQeLZMuy2RYXC
k9BG5ct/8iWeyKleLUwDqOadno8TMjaF0M8ZO0VXOn02iKBmT2NA8zMJkj5oxfx4
ryBtXft3eW3cyCf8zQUnVQ9G+XemVn9BaW7eRMGHhMPGDTxrFeIShNSIxr1GXu72
avx9zqW6h2OpsZfMP71hNg93AjS5ainQ/qGt/D1WdIP+lo/zpu+NSpG5qu7Xwphs
3ABGM2gJ3xmi7zsJW07jEi0F85y17XHD9VcsP6O/cMEcucYY5T5+9k5wMaOLCisp
I1Oq+PTZHQgDrN5YlUJgsy0iWKuxlxh32QeSgeOZbjsPi6nj6CUz4TU9E9kH8Gkl
WbMiWAUPKY2PCK7xrTOShLL8ONSVarKdSOJqzVyHwAO9KLnvhN/5p8+IA91LTTrI
TUS8P/scbX0PVB1tRk/wWJEBllFA6whux5XOUl9bL5hFJoEear7tfSaltCCQLJG5
h8HUQKGj8GgEYlHE+hLWuWWAkqD+ln1WKbo5O5FFjz3gqRs8IZPUXmY/QqIEhhR+
65y0RcReSWJdlIC57FFKFfnh23JOjainsvJvbSdMxkWzaYvsZ0k=
=mLOS
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Adam Feder
On Wed, May 30, 2018 at 12:00 PM, Christopher Schultz <
[hidden email]> wrote:

>
> On 5/30/18 12:33 PM, Rémy Maucherat wrote:
> > It will be included in 9.0.9.
>
> More specifically, the current releases are usually released about
> once per month give or take a few weeks. You shouldn't have to wait
> very long, or you could patch your own build if you'd like.
>
> Looks like r1832519 is what you are looking.
>


 thanks, chris.  :)

ttfn,
ab
Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Alex Marchevskiy
In reply to this post by Adam Feder
Hi Rémy,

Thank you for your quick follow up to the issue posted by Adam. I have been reviewing the patch from r1832519 and it appears that if a connection is established and no bytes are sent, the socket remains open indefinitely waiting for the handshakeReadCompletionHandler to callback. Hence it would be possible for a malicious user to establish enough connections to match the OS file descriptor limit and prevent Tomcat from servicing any new connections simply by keeping the connections open and not sending any data.

In order to work around this case, I was able to include the endpoint connection timeout in the processSNI method and pass it to the read under the case where no bytes are available in the netInBuffer. Please see the diff below:


diff --git a/java/org/apache/tomcat/util/net/SecureNio2Channel.java b/java/org/apache/tomcat/util/net/SecureNio2Channel.java
index b0202b706..5cf044bff 100644
--- a/java/org/apache/tomcat/util/net/SecureNio2Channel.java
+++ b/java/org/apache/tomcat/util/net/SecureNio2Channel.java
@@ -221,8 +221,9 @@ public class SecureNio2Channel extends Nio2Channel  {
            return 0; //we have done our initial handshake
        }

+        long timeout = endpoint.getConnectionTimeout();
        if (!sniComplete) {
-            int sniResult = processSNI();
+            int sniResult = processSNI(timeout);
            if (sniResult == 0) {
                sniComplete = true;
            } else {
@@ -231,7 +232,6 @@ public class SecureNio2Channel extends Nio2Channel  {
        }

        SSLEngineResult handshake = null;
-        long timeout = endpoint.getConnectionTimeout();

        while (!handshakeComplete) {
            switch (handshakeStatus) {
@@ -366,12 +366,13 @@ public class SecureNio2Channel extends Nio2Channel  {
     * present and, if it is, what host name has been requested. Based on the
     * provided host name, configure the SSLEngine for this connection.
     */
-    private int processSNI() throws IOException {
+    private int processSNI(long timeout) throws IOException {
        // If there is no data to process, trigger a read immediately. This is
        // an optimisation for the typical case so we don't create an
        // SNIExtractor only to discover there is no data to process
        if (netInBuffer.position() == 0) {
-            sc.read(netInBuffer, socket, handshakeReadCompletionHandler);
+            sc.read(netInBuffer, Nio2Endpoint.toNio2Timeout(timeout),
+                    TimeUnit.MILLISECONDS, socket, handshakeReadCompletionHandler);
            return 1;
        }


I was hoping to ask for your feedback regarding whether this is a valid way to handle the aforementioned condition. I would be happy to work on getting this merged upstream with proper test coverage if you believe this is reasonable.

Thanks in advance,
Alex
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Rémy Maucherat
On Sat, Jun 2, 2018 at 9:25 PM Alex Marchevskiy <[hidden email]> wrote:

> Hi Rémy,
>
> Thank you for your quick follow up to the issue posted by Adam. I have
> been reviewing the patch from r1832519 and it appears that if a connection
> is established and no bytes are sent, the socket remains open indefinitely
> waiting for the handshakeReadCompletionHandler to callback. Hence it would
> be possible for a malicious user to establish enough connections to match
> the OS file descriptor limit and prevent Tomcat from servicing any new
> connections simply by keeping the connections open and not sending any
> data.
>

Ok, there were three read operations that did not have a timeout and that
is now fixed as well. However, the timeout is often "longish", so it won't
make such a big difference anyway and NIO2 is not supposed to operate with
any real connection limit.

Rémy
Reply | Threaded
Open this post in threaded view
|

Re: Http11Nio2Protocol allows brand-new sockets to live indefinitely? (Tomcat 9.0.8 and others)

Alex Marchevskiy
Thanks Rémy!

> On Jun 2, 2018, at 12:49 PM, Rémy Maucherat <[hidden email]> wrote:
>
> On Sat, Jun 2, 2018 at 9:25 PM Alex Marchevskiy <[hidden email]> wrote:
>
>> Hi Rémy,
>>
>> Thank you for your quick follow up to the issue posted by Adam. I have
>> been reviewing the patch from r1832519 and it appears that if a connection
>> is established and no bytes are sent, the socket remains open indefinitely
>> waiting for the handshakeReadCompletionHandler to callback. Hence it would
>> be possible for a malicious user to establish enough connections to match
>> the OS file descriptor limit and prevent Tomcat from servicing any new
>> connections simply by keeping the connections open and not sending any
>> data.
>>
>
> Ok, there were three read operations that did not have a timeout and that
> is now fixed as well. However, the timeout is often "longish", so it won't
> make such a big difference anyway and NIO2 is not supposed to operate with
> any real connection limit.
>
> Rémy


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]