Tomcat memory growth while using TLS

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Tomcat memory growth while using TLS

Mason Meier
Hello,

I'm running Tomcat-8.5 with TLS and I've noticed substantial memory growth
with requests over time, to the point that if I run Tomcat in Docker and
make constant requests to it, Docker will kill the container due to
excessive memory utilization. The problem occurs with standalone Tomcat as
well. Over the course of millions of requests, the memory usage of the
Tomcat process grows continuously, seemingly without bound.

I've done a fair amount of testing on AWS EC2 instances and some local
machines, and here are my observations:
  * 'org.apache.tomcat.util.net.openssl.OpenSSLImplementation' seems to
increase memory utilization more quickly and consistently than
'org.apache.tomcat.util.net.jsse.JSSEImplementation'. The
JSSEImplementation doesn't cause the memory to grow in certain setups.
  * Limiting the heap size of the JVM does not affect the amount of memory
consumed
  * Other than decreasing a small amount (presumably due to garbage
collection), memory utilization stays consistent after the flow of requests
is stopped

My testing consists of:
1. Installing brand new versions of everything
2. Generating a certificate
3. Changing my server.xml to the one shown below
4. Replacing 'webapps/ROOT/index.jsp' with a shorter payload
5. Starting Tomcat
6. Making millions of parallel requests from a different computer and
watching Tomcat's memory utilization grow

Is there anything in my 'server.xml' that would be cause for concern? Are
there any known memory leaks in Tomcat's OpenSSL implementation? What steps
can I take to debug this problem?

Versions:
Tomcat - apache-tomcat-8.5.37
Java - JDK-1.8u191
OpenSSL - openssl-1.0.2q
APR - apr-1.6.5
Tomcat Native - tomcat-native-1.2.19
OS - Amazon Linux release 2 (Karoo)
uname -a - Linux 4.14.77-81.59.amzn2.x86_64 #1 SMP Mon Nov 12 21:32:48 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

Here is my 'server.xml' file:
<?xml version="1.0" encoding="UTF-8"?>
<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  <Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
  <Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  <Service name="Catalina">
    <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"

 sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
           port="8443"
           SSLEnabled="true" scheme="https" secure="true"
           keystoreFile="server.keystore" keystorePass="<REDACTED>"
           clientAuth="optional" sslProtocol="TLS"/>
    <Engine name="Catalina" defaultHost="localhost">
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">
        <Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

Thanks,
Mason
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

markt
On 08/01/2019 23:51, Mason Meier wrote:
> Hello,
>
> I'm running Tomcat-8.5 with TLS and I've noticed substantial memory growth
> with requests over time, to the point that if I run Tomcat in Docker and
> make constant requests to it, Docker will kill the container due to
> excessive memory utilization. The problem occurs with standalone Tomcat as
> well. Over the course of millions of requests, the memory usage of the
> Tomcat process grows continuously, seemingly without bound.

I've behaviour like this in the past. From memory there was some caching
in the TLS implementation at the root of most of it that could be
controlled with some system properties.

It may be you are seeing the same thing. Or you may have found a memory
leak. The next step would be to use a profiler to see where the memory
is being used.

> I've done a fair amount of testing on AWS EC2 instances and some local
> machines, and here are my observations:
>   * 'org.apache.tomcat.util.net.openssl.OpenSSLImplementation' seems to
> increase memory utilization more quickly and consistently than
> 'org.apache.tomcat.util.net.jsse.JSSEImplementation'. The
> JSSEImplementation doesn't cause the memory to grow in certain setups.

Can you share some configs that demonstrate an issue and some that
don't. That might help narrow down what is going on.

Mark


>   * Limiting the heap size of the JVM does not affect the amount of memory
> consumed
>   * Other than decreasing a small amount (presumably due to garbage
> collection), memory utilization stays consistent after the flow of requests
> is stopped
>
> My testing consists of:
> 1. Installing brand new versions of everything
> 2. Generating a certificate
> 3. Changing my server.xml to the one shown below
> 4. Replacing 'webapps/ROOT/index.jsp' with a shorter payload
> 5. Starting Tomcat
> 6. Making millions of parallel requests from a different computer and
> watching Tomcat's memory utilization grow
>
> Is there anything in my 'server.xml' that would be cause for concern? Are
> there any known memory leaks in Tomcat's OpenSSL implementation? What steps
> can I take to debug this problem?
>
> Versions:
> Tomcat - apache-tomcat-8.5.37
> Java - JDK-1.8u191
> OpenSSL - openssl-1.0.2q
> APR - apr-1.6.5
> Tomcat Native - tomcat-native-1.2.19
> OS - Amazon Linux release 2 (Karoo)
> uname -a - Linux 4.14.77-81.59.amzn2.x86_64 #1 SMP Mon Nov 12 21:32:48 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> Here is my 'server.xml' file:
> <?xml version="1.0" encoding="UTF-8"?>
> <Server port="8005" shutdown="SHUTDOWN">
>   <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
>   <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="on" />
>   <Listener
> className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
>   <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
>   <Listener
> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
>
>   <Service name="Catalina">
>     <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
>
>  sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
>            port="8443"
>            SSLEnabled="true" scheme="https" secure="true"
>            keystoreFile="server.keystore" keystorePass="<REDACTED>"
>            clientAuth="optional" sslProtocol="TLS"/>
>     <Engine name="Catalina" defaultHost="localhost">
>       <Host name="localhost"  appBase="webapps"
>             unpackWARs="true" autoDeploy="true">
>         <Valve className="org.apache.catalina.valves.AccessLogValve"
> directory="logs"
>                prefix="localhost_access_log" suffix=".txt"
>                pattern="%h %l %u %t &quot;%r&quot; %s %b" />
>       </Host>
>     </Engine>
>   </Service>
> </Server>
>
> Thanks,
> Mason
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

info@flyingfischer.ch
Am 09.01.19 um 11:14 schrieb Mark Thomas:

> On 08/01/2019 23:51, Mason Meier wrote:
>> Hello,
>>
>> I'm running Tomcat-8.5 with TLS and I've noticed substantial memory growth
>> with requests over time, to the point that if I run Tomcat in Docker and
>> make constant requests to it, Docker will kill the container due to
>> excessive memory utilization. The problem occurs with standalone Tomcat as
>> well. Over the course of millions of requests, the memory usage of the
>> Tomcat process grows continuously, seemingly without bound.
> I've behaviour like this in the past. From memory there was some caching
> in the TLS implementation at the root of most of it that could be
> controlled with some system properties.
>
> It may be you are seeing the same thing. Or you may have found a memory
> leak. The next step would be to use a profiler to see where the memory
> is being used.
>
>> I've done a fair amount of testing on AWS EC2 instances and some local
>> machines, and here are my observations:
>>   * 'org.apache.tomcat.util.net.openssl.OpenSSLImplementation' seems to
>> increase memory utilization more quickly and consistently than
>> 'org.apache.tomcat.util.net.jsse.JSSEImplementation'. The
>> JSSEImplementation doesn't cause the memory to grow in certain setups.
> Can you share some configs that demonstrate an issue and some that
> don't. That might help narrow down what is going on.
>
> Mark
>

We do also see frequent OOM memories with the latest Tomcat versions.
Free memory shrinks to a critical value. With previous versions this
behaviour was not here. However, we use a limited machine and I need to
investigate further.

Markus

>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

markt
In reply to this post by markt
On 09/01/2019 10:14, Mark Thomas wrote:

> On 08/01/2019 23:51, Mason Meier wrote:
>> Hello,
>>
>> I'm running Tomcat-8.5 with TLS and I've noticed substantial memory growth
>> with requests over time, to the point that if I run Tomcat in Docker and
>> make constant requests to it, Docker will kill the container due to
>> excessive memory utilization. The problem occurs with standalone Tomcat as
>> well. Over the course of millions of requests, the memory usage of the
>> Tomcat process grows continuously, seemingly without bound.
>
> I've behaviour like this in the past. From memory there was some caching
> in the TLS implementation at the root of most of it that could be
> controlled with some system properties.
>
> It may be you are seeing the same thing. Or you may have found a memory
> leak. The next step would be to use a profiler to see where the memory
> is being used.

Using NIO + OpenSSL with the settings from the commented out APR/native
TLS connector in server.xml, the heap and non-heap memory reported by
the profiler reach steady state at fairly low values. However, process
memory continues to steadily increase. Those observations are consistent
with a memory leak in native code (but they are not proof).

Next steps are looking at additional logging to see where the memory
allocations occur.

Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

markt
On 09/01/2019 14:24, Mark Thomas wrote:
> On 09/01/2019 10:14, Mark Thomas wrote:

<snip/>

>> It may be you are seeing the same thing. Or you may have found a memory
>> leak. The next step would be to use a profiler to see where the memory
>> is being used.
>
> Using NIO + OpenSSL with the settings from the commented out APR/native
> TLS connector in server.xml, the heap and non-heap memory reported by
> the profiler reach steady state at fairly low values. However, process
> memory continues to steadily increase. Those observations are consistent
> with a memory leak in native code (but they are not proof).
>
> Next steps are looking at additional logging to see where the memory
> allocations occur.

I didn't really get anywhere with that. There was a lot of noise from
the JVM and the performance hit was several orders of magnitude so there
was no obvious built up of leaked memory.

However, there has been some progress.

I've tested trunk (9.0.x) with:
- Oracle JDK 1.8.0_192
- Tomcat Native 1.2.19
- OpenSSL 1.1.2-dev
- JMeter (20 threads, /index.jsp, no keep-alive)

and the commented out APT/native TLS connector in the default server.xml.

APR/native connector - no leak
NIO+JSSE - no leak
NIO+OpenSSL - leak

Interestingly, the performance of the above was roughly:
NIO+OpenSSL - 5000 req/s
NIO+JSSE    - 4000 req/s
APR/native  -  400 req/s

That difference is so large I wonder if I was doing something wrong.
Something to come back to later.

Given than only NIO+OpenSSL is affected (or at least it is significantly
more affected than the other combinations) I'm going to take another
look at the code focussing on the parts that are specific to that
configuration.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Tomcat memory growth while using TLS

John.E.Gregg-2
Mason, Mark, etc...

> -----Original Message-----
> From: Mark Thomas <[hidden email]>
> Sent: Thursday, January 10, 2019 10:45 AM
> To: [hidden email]
> Subject: Re: Tomcat memory growth while using TLS
>
> On 09/01/2019 14:24, Mark Thomas wrote:
> > On 09/01/2019 10:14, Mark Thomas wrote:
>
> <snip/>
>
> >> It may be you are seeing the same thing. Or you may have found a
> >> memory leak. The next step would be to use a profiler to see where
> >> the memory is being used.
> >
> > Using NIO + OpenSSL with the settings from the commented out
> > APR/native TLS connector in server.xml, the heap and non-heap memory
> > reported by the profiler reach steady state at fairly low values.
> > However, process memory continues to steadily increase. Those
> > observations are consistent with a memory leak in native code (but they
> are not proof).
> >
> > Next steps are looking at additional logging to see where the memory
> > allocations occur.
>
> I didn't really get anywhere with that. There was a lot of noise from the JVM
> and the performance hit was several orders of magnitude so there was no
> obvious built up of leaked memory.
>
> However, there has been some progress.
>
> I've tested trunk (9.0.x) with:
> - Oracle JDK 1.8.0_192
> - Tomcat Native 1.2.19
> - OpenSSL 1.1.2-dev
> - JMeter (20 threads, /index.jsp, no keep-alive)
>
> and the commented out APT/native TLS connector in the default server.xml.
>
> APR/native connector - no leak
> NIO+JSSE - no leak
> NIO+OpenSSL - leak
>
> Interestingly, the performance of the above was roughly:
> NIO+OpenSSL - 5000 req/s
> NIO+JSSE    - 4000 req/s
> APR/native  -  400 req/s
>
> That difference is so large I wonder if I was doing something wrong.
> Something to come back to later.
>
> Given than only NIO+OpenSSL is affected (or at least it is significantly more
> affected than the other combinations) I'm going to take another look at the
> code focussing on the parts that are specific to that configuration.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

The first troubleshooting step should be to take a heap dump and analyze it with a tool like Eclipse MAT.

I don't know anything about the OpenSSL implementation, but it probably has a session cache.  Is there an upper limit on the cache size?  The JSSE cache size is unlimited by default, I think, and objects only expire from the cache after 1 day.

When a client connects, if it already has a session id from a previous handshake, it will ask the server to resume the session.  If it happens to connect to the same server, it will probably get to reuse the existing (cached) session.  If not, then a new session is created and cached.  If you have a lot of servers behind a load balancer with no stickiness, then the session reuse will not be high.  For example, if you have 2 servers, you're only going to get 50% reuse.  If you have 100 servers, only 1% reuse!  )-:  The lower the session reuse rate, the more orphaned sessions there are sitting on the server.

Mark, in your test if you were connecting from the same client to the same server over and over, you probably used the same session over and over, so you wouldn't see much session cache growth, if any.  I'm not sure of a way to tell the client or server to not reuse sessions for testing purposes.  Even round-robining across only 2 servers would be more realistic because the first handshake will result in session 123, then the second (to the other server) would result in session 456 (because the second server doesn't know anything about session 123,) then the third might be to the same server as the first, but by then the client has forgotten about session 123 and tries to resume 456, which doesn't exist on the first server, so you now get session 789 created, and so forth.

John




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

markt
On 10/01/2019 17:31, [hidden email] wrote:

<snip/>

> The first troubleshooting step should be to take a heap dump and analyze it with a tool like Eclipse MAT.

If you read the entire thread you will see that I have already done
that. The Java Heap (in fact all the JVM managed pools) were static
whilst overall memory was growing. That points to a native memory leak
that no Java profiler is going be able to find the root cause.

I also tried various native memory leak detectors but the combination of
the noise from the JVM and the performance impact meant the output was
not useful.

> I don't know anything about the OpenSSL implementation, but it probably has a session cache.  Is there an upper limit on the cache size?  The JSSE cache size is unlimited by default, I think, and objects only expire from the cache after 1 day.

Given that the APR/native connector is not affected but the NIO is, that
points to something other than the SSL session cache (since the
configuration is the same between the two). Again this was mentioned
earlier in the thread.

> When a client connects, if it already has a session id from a previous handshake, it will ask the server to resume the session.  If it happens to connect to the same server, it will probably get to reuse the existing (cached) session.  If not, then a new session is created and cached.  If you have a lot of servers behind a load balancer with no stickiness, then the session reuse will not be high.  For example, if you have 2 servers, you're only going to get 50% reuse.  If you have 100 servers, only 1% reuse!  )-:  The lower the session reuse rate, the more orphaned sessions there are sitting on the server.
>
> Mark, in your test if you were connecting from the same client to the same server over and over, you probably used the same session over and over, so you wouldn't see much session cache growth, if any.

Indeed.  Which, given that the memory usage is growing, is further
evidence that the session cache is not part of it.

I'm currently looking at the Tomcat Native Library and written some
simple tests that mimic TLS connection setup in a tight loop. These
tests also exhibit native memory leaks.

I've just tracked down one leak although it is a relatively small one.
Next steps are to fix that leak and then find the next one. And then repeat.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

markt
On 10/01/2019 19:55, Mark Thomas wrote:

<snip/>

> I've just tracked down one leak although it is a relatively small one.
> Next steps are to fix that leak and then find the next one. And then repeat.

Found it.

The leak impacted NIO and NIO2 when used with OpenSSL.

The bug is in Tomcat Native. I have a fix that I am currently testing.
That fix should be in the next Tomcat Native release.

For those interested in the technical details, Tomcat Native allocates
some Native memory to track various attributes of each connection.

In APR/Native this memory is allocated from a new APR memory pool that
is associated with the connection. When the connection closes, the pool
is destroyed and the memory freed. All is good.

In NIO[2]+OpenSSL the per connection memory was allocated from the APR
memory pool associated with the Connector so the memory was not released
until the Connector was stopped. The fix is to create a pool per
connection and use that for the per connection allocations. With a fix
to do that in place, memory use is now broadly static under extended TLS
load.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 1/11/19 12:23, Mark Thomas wrote:

> On 10/01/2019 19:55, Mark Thomas wrote:
>
> <snip/>
>
>> I've just tracked down one leak although it is a relatively small
>> one. Next steps are to fix that leak and then find the next one.
>> And then repeat.
>
> Found it.
>
> The leak impacted NIO and NIO2 when used with OpenSSL.
>
> The bug is in Tomcat Native. I have a fix that I am currently
> testing. That fix should be in the next Tomcat Native release.
>
> For those interested in the technical details, Tomcat Native
> allocates some Native memory to track various attributes of each
> connection.
>
> In APR/Native this memory is allocated from a new APR memory pool
> that is associated with the connection. When the connection closes,
> the pool is destroyed and the memory freed. All is good.
>
> In NIO[2]+OpenSSL the per connection memory was allocated from the
> APR memory pool associated with the Connector so the memory was not
> released until the Connector was stopped. The fix is to create a
> pool per connection and use that for the per connection
> allocations. With a fix to do that in place, memory use is now
> broadly static under extended TLS load.

Nice work.

I think you can file this under "stuff we knew might be a problem" and
was the reason we were considering reference-counting, etc. to make
sure we didn't leak *connections*. Here, the connection was
cleaned-up, but some of the associated data wasn't being cleaned from
a shared cache. Do I have that right?

Is there a way, short of bouncing Tomcat, to flush that cache? For
example, bouncing or otherwise re-initializing the connector or
something like that?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlw40t8ACgkQHPApP6U8
pFg4DQ/+Pwyp6P1jMmw+pW8ewxJL0FsI2QjZ439jPVHhBmdO+GmQk23FyhGGKHCb
zj6XtAmLR3h6xGxKpzYVufpIFOhPV6r/ZQ7sb3Mj6dAuI6fN3Apbhc73iHb8qTw8
0TX4yo8fhiXLqa6ImgTqJ56a0hgefbwYeZxPnD35J1hioqDfge4oTdgGCig1lrJi
zOQSVuDiBf2QeY22qGjBJFoFYanGmXnWGLIyOmRJfYWZbBQvc7g1bxl28I6AT4b5
ea/pVu1C0YYNbO7+3Rp+iA2yRY1Db5HR1xn7R8X4tKnbetL20QZL/nCRzgCmtdn/
lFki7HzJRyQQU/kNYNyeAOQsQNSS+C7esWWpt3SqNjX5JzCrOXZ2YxeIFKK9wD3s
yHwqRuHpRhCB22ZQczbBhBmuGEWG2Ovs1gS92DCtS4O135ka1FbruuG9O5j7R60A
fF3EnqCTsqlmkYePhmdNgFmvNiGhbCiOF9wh9L5UaXf5iMAnohCsqBWlTQZdGfQZ
YLt8g4P8PGVrIp3Rgj7r8bKvhgRQekgOqkNIRAf9yC8Rcwfm9ANn2mydjFrEX8fk
Z/hh3b4Z0NYAZ+NGbfg1Cpv7qk5cCX4PV8YqAtm9zNU7O//Cf58de7+la1xuZbn9
FOUtv1aJcNvhuw7IjID+ACMtlIsQAXhckyL1CpP85/exQXIRqN0=
=TLMC
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

markt
On 11/01/2019 17:31, Christopher Schultz wrote:

> Mark,
>
> On 1/11/19 12:23, Mark Thomas wrote:
>> On 10/01/2019 19:55, Mark Thomas wrote:
>
>> <snip/>
>
>>> I've just tracked down one leak although it is a relatively small
>>> one. Next steps are to fix that leak and then find the next one.
>>> And then repeat.
>
>> Found it.
>
>> The leak impacted NIO and NIO2 when used with OpenSSL.
>
>> The bug is in Tomcat Native. I have a fix that I am currently
>> testing. That fix should be in the next Tomcat Native release.
>
>> For those interested in the technical details, Tomcat Native
>> allocates some Native memory to track various attributes of each
>> connection.
>
>> In APR/Native this memory is allocated from a new APR memory pool
>> that is associated with the connection. When the connection closes,
>> the pool is destroyed and the memory freed. All is good.
>
>> In NIO[2]+OpenSSL the per connection memory was allocated from the
>> APR memory pool associated with the Connector so the memory was not
>> released until the Connector was stopped. The fix is to create a
>> pool per connection and use that for the per connection
>> allocations. With a fix to do that in place, memory use is now
>> broadly static under extended TLS load.
>
> Nice work.
>
> I think you can file this under "stuff we knew might be a problem" and
> was the reason we were considering reference-counting, etc. to make
> sure we didn't leak *connections*. Here, the connection was
> cleaned-up, but some of the associated data wasn't being cleaned from
> a shared cache. Do I have that right?

Yes. The cache in this case was a memory pool. Think of it as managed
malloc. You can destroy pool and clean up all the memory allocated
through it without having to call free on each of the individual
allocations. It was a tcn_ssl_ctxt_t structure that wasn't being
released. It is a collection of 4 pointers so it wasn't too big but one
for every connection mounts up over time.

> Is there a way, short of bouncing Tomcat, to flush that cache? For
> example, bouncing or otherwise re-initializing the connector or
> something like that?

Yes, but I'm not sure how much practical use it is. You need to unbind
the server socket - the necessary clean-up happens
AbstractEnpoint.unbind(). That will happen on Connector.stop() or
Connector.destroy() depending on bindOnInit. The issue is that unbinding
the socket is going to cause connections to be dropped. The end result
will be similar to bouncing Tomcat although it should be quicker (no
need to reload webapps).

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 1/11/19 13:50, Mark Thomas wrote:

> On 11/01/2019 17:31, Christopher Schultz wrote:
>> Mark,
>>
>> On 1/11/19 12:23, Mark Thomas wrote:
>>> On 10/01/2019 19:55, Mark Thomas wrote:
>>
>>> <snip/>
>>
>>>> I've just tracked down one leak although it is a relatively
>>>> small one. Next steps are to fix that leak and then find the
>>>> next one. And then repeat.
>>
>>> Found it.
>>
>>> The leak impacted NIO and NIO2 when used with OpenSSL.
>>
>>> The bug is in Tomcat Native. I have a fix that I am currently
>>> testing. That fix should be in the next Tomcat Native release.
>>
>>> For those interested in the technical details, Tomcat Native
>>> allocates some Native memory to track various attributes of
>>> each connection.
>>
>>> In APR/Native this memory is allocated from a new APR memory
>>> pool that is associated with the connection. When the
>>> connection closes, the pool is destroyed and the memory freed.
>>> All is good.
>>
>>> In NIO[2]+OpenSSL the per connection memory was allocated from
>>> the APR memory pool associated with the Connector so the memory
>>> was not released until the Connector was stopped. The fix is to
>>> create a pool per connection and use that for the per
>>> connection allocations. With a fix to do that in place, memory
>>> use is now broadly static under extended TLS load.
>>
>> Nice work.
>>
>> I think you can file this under "stuff we knew might be a
>> problem" and was the reason we were considering
>> reference-counting, etc. to make sure we didn't leak
>> *connections*. Here, the connection was cleaned-up, but some of
>> the associated data wasn't being cleaned from a shared cache. Do
>> I have that right?
>
> Yes. The cache in this case was a memory pool. Think of it as
> managed malloc. You can destroy pool and clean up all the memory
> allocated through it without having to call free on each of the
> individual allocations. It was a tcn_ssl_ctxt_t structure that
> wasn't being released. It is a collection of 4 pointers so it
> wasn't too big but one for every connection mounts up over time.

Does this affect all versions of Tomcat where JSSE+OpenSSL are in use?
Sounds like it would.

>> Is there a way, short of bouncing Tomcat, to flush that cache?
>> For example, bouncing or otherwise re-initializing the connector
>> or something like that?
>
> Yes, but I'm not sure how much practical use it is. You need to
> unbind the server socket - the necessary clean-up happens
> AbstractEnpoint.unbind(). That will happen on Connector.stop() or
> Connector.destroy() depending on bindOnInit. The issue is that
> unbinding the socket is going to cause connections to be dropped.
> The end result will be similar to bouncing Tomcat although it
> should be quicker (no need to reload webapps).

I was mostly asking on behalf of others. Some people may find it
easier to bounce their connectors periodically rather than upgrade
right away.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlw5BYAACgkQHPApP6U8
pFhXCxAArAW1vaWcaMsT84rmfBMFGEWhDetSG7Eb51h631yicFnU/XkJD6pYa1JY
xq1LKPbA/MDlTM0saxgmd1u2Os5YoHg5Gk4jHgfFdAiUoZDdEPMheXcvNauZQgmq
+jg13Zh9aNotsIPTtBySlwKlkfJ04i8K80iTZ/HVnMVpWDczpqA9g0DEHkvpMbj6
85lZM4fMExLUswSCPJzliqCeyt7KJr0awlAqL7KUo6ZoGJR5CSCQOaRbZUhNx9w3
wrothMP6J2WHu1cUN+gsMze9dYeicKPoR1aCWrUk6w4PsoelVrS+UfPQEULdTZLi
3V3Ezxdvbv6tnrPNHxuQf/FrpSqU0+T1SocsnnNAT7/f/451Gd9fcNF/R9uyVjBE
Uy4W3gumpnZtC4g6XBmNw5zOF2ltei/2iJjO7WGHOdW87iOfkOoszdj9smMX/MOP
dMlNBejckOfOgJsfT6DZ3ZbvzAbO3WpK2W8AMsGTXhEDkuHsrswzx1/f7jzOFDsz
ZNPpm8q/wLzuBSRDkc9r0hMivJPdWuUWoUkn2GTbB1DnuH7jLbr09Hk1YMU+zsvF
jNEgpI5ZmNDjuP3UHkupNzotlMNh20stvmSXFKqjXHL4RHCtg9XkNixW11Z9roWT
VSHj+Sd28AnxccI9GpT0EmH25cyAct26e75yJonw92PTyfrPLcc=
=MkQ/
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

markt
On 11/01/2019 21:07, Christopher Schultz wrote:

> Mark,
>
> On 1/11/19 13:50, Mark Thomas wrote:
>> On 11/01/2019 17:31, Christopher Schultz wrote:
>>> Mark,
>>>
>>> On 1/11/19 12:23, Mark Thomas wrote:
>>>> On 10/01/2019 19:55, Mark Thomas wrote:
>>>
>>>> <snip/>
>>>
>>>>> I've just tracked down one leak although it is a relatively
>>>>> small one. Next steps are to fix that leak and then find the
>>>>> next one. And then repeat.
>>>
>>>> Found it.
>>>
>>>> The leak impacted NIO and NIO2 when used with OpenSSL.
>>>
>>>> The bug is in Tomcat Native. I have a fix that I am currently
>>>> testing. That fix should be in the next Tomcat Native release.
>>>
>>>> For those interested in the technical details, Tomcat Native
>>>> allocates some Native memory to track various attributes of
>>>> each connection.
>>>
>>>> In APR/Native this memory is allocated from a new APR memory
>>>> pool that is associated with the connection. When the
>>>> connection closes, the pool is destroyed and the memory freed.
>>>> All is good.
>>>
>>>> In NIO[2]+OpenSSL the per connection memory was allocated from
>>>> the APR memory pool associated with the Connector so the memory
>>>> was not released until the Connector was stopped. The fix is to
>>>> create a pool per connection and use that for the per
>>>> connection allocations. With a fix to do that in place, memory
>>>> use is now broadly static under extended TLS load.
>>>
>>> Nice work.
>>>
>>> I think you can file this under "stuff we knew might be a
>>> problem" and was the reason we were considering
>>> reference-counting, etc. to make sure we didn't leak
>>> *connections*. Here, the connection was cleaned-up, but some of
>>> the associated data wasn't being cleaned from a shared cache. Do
>>> I have that right?
>
>> Yes. The cache in this case was a memory pool. Think of it as
>> managed malloc. You can destroy pool and clean up all the memory
>> allocated through it without having to call free on each of the
>> individual allocations. It was a tcn_ssl_ctxt_t structure that
>> wasn't being released. It is a collection of 4 pointers so it
>> wasn't too big but one for every connection mounts up over time.
>
> Does this affect all versions of Tomcat where JSSE+OpenSSL are in use?
> Sounds like it would.

Yes. The bug (and the fix) were entirely in Tomcat Native.

On the plus side, it means only Tomcat Native needs to be updated to get
the fix.

Mark


>
>>> Is there a way, short of bouncing Tomcat, to flush that cache?
>>> For example, bouncing or otherwise re-initializing the connector
>>> or something like that?
>
>> Yes, but I'm not sure how much practical use it is. You need to
>> unbind the server socket - the necessary clean-up happens
>> AbstractEnpoint.unbind(). That will happen on Connector.stop() or
>> Connector.destroy() depending on bindOnInit. The issue is that
>> unbinding the socket is going to cause connections to be dropped.
>> The end result will be similar to bouncing Tomcat although it
>> should be quicker (no need to reload webapps).
>
> I was mostly asking on behalf of others. Some people may find it
> easier to bounce their connectors periodically rather than upgrade
> right away.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

info@flyingfischer.ch
In reply to this post by markt
Am 11.01.19 um 18:23 schrieb Mark Thomas:

> Found it.
>
> The leak impacted NIO and NIO2 when used with OpenSSL.
>
> The bug is in Tomcat Native. I have a fix that I am currently testing.
> That fix should be in the next Tomcat Native release.
>
> For those interested in the technical details, Tomcat Native allocates
> some Native memory to track various attributes of each connection.
>
> In APR/Native this memory is allocated from a new APR memory pool that
> is associated with the connection. When the connection closes, the pool
> is destroyed and the memory freed. All is good.
>
> In NIO[2]+OpenSSL the per connection memory was allocated from the APR
> memory pool associated with the Connector so the memory was not released
> until the Connector was stopped. The fix is to create a pool per
> connection and use that for the per connection allocations. With a fix
> to do that in place, memory use is now broadly static under extended TLS
> load.
>
> Mark
>

Thanks Mark!

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tomcat memory growth while using TLS

John Dale
I love it when a memory leak bites the dust .. it's a great feeling to
extend process life.

Fixed one yesterday in a capture/encode buffer of my mp3 streamer.

Kudos, Mark!

:)

John


On 1/12/19, [hidden email] <[hidden email]> wrote:

> Am 11.01.19 um 18:23 schrieb Mark Thomas:
>> Found it.
>>
>> The leak impacted NIO and NIO2 when used with OpenSSL.
>>
>> The bug is in Tomcat Native. I have a fix that I am currently testing.
>> That fix should be in the next Tomcat Native release.
>>
>> For those interested in the technical details, Tomcat Native allocates
>> some Native memory to track various attributes of each connection.
>>
>> In APR/Native this memory is allocated from a new APR memory pool that
>> is associated with the connection. When the connection closes, the pool
>> is destroyed and the memory freed. All is good.
>>
>> In NIO[2]+OpenSSL the per connection memory was allocated from the APR
>> memory pool associated with the Connector so the memory was not released
>> until the Connector was stopped. The fix is to create a pool per
>> connection and use that for the per connection allocations. With a fix
>> to do that in place, memory use is now broadly static under extended TLS
>> load.
>>
>> Mark
>>
>
> Thanks Mark!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]