Connection Closure due to Fatal Stream with HTTP2

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Hi,

We are observing that under high load, my clients start receiving a GoAway
frame with error:

*Connection[{id}], Stream[{id}] an error occurred during processing that
was fatal to the connection.*

Background : We have implemented our clients to close connections after
every 500-1000 requests (streams). This is a load balancer requirement that
we are working on and hence such a behavior. So with a throughput of around
19k, almost 40 connections are closed and recreated every second.

After we receive this frame, my clients start behaving erroneously. Before
this as well, my clients start sending RST_STREAM with canceled for each
request. Could this be due to the number of connections we open? Is it
related to the version of Tomcat? Or maybe my clients are misbehaving?

Now since I only receive this under heavy load, I can't quite picture
enough reasons for this to happen.

Any possible clues on where I should start looking?

My Stack:
Server - Tomcat 9.0.29
Client - Jetty 9.x

Thanks in advance
Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Michael Osipov
Am 2020-06-13 um 08:42 schrieb Chirag Dewan:

> Hi,
>
> We are observing that under high load, my clients start receiving a GoAway
> frame with error:
>
> *Connection[{id}], Stream[{id}] an error occurred during processing that
> was fatal to the connection.*
>
> Background : We have implemented our clients to close connections after
> every 500-1000 requests (streams). This is a load balancer requirement that
> we are working on and hence such a behavior. So with a throughput of around
> 19k, almost 40 connections are closed and recreated every second.
>
> After we receive this frame, my clients start behaving erroneously. Before
> this as well, my clients start sending RST_STREAM with canceled for each
> request. Could this be due to the number of connections we open? Is it
> related to the version of Tomcat? Or maybe my clients are misbehaving?
>
> Now since I only receive this under heavy load, I can't quite picture
> enough reasons for this to happen.
>
> Any possible clues on where I should start looking?
>
> My Stack:
> Server - Tomcat 9.0.29
> Client - Jetty 9.x

Does the same happen w/o the load balancer?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Hi,

This is without load balancer actually. I am directly sending to Tomcat.

Update:

A part issue I found was to be 9.0.29. I observed that when request were
timed out on client (2seconds), the client would send a RST frame. And the
GoAway from Tomcat was perhaps a bug. In 9.0.36, RST frame is replied with
a RST from Tomcat.

Now the next part to troubleshoot is why after about an hour or so,
requests are timed out at Tomcat.

Could close to 100 HTTP2 connections per second cause this on Tomcat?

Thanks

On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, <[hidden email]> wrote:

> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
> > Hi,
> >
> > We are observing that under high load, my clients start receiving a
> GoAway
> > frame with error:
> >
> > *Connection[{id}], Stream[{id}] an error occurred during processing that
> > was fatal to the connection.*
> >
> > Background : We have implemented our clients to close connections after
> > every 500-1000 requests (streams). This is a load balancer requirement
> that
> > we are working on and hence such a behavior. So with a throughput of
> around
> > 19k, almost 40 connections are closed and recreated every second.
> >
> > After we receive this frame, my clients start behaving erroneously.
> Before
> > this as well, my clients start sending RST_STREAM with canceled for each
> > request. Could this be due to the number of connections we open? Is it
> > related to the version of Tomcat? Or maybe my clients are misbehaving?
> >
> > Now since I only receive this under heavy load, I can't quite picture
> > enough reasons for this to happen.
> >
> > Any possible clues on where I should start looking?
> >
> > My Stack:
> > Server - Tomcat 9.0.29
> > Client - Jetty 9.x
>
> Does the same happen w/o the load balancer?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Hi,

Update: We found that Tomcat goes OOM when a client closes and opens new connections every second. In the memory dump, we see a lot of RequestInfo objects that are causing the memory spike.

After a while, Tomcat goes OOM and start rejecting request(I get a request timed out on my client). This seems like a bug to me. 

For better understanding, let me explain my use case again:

I have a jetty client that sends HTTP2 requests to Tomcat. My requirement is to close a connection after a configurable(say 5000) number of requests/streams and open a new connection that continues to send requests. I close a connection by sending a GoAway frame.

When I execute this use case under load, I see that after ~2hours my requests fail and I get a series of errors like request timeouts(5seconds), invalid window update frame, and connection close exception on my client.
On further debugging, I found that it's a Tomcat memory problem and it goes OOM after sometime under heavy load with multiple connections being re-established by the clients.





Is this a known issue? Or a known behavior with Tomcat?

Please let me know if you any experience with such a situation. Thanks in advance.

On Sun, Jun 14, 2020 at 11:30 AM Chirag Dewan <[hidden email]> wrote:
Hi,

This is without load balancer actually. I am directly sending to Tomcat.

Update:

A part issue I found was to be 9.0.29. I observed that when request were timed out on client (2seconds), the client would send a RST frame. And the GoAway from Tomcat was perhaps a bug. In 9.0.36, RST frame is replied with a RST from Tomcat. 

Now the next part to troubleshoot is why after about an hour or so, requests are timed out at Tomcat.

Could close to 100 HTTP2 connections per second cause this on Tomcat?

Thanks

On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, <[hidden email]> wrote:
Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
> Hi,
>
> We are observing that under high load, my clients start receiving a GoAway
> frame with error:
>
> *Connection[{id}], Stream[{id}] an error occurred during processing that
> was fatal to the connection.*
>
> Background : We have implemented our clients to close connections after
> every 500-1000 requests (streams). This is a load balancer requirement that
> we are working on and hence such a behavior. So with a throughput of around
> 19k, almost 40 connections are closed and recreated every second.
>
> After we receive this frame, my clients start behaving erroneously. Before
> this as well, my clients start sending RST_STREAM with canceled for each
> request. Could this be due to the number of connections we open? Is it
> related to the version of Tomcat? Or maybe my clients are misbehaving?
>
> Now since I only receive this under heavy load, I can't quite picture
> enough reasons for this to happen.
>
> Any possible clues on where I should start looking?
>
> My Stack:
> Server - Tomcat 9.0.29
> Client - Jetty 9.x

Does the same happen w/o the load balancer?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
On 22/06/2020 11:06, Chirag Dewan wrote:

> Hi,
>
> Update: We found that Tomcat goes OOM when a client closes and opens new
> connections every second. In the memory dump, we see a lot of
> RequestInfo objects that are causing the memory spike.
>
> After a while, Tomcat goes OOM and start rejecting request(I get a
> request timed out on my client). This seems like a bug to me. 
>
> For better understanding, let me explain my use case again:
>
> I have a jetty client that sends HTTP2 requests to Tomcat. My
> requirement is to close a connection after a configurable(say 5000)
> number of requests/streams and open a new connection that continues to
> send requests. I close a connection by sending a GoAway frame.
>
> When I execute this use case under load, I see that after ~2hours my
> requests fail and I get a series of errors like request
> timeouts(5seconds), invalid window update frame, and connection close
> exception on my client.
> On further debugging, I found that it's a Tomcat memory problem and it
> goes OOM after sometime under heavy load with multiple connections being
> re-established by the clients.
>
> image.png
>
> image.png
>
> Is this a known issue? Or a known behavior with Tomcat?

Embedded images get dropped by the list software. Post those images
somewhere we can see them.

> Please let me know if you any experience with such a situation. Thanks
> in advance.

Nothing comes to mind.

I'll try some simple tests with HTTP/2.

Mark



>
> On Sun, Jun 14, 2020 at 11:30 AM Chirag Dewan <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     This is without load balancer actually. I am directly sending to Tomcat.
>
>     Update:
>
>     A part issue I found was to be 9.0.29. I observed that when request
>     were timed out on client (2seconds), the client would send a RST
>     frame. And the GoAway from Tomcat was perhaps a bug. In 9.0.36, RST
>     frame is replied with a RST from Tomcat. 
>
>     Now the next part to troubleshoot is why after about an hour or so,
>     requests are timed out at Tomcat.
>
>     Could close to 100 HTTP2 connections per second cause this on Tomcat?
>
>     Thanks
>
>     On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
>         > Hi,
>         >
>         > We are observing that under high load, my clients start
>         receiving a GoAway
>         > frame with error:
>         >
>         > *Connection[{id}], Stream[{id}] an error occurred during
>         processing that
>         > was fatal to the connection.*
>         >
>         > Background : We have implemented our clients to close
>         connections after
>         > every 500-1000 requests (streams). This is a load balancer
>         requirement that
>         > we are working on and hence such a behavior. So with a
>         throughput of around
>         > 19k, almost 40 connections are closed and recreated every second.
>         >
>         > After we receive this frame, my clients start behaving
>         erroneously. Before
>         > this as well, my clients start sending RST_STREAM with
>         canceled for each
>         > request. Could this be due to the number of connections we
>         open? Is it
>         > related to the version of Tomcat? Or maybe my clients are
>         misbehaving?
>         >
>         > Now since I only receive this under heavy load, I can't quite
>         picture
>         > enough reasons for this to happen.
>         >
>         > Any possible clues on where I should start looking?
>         >
>         > My Stack:
>         > Server - Tomcat 9.0.29
>         > Client - Jetty 9.x
>
>         Does the same happen w/o the load balancer?
>
>         ---------------------------------------------------------------------
>         To unsubscribe, e-mail: [hidden email]
>         <mailto:[hidden email]>
>         For additional commands, e-mail: [hidden email]
>         <mailto:[hidden email]>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
On 24/06/2020 12:17, Mark Thomas wrote:

> On 22/06/2020 11:06, Chirag Dewan wrote:
>> Hi,
>>
>> Update: We found that Tomcat goes OOM when a client closes and opens new
>> connections every second. In the memory dump, we see a lot of
>> RequestInfo objects that are causing the memory spike.
>>
>> After a while, Tomcat goes OOM and start rejecting request(I get a
>> request timed out on my client). This seems like a bug to me. 
>>
>> For better understanding, let me explain my use case again:
>>
>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>> requirement is to close a connection after a configurable(say 5000)
>> number of requests/streams and open a new connection that continues to
>> send requests. I close a connection by sending a GoAway frame.
>>
>> When I execute this use case under load, I see that after ~2hours my
>> requests fail and I get a series of errors like request
>> timeouts(5seconds), invalid window update frame, and connection close
>> exception on my client.
>> On further debugging, I found that it's a Tomcat memory problem and it
>> goes OOM after sometime under heavy load with multiple connections being
>> re-established by the clients.
>>
>> image.png
>>
>> image.png
>>
>> Is this a known issue? Or a known behavior with Tomcat?
>
> Embedded images get dropped by the list software. Post those images
> somewhere we can see them.
>
>> Please let me know if you any experience with such a situation. Thanks
>> in advance.
>
> Nothing comes to mind.
>
> I'll try some simple tests with HTTP/2.

I don't see a memory leak (the memory is reclaimed eventually) but I do
see possibilities to release memory associated with request processing
sooner.

Right now you need to allocate more memory to the Java process to enable
Tomcat to handle the HTTP/2 load it is presented with.

It looks like a reasonable chunk of memory is released when the
Connection closes that could be released earlier when the associated
Stream closes. I'll take a look at what can be done in that area. In the
meantime, reducing the number of Streams you allow on a Connection
before it is closed should reduce overall memory usage.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Thanks for the quick check Mark.

These are the images I tried referring to:

https://ibb.co/LzKtRgh

https://ibb.co/2s7hqRL

https://ibb.co/KmKj590


The last one is the MAT screenshot showing many RequestInfo objects.


Thanks,

Chirag

On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:

> On 24/06/2020 12:17, Mark Thomas wrote:
> > On 22/06/2020 11:06, Chirag Dewan wrote:
> >> Hi,
> >>
> >> Update: We found that Tomcat goes OOM when a client closes and opens new
> >> connections every second. In the memory dump, we see a lot of
> >> RequestInfo objects that are causing the memory spike.
> >>
> >> After a while, Tomcat goes OOM and start rejecting request(I get a
> >> request timed out on my client). This seems like a bug to me.
> >>
> >> For better understanding, let me explain my use case again:
> >>
> >> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >> requirement is to close a connection after a configurable(say 5000)
> >> number of requests/streams and open a new connection that continues to
> >> send requests. I close a connection by sending a GoAway frame.
> >>
> >> When I execute this use case under load, I see that after ~2hours my
> >> requests fail and I get a series of errors like request
> >> timeouts(5seconds), invalid window update frame, and connection close
> >> exception on my client.
> >> On further debugging, I found that it's a Tomcat memory problem and it
> >> goes OOM after sometime under heavy load with multiple connections being
> >> re-established by the clients.
> >>
> >> image.png
> >>
> >> image.png
> >>
> >> Is this a known issue? Or a known behavior with Tomcat?
> >
> > Embedded images get dropped by the list software. Post those images
> > somewhere we can see them.
> >
> >> Please let me know if you any experience with such a situation. Thanks
> >> in advance.
> >
> > Nothing comes to mind.
> >
> > I'll try some simple tests with HTTP/2.
>
> I don't see a memory leak (the memory is reclaimed eventually) but I do
> see possibilities to release memory associated with request processing
> sooner.
>
> Right now you need to allocate more memory to the Java process to enable
> Tomcat to handle the HTTP/2 load it is presented with.
>
> It looks like a reasonable chunk of memory is released when the
> Connection closes that could be released earlier when the associated
> Stream closes. I'll take a look at what can be done in that area. In the
> meantime, reducing the number of Streams you allow on a Connection
> before it is closed should reduce overall memory usage.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
On 25/06/2020 11:00, Chirag Dewan wrote:

> Thanks for the quick check Mark.
>
> These are the images I tried referring to:
>
> https://ibb.co/LzKtRgh
>
> https://ibb.co/2s7hqRL
>
> https://ibb.co/KmKj590
>
>
> The last one is the MAT screenshot showing many RequestInfo objects.

Thanks. That certainly looks like a memory leak. I'll take a closer
look. Out of interest, how many threads is the Connector configured to use?

Mark


>
>
> Thanks,
>
> Chirag
>
> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
>
>> On 24/06/2020 12:17, Mark Thomas wrote:
>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>> Hi,
>>>>
>>>> Update: We found that Tomcat goes OOM when a client closes and opens new
>>>> connections every second. In the memory dump, we see a lot of
>>>> RequestInfo objects that are causing the memory spike.
>>>>
>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>> request timed out on my client). This seems like a bug to me.
>>>>
>>>> For better understanding, let me explain my use case again:
>>>>
>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>> requirement is to close a connection after a configurable(say 5000)
>>>> number of requests/streams and open a new connection that continues to
>>>> send requests. I close a connection by sending a GoAway frame.
>>>>
>>>> When I execute this use case under load, I see that after ~2hours my
>>>> requests fail and I get a series of errors like request
>>>> timeouts(5seconds), invalid window update frame, and connection close
>>>> exception on my client.
>>>> On further debugging, I found that it's a Tomcat memory problem and it
>>>> goes OOM after sometime under heavy load with multiple connections being
>>>> re-established by the clients.
>>>>
>>>> image.png
>>>>
>>>> image.png
>>>>
>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>
>>> Embedded images get dropped by the list software. Post those images
>>> somewhere we can see them.
>>>
>>>> Please let me know if you any experience with such a situation. Thanks
>>>> in advance.
>>>
>>> Nothing comes to mind.
>>>
>>> I'll try some simple tests with HTTP/2.
>>
>> I don't see a memory leak (the memory is reclaimed eventually) but I do
>> see possibilities to release memory associated with request processing
>> sooner.
>>
>> Right now you need to allocate more memory to the Java process to enable
>> Tomcat to handle the HTTP/2 load it is presented with.
>>
>> It looks like a reasonable chunk of memory is released when the
>> Connection closes that could be released earlier when the associated
>> Stream closes. I'll take a look at what can be done in that area. In the
>> meantime, reducing the number of Streams you allow on a Connection
>> before it is closed should reduce overall memory usage.
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Hi Mark,

Its the default APR connector with 150 Threads.

Chirag

On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:

> On 25/06/2020 11:00, Chirag Dewan wrote:
> > Thanks for the quick check Mark.
> >
> > These are the images I tried referring to:
> >
> > https://ibb.co/LzKtRgh
> >
> > https://ibb.co/2s7hqRL
> >
> > https://ibb.co/KmKj590
> >
> >
> > The last one is the MAT screenshot showing many RequestInfo objects.
>
> Thanks. That certainly looks like a memory leak. I'll take a closer
> look. Out of interest, how many threads is the Connector configured to use?
>
> Mark
>
>
> >
> >
> > Thanks,
> >
> > Chirag
> >
> > On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
> >
> >> On 24/06/2020 12:17, Mark Thomas wrote:
> >>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>> Hi,
> >>>>
> >>>> Update: We found that Tomcat goes OOM when a client closes and opens
> new
> >>>> connections every second. In the memory dump, we see a lot of
> >>>> RequestInfo objects that are causing the memory spike.
> >>>>
> >>>> After a while, Tomcat goes OOM and start rejecting request(I get a
> >>>> request timed out on my client). This seems like a bug to me.
> >>>>
> >>>> For better understanding, let me explain my use case again:
> >>>>
> >>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>> requirement is to close a connection after a configurable(say 5000)
> >>>> number of requests/streams and open a new connection that continues to
> >>>> send requests. I close a connection by sending a GoAway frame.
> >>>>
> >>>> When I execute this use case under load, I see that after ~2hours my
> >>>> requests fail and I get a series of errors like request
> >>>> timeouts(5seconds), invalid window update frame, and connection close
> >>>> exception on my client.
> >>>> On further debugging, I found that it's a Tomcat memory problem and it
> >>>> goes OOM after sometime under heavy load with multiple connections
> being
> >>>> re-established by the clients.
> >>>>
> >>>> image.png
> >>>>
> >>>> image.png
> >>>>
> >>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>
> >>> Embedded images get dropped by the list software. Post those images
> >>> somewhere we can see them.
> >>>
> >>>> Please let me know if you any experience with such a situation. Thanks
> >>>> in advance.
> >>>
> >>> Nothing comes to mind.
> >>>
> >>> I'll try some simple tests with HTTP/2.
> >>
> >> I don't see a memory leak (the memory is reclaimed eventually) but I do
> >> see possibilities to release memory associated with request processing
> >> sooner.
> >>
> >> Right now you need to allocate more memory to the Java process to enable
> >> Tomcat to handle the HTTP/2 load it is presented with.
> >>
> >> It looks like a reasonable chunk of memory is released when the
> >> Connection closes that could be released earlier when the associated
> >> Stream closes. I'll take a look at what can be done in that area. In the
> >> meantime, reducing the number of Streams you allow on a Connection
> >> before it is closed should reduce overall memory usage.
> >>
> >> Mark
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
Thanks.

I've looked at the code and I have tried various tests but I am unable
to re-create a memory leak.

The code used to (before I made a few changes this afternoon) retain a
lot more memory per Stream and it is possible that what you are seeing
is a system that doesn't have enough memory to achieve steady state.

If you are able to build the latest 9.0.x and test that, that could be
helpful. Alternatively, I could provide a test build for you to
experiment with.

Some additional questions that might aid understanding:

- What is the typical response size for one of these requests?
- How long does a typical test take to process?
- What are the GC roots for those RequestInfo objects?

Thanks again,

Mark




On 25/06/2020 15:10, Chirag Dewan wrote:

> Hi Mark,
>
> Its the default APR connector with 150 Threads.
>
> Chirag
>
> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:
>
>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>> Thanks for the quick check Mark.
>>>
>>> These are the images I tried referring to:
>>>
>>> https://ibb.co/LzKtRgh
>>>
>>> https://ibb.co/2s7hqRL
>>>
>>> https://ibb.co/KmKj590
>>>
>>>
>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>
>> Thanks. That certainly looks like a memory leak. I'll take a closer
>> look. Out of interest, how many threads is the Connector configured to use?
>>
>> Mark
>>
>>
>>>
>>>
>>> Thanks,
>>>
>>> Chirag
>>>
>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
>>>
>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Update: We found that Tomcat goes OOM when a client closes and opens
>> new
>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>
>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>
>>>>>> For better understanding, let me explain my use case again:
>>>>>>
>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>> requirement is to close a connection after a configurable(say 5000)
>>>>>> number of requests/streams and open a new connection that continues to
>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>
>>>>>> When I execute this use case under load, I see that after ~2hours my
>>>>>> requests fail and I get a series of errors like request
>>>>>> timeouts(5seconds), invalid window update frame, and connection close
>>>>>> exception on my client.
>>>>>> On further debugging, I found that it's a Tomcat memory problem and it
>>>>>> goes OOM after sometime under heavy load with multiple connections
>> being
>>>>>> re-established by the clients.
>>>>>>
>>>>>> image.png
>>>>>>
>>>>>> image.png
>>>>>>
>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>
>>>>> Embedded images get dropped by the list software. Post those images
>>>>> somewhere we can see them.
>>>>>
>>>>>> Please let me know if you any experience with such a situation. Thanks
>>>>>> in advance.
>>>>>
>>>>> Nothing comes to mind.
>>>>>
>>>>> I'll try some simple tests with HTTP/2.
>>>>
>>>> I don't see a memory leak (the memory is reclaimed eventually) but I do
>>>> see possibilities to release memory associated with request processing
>>>> sooner.
>>>>
>>>> Right now you need to allocate more memory to the Java process to enable
>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>
>>>> It looks like a reasonable chunk of memory is released when the
>>>> Connection closes that could be released earlier when the associated
>>>> Stream closes. I'll take a look at what can be done in that area. In the
>>>> meantime, reducing the number of Streams you allow on a Connection
>>>> before it is closed should reduce overall memory usage.
>>>>
>>>> Mark
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Thanks Mark.

*What is the typical response size for one of these requests? *
It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
response in my production use case, but the purpose here was to isolate as
many things as possible. Hence a dummy response.

*How long does a typical test take to process? *
I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
of streams on my client was 500 - which means about 40 connections per
second.

* What are the GC roots for those RequestInfo objects?*
https://ibb.co/fMRmCXZ

I hope I was able to answer everything as expected. Thanks.

On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:

> Thanks.
>
> I've looked at the code and I have tried various tests but I am unable
> to re-create a memory leak.
>
> The code used to (before I made a few changes this afternoon) retain a
> lot more memory per Stream and it is possible that what you are seeing
> is a system that doesn't have enough memory to achieve steady state.
>
> If you are able to build the latest 9.0.x and test that, that could be
> helpful. Alternatively, I could provide a test build for you to
> experiment with.
>
> Some additional questions that might aid understanding:
>
> - What is the typical response size for one of these requests?
> - How long does a typical test take to process?
> - What are the GC roots for those RequestInfo objects?
>
> Thanks again,
>
> Mark
>
>
>
>
> On 25/06/2020 15:10, Chirag Dewan wrote:
> > Hi Mark,
> >
> > Its the default APR connector with 150 Threads.
> >
> > Chirag
> >
> > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:
> >
> >> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>> Thanks for the quick check Mark.
> >>>
> >>> These are the images I tried referring to:
> >>>
> >>> https://ibb.co/LzKtRgh
> >>>
> >>> https://ibb.co/2s7hqRL
> >>>
> >>> https://ibb.co/KmKj590
> >>>
> >>>
> >>> The last one is the MAT screenshot showing many RequestInfo objects.
> >>
> >> Thanks. That certainly looks like a memory leak. I'll take a closer
> >> look. Out of interest, how many threads is the Connector configured to
> use?
> >>
> >> Mark
> >>
> >>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Chirag
> >>>
> >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
> >>>
> >>>> On 24/06/2020 12:17, Mark Thomas wrote:
> >>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Update: We found that Tomcat goes OOM when a client closes and opens
> >> new
> >>>>>> connections every second. In the memory dump, we see a lot of
> >>>>>> RequestInfo objects that are causing the memory spike.
> >>>>>>
> >>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
> >>>>>> request timed out on my client). This seems like a bug to me.
> >>>>>>
> >>>>>> For better understanding, let me explain my use case again:
> >>>>>>
> >>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>>>> requirement is to close a connection after a configurable(say 5000)
> >>>>>> number of requests/streams and open a new connection that continues
> to
> >>>>>> send requests. I close a connection by sending a GoAway frame.
> >>>>>>
> >>>>>> When I execute this use case under load, I see that after ~2hours my
> >>>>>> requests fail and I get a series of errors like request
> >>>>>> timeouts(5seconds), invalid window update frame, and connection
> close
> >>>>>> exception on my client.
> >>>>>> On further debugging, I found that it's a Tomcat memory problem and
> it
> >>>>>> goes OOM after sometime under heavy load with multiple connections
> >> being
> >>>>>> re-established by the clients.
> >>>>>>
> >>>>>> image.png
> >>>>>>
> >>>>>> image.png
> >>>>>>
> >>>>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>>>
> >>>>> Embedded images get dropped by the list software. Post those images
> >>>>> somewhere we can see them.
> >>>>>
> >>>>>> Please let me know if you any experience with such a situation.
> Thanks
> >>>>>> in advance.
> >>>>>
> >>>>> Nothing comes to mind.
> >>>>>
> >>>>> I'll try some simple tests with HTTP/2.
> >>>>
> >>>> I don't see a memory leak (the memory is reclaimed eventually) but I
> do
> >>>> see possibilities to release memory associated with request processing
> >>>> sooner.
> >>>>
> >>>> Right now you need to allocate more memory to the Java process to
> enable
> >>>> Tomcat to handle the HTTP/2 load it is presented with.
> >>>>
> >>>> It looks like a reasonable chunk of memory is released when the
> >>>> Connection closes that could be released earlier when the associated
> >>>> Stream closes. I'll take a look at what can be done in that area. In
> the
> >>>> meantime, reducing the number of Streams you allow on a Connection
> >>>> before it is closed should reduce overall memory usage.
> >>>>
> >>>> Mark
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: [hidden email]
> >>>> For additional commands, e-mail: [hidden email]
> >>>>
> >>>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
Hi,

Thanks for the additional information. The GC roots were particularly
informative.

Those RequestInfo objects are associated with HTTP/1.1 requests, not
HTTP/2 requests.

Some further questions to try and track down what is going on:

- Are you using h2c or h2 in your test?

- Do you see the same issue if you switch the the NIO connector? Note
  performance differences between NIO and NIO2 are very small.

- How long does a single request take to process?

Thanks,

Mark

On 26/06/2020 09:24, Chirag Dewan wrote:

> Thanks Mark.
>
> *What is the typical response size for one of these requests? *
> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> response in my production use case, but the purpose here was to isolate as
> many things as possible. Hence a dummy response.
>
> *How long does a typical test take to process? *
> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> of streams on my client was 500 - which means about 40 connections per
> second.
>
> * What are the GC roots for those RequestInfo objects?*
> https://ibb.co/fMRmCXZ
>
> I hope I was able to answer everything as expected. Thanks.
>
> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
>
>> Thanks.
>>
>> I've looked at the code and I have tried various tests but I am unable
>> to re-create a memory leak.
>>
>> The code used to (before I made a few changes this afternoon) retain a
>> lot more memory per Stream and it is possible that what you are seeing
>> is a system that doesn't have enough memory to achieve steady state.
>>
>> If you are able to build the latest 9.0.x and test that, that could be
>> helpful. Alternatively, I could provide a test build for you to
>> experiment with.
>>
>> Some additional questions that might aid understanding:
>>
>> - What is the typical response size for one of these requests?
>> - How long does a typical test take to process?
>> - What are the GC roots for those RequestInfo objects?
>>
>> Thanks again,
>>
>> Mark
>>
>>
>>
>>
>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>> Hi Mark,
>>>
>>> Its the default APR connector with 150 Threads.
>>>
>>> Chirag
>>>
>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:
>>>
>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>>>> Thanks for the quick check Mark.
>>>>>
>>>>> These are the images I tried referring to:
>>>>>
>>>>> https://ibb.co/LzKtRgh
>>>>>
>>>>> https://ibb.co/2s7hqRL
>>>>>
>>>>> https://ibb.co/KmKj590
>>>>>
>>>>>
>>>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>>>
>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
>>>> look. Out of interest, how many threads is the Connector configured to
>> use?
>>>>
>>>> Mark
>>>>
>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Chirag
>>>>>
>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
>>>>>
>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and opens
>>>> new
>>>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>>>
>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>>>
>>>>>>>> For better understanding, let me explain my use case again:
>>>>>>>>
>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>>>> requirement is to close a connection after a configurable(say 5000)
>>>>>>>> number of requests/streams and open a new connection that continues
>> to
>>>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>>>
>>>>>>>> When I execute this use case under load, I see that after ~2hours my
>>>>>>>> requests fail and I get a series of errors like request
>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
>> close
>>>>>>>> exception on my client.
>>>>>>>> On further debugging, I found that it's a Tomcat memory problem and
>> it
>>>>>>>> goes OOM after sometime under heavy load with multiple connections
>>>> being
>>>>>>>> re-established by the clients.
>>>>>>>>
>>>>>>>> image.png
>>>>>>>>
>>>>>>>> image.png
>>>>>>>>
>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>>>
>>>>>>> Embedded images get dropped by the list software. Post those images
>>>>>>> somewhere we can see them.
>>>>>>>
>>>>>>>> Please let me know if you any experience with such a situation.
>> Thanks
>>>>>>>> in advance.
>>>>>>>
>>>>>>> Nothing comes to mind.
>>>>>>>
>>>>>>> I'll try some simple tests with HTTP/2.
>>>>>>
>>>>>> I don't see a memory leak (the memory is reclaimed eventually) but I
>> do
>>>>>> see possibilities to release memory associated with request processing
>>>>>> sooner.
>>>>>>
>>>>>> Right now you need to allocate more memory to the Java process to
>> enable
>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>>>
>>>>>> It looks like a reasonable chunk of memory is released when the
>>>>>> Connection closes that could be released earlier when the associated
>>>>>> Stream closes. I'll take a look at what can be done in that area. In
>> the
>>>>>> meantime, reducing the number of Streams you allow on a Connection
>>>>>> before it is closed should reduce overall memory usage.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Hey Mark,

*Are you using h2c or h2 in your test?*
We are using h2c


*Do you see the same issue if you switch the the NIO connector? Note
performance differences between NIO and NIO2 are very small.*

I havent tried with NIO honestly. Can quickly execute and check. Will
update the results.

*How long does a single request take to process?*
In normal scenarios, less than 3ms.

Thanks,
Chirag

On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <[hidden email]> wrote:

> Hi,
>
> Thanks for the additional information. The GC roots were particularly
> informative.
>
> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> HTTP/2 requests.
>
> Some further questions to try and track down what is going on:
>
> - Are you using h2c or h2 in your test?
>
> - Do you see the same issue if you switch the the NIO connector? Note
>   performance differences between NIO and NIO2 are very small.
>
> - How long does a single request take to process?
>
> Thanks,
>
> Mark
>
> On 26/06/2020 09:24, Chirag Dewan wrote:
> > Thanks Mark.
> >
> > *What is the typical response size for one of these requests? *
> > It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> > response in my production use case, but the purpose here was to isolate
> as
> > many things as possible. Hence a dummy response.
> >
> > *How long does a typical test take to process? *
> > I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> > of streams on my client was 500 - which means about 40 connections per
> > second.
> >
> > * What are the GC roots for those RequestInfo objects?*
> > https://ibb.co/fMRmCXZ
> >
> > I hope I was able to answer everything as expected. Thanks.
> >
> > On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
> >
> >> Thanks.
> >>
> >> I've looked at the code and I have tried various tests but I am unable
> >> to re-create a memory leak.
> >>
> >> The code used to (before I made a few changes this afternoon) retain a
> >> lot more memory per Stream and it is possible that what you are seeing
> >> is a system that doesn't have enough memory to achieve steady state.
> >>
> >> If you are able to build the latest 9.0.x and test that, that could be
> >> helpful. Alternatively, I could provide a test build for you to
> >> experiment with.
> >>
> >> Some additional questions that might aid understanding:
> >>
> >> - What is the typical response size for one of these requests?
> >> - How long does a typical test take to process?
> >> - What are the GC roots for those RequestInfo objects?
> >>
> >> Thanks again,
> >>
> >> Mark
> >>
> >>
> >>
> >>
> >> On 25/06/2020 15:10, Chirag Dewan wrote:
> >>> Hi Mark,
> >>>
> >>> Its the default APR connector with 150 Threads.
> >>>
> >>> Chirag
> >>>
> >>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:
> >>>
> >>>> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>>>> Thanks for the quick check Mark.
> >>>>>
> >>>>> These are the images I tried referring to:
> >>>>>
> >>>>> https://ibb.co/LzKtRgh
> >>>>>
> >>>>> https://ibb.co/2s7hqRL
> >>>>>
> >>>>> https://ibb.co/KmKj590
> >>>>>
> >>>>>
> >>>>> The last one is the MAT screenshot showing many RequestInfo objects.
> >>>>
> >>>> Thanks. That certainly looks like a memory leak. I'll take a closer
> >>>> look. Out of interest, how many threads is the Connector configured to
> >> use?
> >>>>
> >>>> Mark
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Chirag
> >>>>>
> >>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]>
> wrote:
> >>>>>
> >>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
> >>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
> opens
> >>>> new
> >>>>>>>> connections every second. In the memory dump, we see a lot of
> >>>>>>>> RequestInfo objects that are causing the memory spike.
> >>>>>>>>
> >>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
> >>>>>>>> request timed out on my client). This seems like a bug to me.
> >>>>>>>>
> >>>>>>>> For better understanding, let me explain my use case again:
> >>>>>>>>
> >>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>>>>>> requirement is to close a connection after a configurable(say
> 5000)
> >>>>>>>> number of requests/streams and open a new connection that
> continues
> >> to
> >>>>>>>> send requests. I close a connection by sending a GoAway frame.
> >>>>>>>>
> >>>>>>>> When I execute this use case under load, I see that after ~2hours
> my
> >>>>>>>> requests fail and I get a series of errors like request
> >>>>>>>> timeouts(5seconds), invalid window update frame, and connection
> >> close
> >>>>>>>> exception on my client.
> >>>>>>>> On further debugging, I found that it's a Tomcat memory problem
> and
> >> it
> >>>>>>>> goes OOM after sometime under heavy load with multiple connections
> >>>> being
> >>>>>>>> re-established by the clients.
> >>>>>>>>
> >>>>>>>> image.png
> >>>>>>>>
> >>>>>>>> image.png
> >>>>>>>>
> >>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>>>>>
> >>>>>>> Embedded images get dropped by the list software. Post those images
> >>>>>>> somewhere we can see them.
> >>>>>>>
> >>>>>>>> Please let me know if you any experience with such a situation.
> >> Thanks
> >>>>>>>> in advance.
> >>>>>>>
> >>>>>>> Nothing comes to mind.
> >>>>>>>
> >>>>>>> I'll try some simple tests with HTTP/2.
> >>>>>>
> >>>>>> I don't see a memory leak (the memory is reclaimed eventually) but I
> >> do
> >>>>>> see possibilities to release memory associated with request
> processing
> >>>>>> sooner.
> >>>>>>
> >>>>>> Right now you need to allocate more memory to the Java process to
> >> enable
> >>>>>> Tomcat to handle the HTTP/2 load it is presented with.
> >>>>>>
> >>>>>> It looks like a reasonable chunk of memory is released when the
> >>>>>> Connection closes that could be released earlier when the associated
> >>>>>> Stream closes. I'll take a look at what can be done in that area. In
> >> the
> >>>>>> meantime, reducing the number of Streams you allow on a Connection
> >>>>>> before it is closed should reduce overall memory usage.
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: [hidden email]
> >>>>>> For additional commands, e-mail: [hidden email]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: [hidden email]
> >>>> For additional commands, e-mail: [hidden email]
> >>>>
> >>>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
Aha!

h2c could be the significant factor here. Let me take a look.

Are you in a position to test against a dev build if the need arises?

Mark


On 26/06/2020 11:30, Chirag Dewan wrote:

> Hey Mark,
>
> *Are you using h2c or h2 in your test?*
> We are using h2c
>
>
> *Do you see the same issue if you switch the the NIO connector? Note
> performance differences between NIO and NIO2 are very small.*
>
> I havent tried with NIO honestly. Can quickly execute and check. Will
> update the results.
>
> *How long does a single request take to process?*
> In normal scenarios, less than 3ms.
>
> Thanks,
> Chirag
>
> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <[hidden email]> wrote:
>
>> Hi,
>>
>> Thanks for the additional information. The GC roots were particularly
>> informative.
>>
>> Those RequestInfo objects are associated with HTTP/1.1 requests, not
>> HTTP/2 requests.
>>
>> Some further questions to try and track down what is going on:
>>
>> - Are you using h2c or h2 in your test?
>>
>> - Do you see the same issue if you switch the the NIO connector? Note
>>   performance differences between NIO and NIO2 are very small.
>>
>> - How long does a single request take to process?
>>
>> Thanks,
>>
>> Mark
>>
>> On 26/06/2020 09:24, Chirag Dewan wrote:
>>> Thanks Mark.
>>>
>>> *What is the typical response size for one of these requests? *
>>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
>>> response in my production use case, but the purpose here was to isolate
>> as
>>> many things as possible. Hence a dummy response.
>>>
>>> *How long does a typical test take to process? *
>>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
>>> of streams on my client was 500 - which means about 40 connections per
>>> second.
>>>
>>> * What are the GC roots for those RequestInfo objects?*
>>> https://ibb.co/fMRmCXZ
>>>
>>> I hope I was able to answer everything as expected. Thanks.
>>>
>>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
>>>
>>>> Thanks.
>>>>
>>>> I've looked at the code and I have tried various tests but I am unable
>>>> to re-create a memory leak.
>>>>
>>>> The code used to (before I made a few changes this afternoon) retain a
>>>> lot more memory per Stream and it is possible that what you are seeing
>>>> is a system that doesn't have enough memory to achieve steady state.
>>>>
>>>> If you are able to build the latest 9.0.x and test that, that could be
>>>> helpful. Alternatively, I could provide a test build for you to
>>>> experiment with.
>>>>
>>>> Some additional questions that might aid understanding:
>>>>
>>>> - What is the typical response size for one of these requests?
>>>> - How long does a typical test take to process?
>>>> - What are the GC roots for those RequestInfo objects?
>>>>
>>>> Thanks again,
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>>>> Hi Mark,
>>>>>
>>>>> Its the default APR connector with 150 Threads.
>>>>>
>>>>> Chirag
>>>>>
>>>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:
>>>>>
>>>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>>>>>> Thanks for the quick check Mark.
>>>>>>>
>>>>>>> These are the images I tried referring to:
>>>>>>>
>>>>>>> https://ibb.co/LzKtRgh
>>>>>>>
>>>>>>> https://ibb.co/2s7hqRL
>>>>>>>
>>>>>>> https://ibb.co/KmKj590
>>>>>>>
>>>>>>>
>>>>>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>>>>>
>>>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
>>>>>> look. Out of interest, how many threads is the Connector configured to
>>>> use?
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Chirag
>>>>>>>
>>>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]>
>> wrote:
>>>>>>>
>>>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
>> opens
>>>>>> new
>>>>>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>>>>>
>>>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>>>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>>>>>
>>>>>>>>>> For better understanding, let me explain my use case again:
>>>>>>>>>>
>>>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>>>>>> requirement is to close a connection after a configurable(say
>> 5000)
>>>>>>>>>> number of requests/streams and open a new connection that
>> continues
>>>> to
>>>>>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>>>>>
>>>>>>>>>> When I execute this use case under load, I see that after ~2hours
>> my
>>>>>>>>>> requests fail and I get a series of errors like request
>>>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
>>>> close
>>>>>>>>>> exception on my client.
>>>>>>>>>> On further debugging, I found that it's a Tomcat memory problem
>> and
>>>> it
>>>>>>>>>> goes OOM after sometime under heavy load with multiple connections
>>>>>> being
>>>>>>>>>> re-established by the clients.
>>>>>>>>>>
>>>>>>>>>> image.png
>>>>>>>>>>
>>>>>>>>>> image.png
>>>>>>>>>>
>>>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>>>>>
>>>>>>>>> Embedded images get dropped by the list software. Post those images
>>>>>>>>> somewhere we can see them.
>>>>>>>>>
>>>>>>>>>> Please let me know if you any experience with such a situation.
>>>> Thanks
>>>>>>>>>> in advance.
>>>>>>>>>
>>>>>>>>> Nothing comes to mind.
>>>>>>>>>
>>>>>>>>> I'll try some simple tests with HTTP/2.
>>>>>>>>
>>>>>>>> I don't see a memory leak (the memory is reclaimed eventually) but I
>>>> do
>>>>>>>> see possibilities to release memory associated with request
>> processing
>>>>>>>> sooner.
>>>>>>>>
>>>>>>>> Right now you need to allocate more memory to the Java process to
>>>> enable
>>>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>>>>>
>>>>>>>> It looks like a reasonable chunk of memory is released when the
>>>>>>>> Connection closes that could be released earlier when the associated
>>>>>>>> Stream closes. I'll take a look at what can be done in that area. In
>>>> the
>>>>>>>> meantime, reducing the number of Streams you allow on a Connection
>>>>>>>> before it is closed should reduce overall memory usage.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Absolutely Mark. Shouldn't take long.

On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas, <[hidden email]> wrote:

> Aha!
>
> h2c could be the significant factor here. Let me take a look.
>
> Are you in a position to test against a dev build if the need arises?
>
> Mark
>
>
> On 26/06/2020 11:30, Chirag Dewan wrote:
> > Hey Mark,
> >
> > *Are you using h2c or h2 in your test?*
> > We are using h2c
> >
> >
> > *Do you see the same issue if you switch the the NIO connector? Note
> > performance differences between NIO and NIO2 are very small.*
> >
> > I havent tried with NIO honestly. Can quickly execute and check. Will
> > update the results.
> >
> > *How long does a single request take to process?*
> > In normal scenarios, less than 3ms.
> >
> > Thanks,
> > Chirag
> >
> > On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <[hidden email]> wrote:
> >
> >> Hi,
> >>
> >> Thanks for the additional information. The GC roots were particularly
> >> informative.
> >>
> >> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> >> HTTP/2 requests.
> >>
> >> Some further questions to try and track down what is going on:
> >>
> >> - Are you using h2c or h2 in your test?
> >>
> >> - Do you see the same issue if you switch the the NIO connector? Note
> >>   performance differences between NIO and NIO2 are very small.
> >>
> >> - How long does a single request take to process?
> >>
> >> Thanks,
> >>
> >> Mark
> >>
> >> On 26/06/2020 09:24, Chirag Dewan wrote:
> >>> Thanks Mark.
> >>>
> >>> *What is the typical response size for one of these requests? *
> >>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> >>> response in my production use case, but the purpose here was to isolate
> >> as
> >>> many things as possible. Hence a dummy response.
> >>>
> >>> *How long does a typical test take to process? *
> >>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
> number
> >>> of streams on my client was 500 - which means about 40 connections per
> >>> second.
> >>>
> >>> * What are the GC roots for those RequestInfo objects?*
> >>> https://ibb.co/fMRmCXZ
> >>>
> >>> I hope I was able to answer everything as expected. Thanks.
> >>>
> >>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
> >>>
> >>>> Thanks.
> >>>>
> >>>> I've looked at the code and I have tried various tests but I am unable
> >>>> to re-create a memory leak.
> >>>>
> >>>> The code used to (before I made a few changes this afternoon) retain a
> >>>> lot more memory per Stream and it is possible that what you are seeing
> >>>> is a system that doesn't have enough memory to achieve steady state.
> >>>>
> >>>> If you are able to build the latest 9.0.x and test that, that could be
> >>>> helpful. Alternatively, I could provide a test build for you to
> >>>> experiment with.
> >>>>
> >>>> Some additional questions that might aid understanding:
> >>>>
> >>>> - What is the typical response size for one of these requests?
> >>>> - How long does a typical test take to process?
> >>>> - What are the GC roots for those RequestInfo objects?
> >>>>
> >>>> Thanks again,
> >>>>
> >>>> Mark
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 25/06/2020 15:10, Chirag Dewan wrote:
> >>>>> Hi Mark,
> >>>>>
> >>>>> Its the default APR connector with 150 Threads.
> >>>>>
> >>>>> Chirag
> >>>>>
> >>>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:
> >>>>>
> >>>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>>>>>> Thanks for the quick check Mark.
> >>>>>>>
> >>>>>>> These are the images I tried referring to:
> >>>>>>>
> >>>>>>> https://ibb.co/LzKtRgh
> >>>>>>>
> >>>>>>> https://ibb.co/2s7hqRL
> >>>>>>>
> >>>>>>> https://ibb.co/KmKj590
> >>>>>>>
> >>>>>>>
> >>>>>>> The last one is the MAT screenshot showing many RequestInfo
> objects.
> >>>>>>
> >>>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
> >>>>>> look. Out of interest, how many threads is the Connector configured
> to
> >>>> use?
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Chirag
> >>>>>>>
> >>>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]>
> >> wrote:
> >>>>>>>
> >>>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
> >>>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
> >> opens
> >>>>>> new
> >>>>>>>>>> connections every second. In the memory dump, we see a lot of
> >>>>>>>>>> RequestInfo objects that are causing the memory spike.
> >>>>>>>>>>
> >>>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I
> get a
> >>>>>>>>>> request timed out on my client). This seems like a bug to me.
> >>>>>>>>>>
> >>>>>>>>>> For better understanding, let me explain my use case again:
> >>>>>>>>>>
> >>>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>>>>>>>> requirement is to close a connection after a configurable(say
> >> 5000)
> >>>>>>>>>> number of requests/streams and open a new connection that
> >> continues
> >>>> to
> >>>>>>>>>> send requests. I close a connection by sending a GoAway frame.
> >>>>>>>>>>
> >>>>>>>>>> When I execute this use case under load, I see that after
> ~2hours
> >> my
> >>>>>>>>>> requests fail and I get a series of errors like request
> >>>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
> >>>> close
> >>>>>>>>>> exception on my client.
> >>>>>>>>>> On further debugging, I found that it's a Tomcat memory problem
> >> and
> >>>> it
> >>>>>>>>>> goes OOM after sometime under heavy load with multiple
> connections
> >>>>>> being
> >>>>>>>>>> re-established by the clients.
> >>>>>>>>>>
> >>>>>>>>>> image.png
> >>>>>>>>>>
> >>>>>>>>>> image.png
> >>>>>>>>>>
> >>>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>>>>>>>
> >>>>>>>>> Embedded images get dropped by the list software. Post those
> images
> >>>>>>>>> somewhere we can see them.
> >>>>>>>>>
> >>>>>>>>>> Please let me know if you any experience with such a situation.
> >>>> Thanks
> >>>>>>>>>> in advance.
> >>>>>>>>>
> >>>>>>>>> Nothing comes to mind.
> >>>>>>>>>
> >>>>>>>>> I'll try some simple tests with HTTP/2.
> >>>>>>>>
> >>>>>>>> I don't see a memory leak (the memory is reclaimed eventually)
> but I
> >>>> do
> >>>>>>>> see possibilities to release memory associated with request
> >> processing
> >>>>>>>> sooner.
> >>>>>>>>
> >>>>>>>> Right now you need to allocate more memory to the Java process to
> >>>> enable
> >>>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
> >>>>>>>>
> >>>>>>>> It looks like a reasonable chunk of memory is released when the
> >>>>>>>> Connection closes that could be released earlier when the
> associated
> >>>>>>>> Stream closes. I'll take a look at what can be done in that area.
> In
> >>>> the
> >>>>>>>> meantime, reducing the number of Streams you allow on a Connection
> >>>>>>>> before it is closed should reduce overall memory usage.
> >>>>>>>>
> >>>>>>>> Mark
> >>>>>>>>
> >>>>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: [hidden email]
> >>>>>>>> For additional commands, e-mail: [hidden email]
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: [hidden email]
> >>>>>> For additional commands, e-mail: [hidden email]
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: [hidden email]
> >>>> For additional commands, e-mail: [hidden email]
> >>>>
> >>>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
On 26/06/2020 12:45, Chirag Dewan wrote:
> Absolutely Mark. Shouldn't take long.

Great. I think I have found a potential root cause. If I am right, NIO
will show the same issues NIO2 did.

I should have a test build for you shortly.

Mark


>
> On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas, <[hidden email]> wrote:
>
>> Aha!
>>
>> h2c could be the significant factor here. Let me take a look.
>>
>> Are you in a position to test against a dev build if the need arises?
>>
>> Mark
>>
>>
>> On 26/06/2020 11:30, Chirag Dewan wrote:
>>> Hey Mark,
>>>
>>> *Are you using h2c or h2 in your test?*
>>> We are using h2c
>>>
>>>
>>> *Do you see the same issue if you switch the the NIO connector? Note
>>> performance differences between NIO and NIO2 are very small.*
>>>
>>> I havent tried with NIO honestly. Can quickly execute and check. Will
>>> update the results.
>>>
>>> *How long does a single request take to process?*
>>> In normal scenarios, less than 3ms.
>>>
>>> Thanks,
>>> Chirag
>>>
>>> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <[hidden email]> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thanks for the additional information. The GC roots were particularly
>>>> informative.
>>>>
>>>> Those RequestInfo objects are associated with HTTP/1.1 requests, not
>>>> HTTP/2 requests.
>>>>
>>>> Some further questions to try and track down what is going on:
>>>>
>>>> - Are you using h2c or h2 in your test?
>>>>
>>>> - Do you see the same issue if you switch the the NIO connector? Note
>>>>   performance differences between NIO and NIO2 are very small.
>>>>
>>>> - How long does a single request take to process?
>>>>
>>>> Thanks,
>>>>
>>>> Mark
>>>>
>>>> On 26/06/2020 09:24, Chirag Dewan wrote:
>>>>> Thanks Mark.
>>>>>
>>>>> *What is the typical response size for one of these requests? *
>>>>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
>>>>> response in my production use case, but the purpose here was to isolate
>>>> as
>>>>> many things as possible. Hence a dummy response.
>>>>>
>>>>> *How long does a typical test take to process? *
>>>>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
>> number
>>>>> of streams on my client was 500 - which means about 40 connections per
>>>>> second.
>>>>>
>>>>> * What are the GC roots for those RequestInfo objects?*
>>>>> https://ibb.co/fMRmCXZ
>>>>>
>>>>> I hope I was able to answer everything as expected. Thanks.
>>>>>
>>>>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <[hidden email]> wrote:
>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> I've looked at the code and I have tried various tests but I am unable
>>>>>> to re-create a memory leak.
>>>>>>
>>>>>> The code used to (before I made a few changes this afternoon) retain a
>>>>>> lot more memory per Stream and it is possible that what you are seeing
>>>>>> is a system that doesn't have enough memory to achieve steady state.
>>>>>>
>>>>>> If you are able to build the latest 9.0.x and test that, that could be
>>>>>> helpful. Alternatively, I could provide a test build for you to
>>>>>> experiment with.
>>>>>>
>>>>>> Some additional questions that might aid understanding:
>>>>>>
>>>>>> - What is the typical response size for one of these requests?
>>>>>> - How long does a typical test take to process?
>>>>>> - What are the GC roots for those RequestInfo objects?
>>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>>>>>> Hi Mark,
>>>>>>>
>>>>>>> Its the default APR connector with 150 Threads.
>>>>>>>
>>>>>>> Chirag
>>>>>>>
>>>>>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <[hidden email]> wrote:
>>>>>>>
>>>>>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>>>>>>>> Thanks for the quick check Mark.
>>>>>>>>>
>>>>>>>>> These are the images I tried referring to:
>>>>>>>>>
>>>>>>>>> https://ibb.co/LzKtRgh
>>>>>>>>>
>>>>>>>>> https://ibb.co/2s7hqRL
>>>>>>>>>
>>>>>>>>> https://ibb.co/KmKj590
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The last one is the MAT screenshot showing many RequestInfo
>> objects.
>>>>>>>>
>>>>>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
>>>>>>>> look. Out of interest, how many threads is the Connector configured
>> to
>>>>>> use?
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Chirag
>>>>>>>>>
>>>>>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <[hidden email]>
>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
>>>> opens
>>>>>>>> new
>>>>>>>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>>>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>>>>>>>
>>>>>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I
>> get a
>>>>>>>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>>>>>>>
>>>>>>>>>>>> For better understanding, let me explain my use case again:
>>>>>>>>>>>>
>>>>>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>>>>>>>> requirement is to close a connection after a configurable(say
>>>> 5000)
>>>>>>>>>>>> number of requests/streams and open a new connection that
>>>> continues
>>>>>> to
>>>>>>>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>>>>>>>
>>>>>>>>>>>> When I execute this use case under load, I see that after
>> ~2hours
>>>> my
>>>>>>>>>>>> requests fail and I get a series of errors like request
>>>>>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
>>>>>> close
>>>>>>>>>>>> exception on my client.
>>>>>>>>>>>> On further debugging, I found that it's a Tomcat memory problem
>>>> and
>>>>>> it
>>>>>>>>>>>> goes OOM after sometime under heavy load with multiple
>> connections
>>>>>>>> being
>>>>>>>>>>>> re-established by the clients.
>>>>>>>>>>>>
>>>>>>>>>>>> image.png
>>>>>>>>>>>>
>>>>>>>>>>>> image.png
>>>>>>>>>>>>
>>>>>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>>>>>>>
>>>>>>>>>>> Embedded images get dropped by the list software. Post those
>> images
>>>>>>>>>>> somewhere we can see them.
>>>>>>>>>>>
>>>>>>>>>>>> Please let me know if you any experience with such a situation.
>>>>>> Thanks
>>>>>>>>>>>> in advance.
>>>>>>>>>>>
>>>>>>>>>>> Nothing comes to mind.
>>>>>>>>>>>
>>>>>>>>>>> I'll try some simple tests with HTTP/2.
>>>>>>>>>>
>>>>>>>>>> I don't see a memory leak (the memory is reclaimed eventually)
>> but I
>>>>>> do
>>>>>>>>>> see possibilities to release memory associated with request
>>>> processing
>>>>>>>>>> sooner.
>>>>>>>>>>
>>>>>>>>>> Right now you need to allocate more memory to the Java process to
>>>>>> enable
>>>>>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>>>>>>>
>>>>>>>>>> It looks like a reasonable chunk of memory is released when the
>>>>>>>>>> Connection closes that could be released earlier when the
>> associated
>>>>>>>>>> Stream closes. I'll take a look at what can be done in that area.
>> In
>>>>>> the
>>>>>>>>>> meantime, reducing the number of Streams you allow on a Connection
>>>>>>>>>> before it is closed should reduce overall memory usage.
>>>>>>>>>>
>>>>>>>>>> Mark
>>>>>>>>>>
>>>>>>>>>>
>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

markt
On 26/06/2020 12:48, Mark Thomas wrote:
> On 26/06/2020 12:45, Chirag Dewan wrote:
>> Absolutely Mark. Shouldn't take long.
>
> Great. I think I have found a potential root cause. If I am right, NIO
> will show the same issues NIO2 did.
>
> I should have a test build for you shortly.

Try this:
https://people.apache.org/~markt/dev/v9.0.37-dev/

Please note:

This is NOT an official Apache Release.

This build is only intended to be used to test this issue.

Using the test build is it your own risk.

Thanks,

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Connection Closure due to Fatal Stream with HTTP2

Chirag Dewan-2
Thanks a lot, Mark.

https://ibb.co/LgzFh6t - Memory snapshot after 15minutes of the test.
It's certainly better than the graph with 9.0.36, but I will wait for this
test to run for another few hours. I will update later.

Cheers,
Chirag

On Fri, Jun 26, 2020 at 6:20 PM Mark Thomas <[hidden email]> wrote:

> On 26/06/2020 12:48, Mark Thomas wrote:
> > On 26/06/2020 12:45, Chirag Dewan wrote:
> >> Absolutely Mark. Shouldn't take long.
> >
> > Great. I think I have found a potential root cause. If I am right, NIO
> > will show the same issues NIO2 did.
> >
> > I should have a test build for you shortly.
>
> Try this:
> https://people.apache.org/~markt/dev/v9.0.37-dev/
>
> Please note:
>
> This is NOT an official Apache Release.
>
> This build is only intended to be used to test this issue.
>
> Using the test build is it your own risk.
>
> Thanks,
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>