Timeout waiting to read data from client

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Timeout waiting to read data from client

Jerry Malcolm
I have a relatively new environment with a standalone tomcat (8.5)
running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP.  Its a direct
connection to port 443.  (Well technically, I have firewallD in the flow
in order to route the protected port 80 to port 8080 and 443 to 8443 for
TC).

I am doing some stress testing on the server and failing miserably.  I
am sending around 130 ajax calls in rapid succession using HTTP/2. 
These are all very simple small page (JSP) requests.  Not a lot of
processing required. The first ~75 requests process normally.  Then
everything hangs up.  In the tomcat logs I'm getting a bunch of "Timeout
waiting to read data from client" exceptions.  And in the stacktrace for
these exceptions, they are all occurring when I'm trying to access a
parameter from the request.  Looking at the request network timing in
the browser console, I see a bunch of the requests returning in typical
time of a few milliseconds.  Then another large block of requests that
all start returning around 4 seconds, then another block that wait until
8 seconds to return.  I've tried firefox and chrome with the same results.

I've been using httpd in front of TC for years.  So this is the first
time I'm running TC standalone.  It is very likely I've got some
parameters set horribly wrong.  But I have no clue where to start. This
is not a tiny EC2, and my internet connection is not showing any signs
of problems.  So I really don't think this is a performance-related
problem.  The problem is very consistent and reproducible with the same
counts of success/failure calls. What could be causing the "Timeout
waiting to read data from client" after 75 calls, and then cause blocks
of calls to wait 4 seconds, 8 seconds, etc before responding?  I really
need to handle more simultaneous load that this is currently allowing.

Thanks in advance for the education.

Jerry


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Timeout waiting to read data from client

Christopher Schultz-2
Jerry,

On 11/6/20 19:49, Jerry Malcolm wrote:

> I have a relatively new environment with a standalone tomcat (8.5)
> running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP.  Its a direct
> connection to port 443.  (Well technically, I have firewallD in the flow
> in order to route the protected port 80 to port 8080 and 443 to 8443 for
> TC).
>
> I am doing some stress testing on the server and failing miserably.  I
> am sending around 130 ajax calls in rapid succession using HTTP/2. These
> are all very simple small page (JSP) requests.  Not a lot of processing
> required. The first ~75 requests process normally.  Then everything
> hangs up.  In the tomcat logs I'm getting a bunch of "Timeout waiting to
> read data from client" exceptions.  And in the stacktrace for these
> exceptions, they are all occurring when I'm trying to access a parameter
> from the request.  Looking at the request network timing in the browser
> console, I see a bunch of the requests returning in typical time of a
> few milliseconds.  Then another large block of requests that all start
> returning around 4 seconds, then another block that wait until 8 seconds
> to return.  I've tried firefox and chrome with the same results.
>
> I've been using httpd in front of TC for years.  So this is the first
> time I'm running TC standalone.  It is very likely I've got some
> parameters set horribly wrong.  But I have no clue where to start. This
> is not a tiny EC2, and my internet connection is not showing any signs
> of problems.  So I really don't think this is a performance-related
> problem.  The problem is very consistent and reproducible with the same
> counts of success/failure calls. What could be causing the "Timeout
> waiting to read data from client" after 75 calls, and then cause blocks
> of calls to wait 4 seconds, 8 seconds, etc before responding?  I really
> need to handle more simultaneous load that this is currently allowing.
>
> Thanks in advance for the education.

Are you using HTTP Keepalives on your connections? Are you actually
re-using those connections in your test? What is your keepalive timeout
on your <Connector>. Actually, what is your whole <Connector> configuration?

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Timeout waiting to read data from client

Jerry Malcolm

On 11/7/2020 6:56 PM, Christopher Schultz wrote:

> Jerry,
>
> On 11/6/20 19:49, Jerry Malcolm wrote:
>> I have a relatively new environment with a standalone tomcat (8.5)
>> running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP.  Its a direct
>> connection to port 443.  (Well technically, I have firewallD in the
>> flow in order to route the protected port 80 to port 8080 and 443 to
>> 8443 for TC).
>>
>> I am doing some stress testing on the server and failing miserably. 
>> I am sending around 130 ajax calls in rapid succession using HTTP/2.
>> These are all very simple small page (JSP) requests.  Not a lot of
>> processing required. The first ~75 requests process normally.  Then
>> everything hangs up.  In the tomcat logs I'm getting a bunch of
>> "Timeout waiting to read data from client" exceptions.  And in the
>> stacktrace for these exceptions, they are all occurring when I'm
>> trying to access a parameter from the request.  Looking at the
>> request network timing in the browser console, I see a bunch of the
>> requests returning in typical time of a few milliseconds.  Then
>> another large block of requests that all start returning around 4
>> seconds, then another block that wait until 8 seconds to return. 
>> I've tried firefox and chrome with the same results.
>>
>> I've been using httpd in front of TC for years.  So this is the first
>> time I'm running TC standalone.  It is very likely I've got some
>> parameters set horribly wrong.  But I have no clue where to start.
>> This is not a tiny EC2, and my internet connection is not showing any
>> signs of problems.  So I really don't think this is a
>> performance-related problem.  The problem is very consistent and
>> reproducible with the same counts of success/failure calls. What
>> could be causing the "Timeout waiting to read data from client" after
>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8
>> seconds, etc before responding?  I really need to handle more
>> simultaneous load that this is currently allowing.
>>
>> Thanks in advance for the education.
>
> Are you using HTTP Keepalives on your connections? Are you actually
> re-using those connections in your test? What is your keepalive
> timeout on your <Connector>. Actually, what is your whole <Connector>
> configuration?
>
> -chris
>
Hi Chris, here are my two connector definitions from server.xml:

     <Connector
       port="8080"
       protocol="HTTP/1.1"
       connectionTimeout="20000"
       redirectPort="443" />

     <Connector
       port="8443"
       maxThreads="150"
       connectionTimeout="20000"
       SSLEnabled="true"
       scheme="https"
       secure="true"
       clientAuth="false"
       SSLCertificateFile="ssl/aaaaa.com/cert.pem"
       SSLCertificateChainFile="ssl/aaaaa.com/chain.pem"
       SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem">
         <UpgradeProtocol
className="org.apache.coyote.http2.Http2Protocol" />
     </Connector>

> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Timeout waiting to read data from client

markt
On 08/11/2020 01:33, Jerry Malcolm wrote:

>
> On 11/7/2020 6:56 PM, Christopher Schultz wrote:
>> Jerry,
>>
>> On 11/6/20 19:49, Jerry Malcolm wrote:
>>> I have a relatively new environment with a standalone tomcat (8.5)
>>> running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP.  Its a direct
>>> connection to port 443.  (Well technically, I have firewallD in the
>>> flow in order to route the protected port 80 to port 8080 and 443 to
>>> 8443 for TC).
>>>
>>> I am doing some stress testing on the server and failing miserably. 
>>> I am sending around 130 ajax calls in rapid succession using HTTP/2.
>>> These are all very simple small page (JSP) requests.  Not a lot of
>>> processing required. The first ~75 requests process normally.  Then
>>> everything hangs up.  In the tomcat logs I'm getting a bunch of
>>> "Timeout waiting to read data from client" exceptions.  And in the
>>> stacktrace for these exceptions, they are all occurring when I'm
>>> trying to access a parameter from the request.  Looking at the
>>> request network timing in the browser console, I see a bunch of the
>>> requests returning in typical time of a few milliseconds.  Then
>>> another large block of requests that all start returning around 4
>>> seconds, then another block that wait until 8 seconds to return. 
>>> I've tried firefox and chrome with the same results.
>>>
>>> I've been using httpd in front of TC for years.  So this is the first
>>> time I'm running TC standalone.  It is very likely I've got some
>>> parameters set horribly wrong.  But I have no clue where to start.
>>> This is not a tiny EC2, and my internet connection is not showing any
>>> signs of problems.  So I really don't think this is a
>>> performance-related problem.  The problem is very consistent and
>>> reproducible with the same counts of success/failure calls. What
>>> could be causing the "Timeout waiting to read data from client" after
>>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8
>>> seconds, etc before responding?  I really need to handle more
>>> simultaneous load that this is currently allowing.
>>>
>>> Thanks in advance for the education.
>>
>> Are you using HTTP Keepalives on your connections? Are you actually
>> re-using those connections in your test? What is your keepalive
>> timeout on your <Connector>. Actually, what is your whole <Connector>
>> configuration?
>>
>> -chris
>>
> Hi Chris, here are my two connector definitions from server.xml:
>
>     <Connector
>       port="8080"
>       protocol="HTTP/1.1"
>       connectionTimeout="20000"
>       redirectPort="443" />
>
>     <Connector
>       port="8443"
>       maxThreads="150"
>       connectionTimeout="20000"
>       SSLEnabled="true"
>       scheme="https"
>       secure="true"
>       clientAuth="false"
>       SSLCertificateFile="ssl/aaaaa.com/cert.pem"
>       SSLCertificateChainFile="ssl/aaaaa.com/chain.pem"
>       SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem">
>         <UpgradeProtocol
> className="org.apache.coyote.http2.Http2Protocol" />
>     </Connector>

How are you stress testing this? All on a single HTTP/2 connection or
multiple connections? With which tool?

You might want to test HTTP/1.1 requests (with and without TLS) to see
if the problem is specific to HTTP/2 or TLS as that should help narrow
down the root cause.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Timeout waiting to read data from client

Jerry Malcolm

On 11/9/2020 3:10 AM, Mark Thomas wrote:

> On 08/11/2020 01:33, Jerry Malcolm wrote:
>> On 11/7/2020 6:56 PM, Christopher Schultz wrote:
>>> Jerry,
>>>
>>> On 11/6/20 19:49, Jerry Malcolm wrote:
>>>> I have a relatively new environment with a standalone tomcat (8.5)
>>>> running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP.  Its a direct
>>>> connection to port 443.  (Well technically, I have firewallD in the
>>>> flow in order to route the protected port 80 to port 8080 and 443 to
>>>> 8443 for TC).
>>>>
>>>> I am doing some stress testing on the server and failing miserably.
>>>> I am sending around 130 ajax calls in rapid succession using HTTP/2.
>>>> These are all very simple small page (JSP) requests.  Not a lot of
>>>> processing required. The first ~75 requests process normally.  Then
>>>> everything hangs up.  In the tomcat logs I'm getting a bunch of
>>>> "Timeout waiting to read data from client" exceptions.  And in the
>>>> stacktrace for these exceptions, they are all occurring when I'm
>>>> trying to access a parameter from the request.  Looking at the
>>>> request network timing in the browser console, I see a bunch of the
>>>> requests returning in typical time of a few milliseconds.  Then
>>>> another large block of requests that all start returning around 4
>>>> seconds, then another block that wait until 8 seconds to return.
>>>> I've tried firefox and chrome with the same results.
>>>>
>>>> I've been using httpd in front of TC for years.  So this is the first
>>>> time I'm running TC standalone.  It is very likely I've got some
>>>> parameters set horribly wrong.  But I have no clue where to start.
>>>> This is not a tiny EC2, and my internet connection is not showing any
>>>> signs of problems.  So I really don't think this is a
>>>> performance-related problem.  The problem is very consistent and
>>>> reproducible with the same counts of success/failure calls. What
>>>> could be causing the "Timeout waiting to read data from client" after
>>>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8
>>>> seconds, etc before responding?  I really need to handle more
>>>> simultaneous load that this is currently allowing.
>>>>
>>>> Thanks in advance for the education.
>>> Are you using HTTP Keepalives on your connections? Are you actually
>>> re-using those connections in your test? What is your keepalive
>>> timeout on your <Connector>. Actually, what is your whole <Connector>
>>> configuration?
>>>
>>> -chris
>>>
>> Hi Chris, here are my two connector definitions from server.xml:
>>
>>      <Connector
>>        port="8080"
>>        protocol="HTTP/1.1"
>>        connectionTimeout="20000"
>>        redirectPort="443" />
>>
>>      <Connector
>>        port="8443"
>>        maxThreads="150"
>>        connectionTimeout="20000"
>>        SSLEnabled="true"
>>        scheme="https"
>>        secure="true"
>>        clientAuth="false"
>>        SSLCertificateFile="ssl/aaaaa.com/cert.pem"
>>        SSLCertificateChainFile="ssl/aaaaa.com/chain.pem"
>>        SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem">
>>          <UpgradeProtocol
>> className="org.apache.coyote.http2.Http2Protocol" />
>>      </Connector>
> How are you stress testing this? All on a single HTTP/2 connection or
> multiple connections? With which tool?
>
> You might want to test HTTP/1.1 requests (with and without TLS) to see
> if the problem is specific to HTTP/2 or TLS as that should help narrow
> down the root cause.
>
> Mark

Hi Mark, technically it's not a 'designed' stress test.  It's real
production code that just happens to stress the server more than usual. 
It's just a page that makes a bunch of ajax calls, and the responses to
each of those issue a second ajax call.

If you don't see anything obvious in my configuration, we will
definitely pursue the http/1.1 options, etc.  I just wanted to eliminate
the chance of obvious 'pilot error' before digging deeper.

Specifically, where is that error detected in the TC flow? In my logs it
fails on getting request parameters.  It sounds like the input reader
for the request is getting blocked.    But the first part of the request
is getting in since it does route to the appropriate JSP.  Just seems
strange that the http/2 or ssl layers would let half of the request in
and then block the rest of the request.  The browser appears to be
sending everything.  And it fails the same using firefox or chrome.  Any
ideas?

Thx

>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Timeout waiting to read data from client

Jerry Malcolm
On 11/9/2020 11:05 AM, Jerry Malcolm wrote:

>
> On 11/9/2020 3:10 AM, Mark Thomas wrote:
>> On 08/11/2020 01:33, Jerry Malcolm wrote:
>>> On 11/7/2020 6:56 PM, Christopher Schultz wrote:
>>>> Jerry,
>>>>
>>>> On 11/6/20 19:49, Jerry Malcolm wrote:
>>>>> I have a relatively new environment with a standalone tomcat (8.5)
>>>>> running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP. Its a direct
>>>>> connection to port 443.  (Well technically, I have firewallD in the
>>>>> flow in order to route the protected port 80 to port 8080 and 443 to
>>>>> 8443 for TC).
>>>>>
>>>>> I am doing some stress testing on the server and failing miserably.
>>>>> I am sending around 130 ajax calls in rapid succession using HTTP/2.
>>>>> These are all very simple small page (JSP) requests.  Not a lot of
>>>>> processing required. The first ~75 requests process normally.  Then
>>>>> everything hangs up.  In the tomcat logs I'm getting a bunch of
>>>>> "Timeout waiting to read data from client" exceptions. And in the
>>>>> stacktrace for these exceptions, they are all occurring when I'm
>>>>> trying to access a parameter from the request.  Looking at the
>>>>> request network timing in the browser console, I see a bunch of the
>>>>> requests returning in typical time of a few milliseconds. Then
>>>>> another large block of requests that all start returning around 4
>>>>> seconds, then another block that wait until 8 seconds to return.
>>>>> I've tried firefox and chrome with the same results.
>>>>>
>>>>> I've been using httpd in front of TC for years.  So this is the first
>>>>> time I'm running TC standalone.  It is very likely I've got some
>>>>> parameters set horribly wrong.  But I have no clue where to start.
>>>>> This is not a tiny EC2, and my internet connection is not showing any
>>>>> signs of problems.  So I really don't think this is a
>>>>> performance-related problem.  The problem is very consistent and
>>>>> reproducible with the same counts of success/failure calls. What
>>>>> could be causing the "Timeout waiting to read data from client" after
>>>>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8
>>>>> seconds, etc before responding?  I really need to handle more
>>>>> simultaneous load that this is currently allowing.
>>>>>
>>>>> Thanks in advance for the education.
>>>> Are you using HTTP Keepalives on your connections? Are you actually
>>>> re-using those connections in your test? What is your keepalive
>>>> timeout on your <Connector>. Actually, what is your whole <Connector>
>>>> configuration?
>>>>
>>>> -chris
>>>>
>>> Hi Chris, here are my two connector definitions from server.xml:
>>>
>>>      <Connector
>>>        port="8080"
>>>        protocol="HTTP/1.1"
>>>        connectionTimeout="20000"
>>>        redirectPort="443" />
>>>
>>>      <Connector
>>>        port="8443"
>>>        maxThreads="150"
>>>        connectionTimeout="20000"
>>>        SSLEnabled="true"
>>>        scheme="https"
>>>        secure="true"
>>>        clientAuth="false"
>>>        SSLCertificateFile="ssl/aaaaa.com/cert.pem"
>>>        SSLCertificateChainFile="ssl/aaaaa.com/chain.pem"
>>>        SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem">
>>>          <UpgradeProtocol
>>> className="org.apache.coyote.http2.Http2Protocol" />
>>>      </Connector>
>> How are you stress testing this? All on a single HTTP/2 connection or
>> multiple connections? With which tool?
>>
>> You might want to test HTTP/1.1 requests (with and without TLS) to see
>> if the problem is specific to HTTP/2 or TLS as that should help narrow
>> down the root cause.
>>
>> Mark
>
> Hi Mark, technically it's not a 'designed' stress test.  It's real
> production code that just happens to stress the server more than
> usual.  It's just a page that makes a bunch of ajax calls, and the
> responses to each of those issue a second ajax call.
>
> If you don't see anything obvious in my configuration, we will
> definitely pursue the http/1.1 options, etc.  I just wanted to
> eliminate the chance of obvious 'pilot error' before digging deeper.
>
> Specifically, where is that error detected in the TC flow? In my logs
> it fails on getting request parameters.  It sounds like the input
> reader for the request is getting blocked.    But the first part of
> the request is getting in since it does route to the appropriate JSP. 
> Just seems strange that the http/2 or ssl layers would let half of the
> request in and then block the rest of the request.  The browser
> appears to be sending everything.  And it fails the same using firefox
> or chrome.  Any ideas?
>
> Thx
>
>
Update on this.  One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM
after things locked up.  I removed the http2 'upgrade protocol' line
from my connector, and everything works.  So it's apparently something
wrong with my http2 setup.  Ideas? (See my connector config above in
this thread).

Thx

Reply | Threaded
Open this post in threaded view
|

Re: Timeout waiting to read data from client

mgrigorov
On Thu, Nov 12, 2020 at 12:34 AM Jerry Malcolm <[hidden email]>
wrote:

> On 11/9/2020 11:05 AM, Jerry Malcolm wrote:
> >
> > On 11/9/2020 3:10 AM, Mark Thomas wrote:
> >> On 08/11/2020 01:33, Jerry Malcolm wrote:
> >>> On 11/7/2020 6:56 PM, Christopher Schultz wrote:
> >>>> Jerry,
> >>>>
> >>>> On 11/6/20 19:49, Jerry Malcolm wrote:
> >>>>> I have a relatively new environment with a standalone tomcat (8.5)
> >>>>> running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP. Its a direct
> >>>>> connection to port 443.  (Well technically, I have firewallD in the
> >>>>> flow in order to route the protected port 80 to port 8080 and 443 to
> >>>>> 8443 for TC).
> >>>>>
> >>>>> I am doing some stress testing on the server and failing miserably.
> >>>>> I am sending around 130 ajax calls in rapid succession using HTTP/2.
> >>>>> These are all very simple small page (JSP) requests.  Not a lot of
> >>>>> processing required. The first ~75 requests process normally.  Then
> >>>>> everything hangs up.  In the tomcat logs I'm getting a bunch of
> >>>>> "Timeout waiting to read data from client" exceptions. And in the
> >>>>> stacktrace for these exceptions, they are all occurring when I'm
> >>>>> trying to access a parameter from the request.  Looking at the
> >>>>> request network timing in the browser console, I see a bunch of the
> >>>>> requests returning in typical time of a few milliseconds. Then
> >>>>> another large block of requests that all start returning around 4
> >>>>> seconds, then another block that wait until 8 seconds to return.
> >>>>> I've tried firefox and chrome with the same results.
> >>>>>
> >>>>> I've been using httpd in front of TC for years.  So this is the first
> >>>>> time I'm running TC standalone.  It is very likely I've got some
> >>>>> parameters set horribly wrong.  But I have no clue where to start.
> >>>>> This is not a tiny EC2, and my internet connection is not showing any
> >>>>> signs of problems.  So I really don't think this is a
> >>>>> performance-related problem.  The problem is very consistent and
> >>>>> reproducible with the same counts of success/failure calls. What
> >>>>> could be causing the "Timeout waiting to read data from client" after
> >>>>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8
> >>>>> seconds, etc before responding?  I really need to handle more
> >>>>> simultaneous load that this is currently allowing.
> >>>>>
> >>>>> Thanks in advance for the education.
> >>>> Are you using HTTP Keepalives on your connections? Are you actually
> >>>> re-using those connections in your test? What is your keepalive
> >>>> timeout on your <Connector>. Actually, what is your whole <Connector>
> >>>> configuration?
> >>>>
> >>>> -chris
> >>>>
> >>> Hi Chris, here are my two connector definitions from server.xml:
> >>>
> >>>      <Connector
> >>>        port="8080"
> >>>        protocol="HTTP/1.1"
> >>>        connectionTimeout="20000"
> >>>        redirectPort="443" />
> >>>
> >>>      <Connector
> >>>        port="8443"
> >>>        maxThreads="150"
> >>>        connectionTimeout="20000"
> >>>        SSLEnabled="true"
> >>>        scheme="https"
> >>>        secure="true"
> >>>        clientAuth="false"
> >>>        SSLCertificateFile="ssl/aaaaa.com/cert.pem"
> >>>        SSLCertificateChainFile="ssl/aaaaa.com/chain.pem"
> >>>        SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem">
> >>>          <UpgradeProtocol
> >>> className="org.apache.coyote.http2.Http2Protocol" />
> >>>      </Connector>
> >> How are you stress testing this? All on a single HTTP/2 connection or
> >> multiple connections? With which tool?
> >>
> >> You might want to test HTTP/1.1 requests (with and without TLS) to see
> >> if the problem is specific to HTTP/2 or TLS as that should help narrow
> >> down the root cause.
> >>
> >> Mark
> >
> > Hi Mark, technically it's not a 'designed' stress test.  It's real
> > production code that just happens to stress the server more than
> > usual.  It's just a page that makes a bunch of ajax calls, and the
> > responses to each of those issue a second ajax call.
> >
> > If you don't see anything obvious in my configuration, we will
> > definitely pursue the http/1.1 options, etc.  I just wanted to
> > eliminate the chance of obvious 'pilot error' before digging deeper.
> >
> > Specifically, where is that error detected in the TC flow? In my logs
> > it fails on getting request parameters.  It sounds like the input
> > reader for the request is getting blocked.    But the first part of
> > the request is getting in since it does route to the appropriate JSP.
> > Just seems strange that the http/2 or ssl layers would let half of the
> > request in and then block the rest of the request.  The browser
> > appears to be sending everything.  And it fails the same using firefox
> > or chrome.  Any ideas?
> >
> > Thx
> >
> >
> Update on this.  One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM
> after things locked up.  I removed the http2 'upgrade protocol' line
> from my connector, and everything works.  So it's apparently something
> wrong with my http2 setup.  Ideas? (See my connector config above in
> this thread).
>

Which version exactly do you use ?
There were several big improvements to HTTP2 in the last few releases.


>
> Thx
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Timeout waiting to read data from client

markt
In reply to this post by Jerry Malcolm
On 11/11/2020 22:32, Jerry Malcolm wrote:

> On 11/9/2020 11:05 AM, Jerry Malcolm wrote:
>>
>> On 11/9/2020 3:10 AM, Mark Thomas wrote:
>>> On 08/11/2020 01:33, Jerry Malcolm wrote:
>>>> On 11/7/2020 6:56 PM, Christopher Schultz wrote:
>>>>> Jerry,
>>>>>
>>>>> On 11/6/20 19:49, Jerry Malcolm wrote:
>>>>>> I have a relatively new environment with a standalone tomcat (8.5)
>>>>>> running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP. Its a direct
>>>>>> connection to port 443.  (Well technically, I have firewallD in the
>>>>>> flow in order to route the protected port 80 to port 8080 and 443 to
>>>>>> 8443 for TC).
>>>>>>
>>>>>> I am doing some stress testing on the server and failing miserably.
>>>>>> I am sending around 130 ajax calls in rapid succession using HTTP/2.
>>>>>> These are all very simple small page (JSP) requests.  Not a lot of
>>>>>> processing required. The first ~75 requests process normally.  Then
>>>>>> everything hangs up.  In the tomcat logs I'm getting a bunch of
>>>>>> "Timeout waiting to read data from client" exceptions. And in the
>>>>>> stacktrace for these exceptions, they are all occurring when I'm
>>>>>> trying to access a parameter from the request.  Looking at the
>>>>>> request network timing in the browser console, I see a bunch of the
>>>>>> requests returning in typical time of a few milliseconds. Then
>>>>>> another large block of requests that all start returning around 4
>>>>>> seconds, then another block that wait until 8 seconds to return.
>>>>>> I've tried firefox and chrome with the same results.
>>>>>>
>>>>>> I've been using httpd in front of TC for years.  So this is the first
>>>>>> time I'm running TC standalone.  It is very likely I've got some
>>>>>> parameters set horribly wrong.  But I have no clue where to start.
>>>>>> This is not a tiny EC2, and my internet connection is not showing any
>>>>>> signs of problems.  So I really don't think this is a
>>>>>> performance-related problem.  The problem is very consistent and
>>>>>> reproducible with the same counts of success/failure calls. What
>>>>>> could be causing the "Timeout waiting to read data from client" after
>>>>>> 75 calls, and then cause blocks of calls to wait 4 seconds, 8
>>>>>> seconds, etc before responding?  I really need to handle more
>>>>>> simultaneous load that this is currently allowing.
>>>>>>
>>>>>> Thanks in advance for the education.
>>>>> Are you using HTTP Keepalives on your connections? Are you actually
>>>>> re-using those connections in your test? What is your keepalive
>>>>> timeout on your <Connector>. Actually, what is your whole <Connector>
>>>>> configuration?
>>>>>
>>>>> -chris
>>>>>
>>>> Hi Chris, here are my two connector definitions from server.xml:
>>>>
>>>>      <Connector
>>>>        port="8080"
>>>>        protocol="HTTP/1.1"
>>>>        connectionTimeout="20000"
>>>>        redirectPort="443" />
>>>>
>>>>      <Connector
>>>>        port="8443"
>>>>        maxThreads="150"
>>>>        connectionTimeout="20000"
>>>>        SSLEnabled="true"
>>>>        scheme="https"
>>>>        secure="true"
>>>>        clientAuth="false"
>>>>        SSLCertificateFile="ssl/aaaaa.com/cert.pem"
>>>>        SSLCertificateChainFile="ssl/aaaaa.com/chain.pem"
>>>>        SSLCertificateKeyFile="ssl/aaaaa.com/privkey.pem">
>>>>          <UpgradeProtocol
>>>> className="org.apache.coyote.http2.Http2Protocol" />
>>>>      </Connector>
>>> How are you stress testing this? All on a single HTTP/2 connection or
>>> multiple connections? With which tool?
>>>
>>> You might want to test HTTP/1.1 requests (with and without TLS) to see
>>> if the problem is specific to HTTP/2 or TLS as that should help narrow
>>> down the root cause.
>>>
>>> Mark
>>
>> Hi Mark, technically it's not a 'designed' stress test.  It's real
>> production code that just happens to stress the server more than
>> usual.  It's just a page that makes a bunch of ajax calls, and the
>> responses to each of those issue a second ajax call.
>>
>> If you don't see anything obvious in my configuration, we will
>> definitely pursue the http/1.1 options, etc.  I just wanted to
>> eliminate the chance of obvious 'pilot error' before digging deeper.
>>
>> Specifically, where is that error detected in the TC flow? In my logs
>> it fails on getting request parameters.  It sounds like the input
>> reader for the request is getting blocked.    But the first part of
>> the request is getting in since it does route to the appropriate JSP. 
>> Just seems strange that the http/2 or ssl layers would let half of the
>> request in and then block the rest of the request.  The browser
>> appears to be sending everything.  And it fails the same using firefox
>> or chrome.  Any ideas?
>>
>> Thx
>>
>>
> Update on this.  One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM
> after things locked up.  I removed the http2 'upgrade protocol' line
> from my connector, and everything works.  So it's apparently something
> wrong with my http2 setup.  Ideas? (See my connector config above in
> this thread).

Tomcat only issues that error if the client ignores (in violation of the
HTTP/2 spec) the server specified limit on the number of concurrent streams.

Take a look at the "maxConcurrentStreams" and
"maxConcurrentStreamExecution" attributes for Tomcat's HTTP/2 configuration:
http://tomcat.apache.org/tomcat-9.0-doc/config/http2.html

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]