GOAWAY sent on 404 with large payload in http2

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

GOAWAY sent on 404 with large payload in http2

Doug Whitfield
Hi folks,

It is unclear if this is a Tomcat issue, a protocol issue, or something else. I would like some help figuring out if it is a Tomcat issue and then resolving the issue if it is. We have seen this issue in Tomcat 9.0.38 through 9.0.43.

For a handful of requests, Tomcat sends GOAWAY with below reason :
GOAWAY with FRAME_SIZE_ERROR : The payload is [2105376] bytes long but the maximum frame size is [16384]
The payload size here from the user point of view is around 55 KB, but we’ve tested similar payload sizes with similar results.



Steps to reproduce:
1. client sends an request of size 90KB to an unknown URL (means servlet-context was not deployed) towards tomcat.
2. Tomcat sends 404 page not found which is expected.
3. Tomcats sends RST_STREAM with CANCEL as reason.
4. Tomcat doesn't send WINDOW_UPDATE after this.
5. Client was not able to send further DATA frames towards Tomcat.
6. Client is able to send only HEADERS (to an deployed URL), but no DATA, all requests on that connection fails.
The above flow works properly when client sends an request to an correct deployed URL. The obvious response is to just use the correct URLs in applications, but we want to make sure this is not a security vulnerability.

All properties are the tomcat defaults.
Here are the connector details:
Connector connector = new Connector();
connector.setPort(1080);
Http2Protocol http2Protocol = new Http2Protocol();
connector.addUpgradeProtocol( http2Protocol );
tomcat.setConnector(connector);


Windows 10 (but also reproduced on Red Hat 7.4)
Processor: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz 1.90GHz
RAM:16 GB
System Type : 64 bit

How we reproduced:
With JMeter as simulation client, on configuring 700 threads (700 connections) to connect towards Tomcat Server 9.0.43 embedded in the application and on sending 20 requests per second with payload of 55KB (same request with just one json value sent uniquely via the Random number generator) and on running the test in an infinite loop , few requests are getting timed out . On analyzing the PCAP for the requests that timed-out we see that tomcat sends GOAWAY with  PROTOCOL ERROR / FRAME_SIZE_ERROR .

Client:
JMeter 5.3 with additional HTTP2 sampler and Xmx 5g.
No of threads: 700
Ramp-up period: 10 seconds
Loop: Infinite
Payload size: around 55KB
Constant Throughput Timer added to limit the tps to 20.
Random Variable Generator added to the JSON request to uniquely identify for which request the exception is printed and to map it in the PCAP collected .
Response Timeout : 5000 ms

SERVER:
The input requests are processed asynchronously with 40 threads.




Are there any logs I should look at, and at what log level? There wasn’t anything obviously useful at FINER.

Please let me know if there is any additional information that would be useful.


Best Regards,
--

Doug Whitfield | Enterprise Architect, OpenLogic<https://www.openlogic.com/?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link>
Perforce Software<http://www.perforce.com/?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link>
Visit us on: LinkedIn<https://www.linkedin.com/company/perforce?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link> | Twitter<https://twitter.com/perforce?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link> | Facebook<https://www.facebook.com/perforce/?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link> | YouTube<https://www.youtube.com/user/perforcesoftware?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link>




This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.

Reply | Threaded
Open this post in threaded view
|

Re: GOAWAY sent on 404 with large payload in http2

Doug Whitfield
Regarding the email thread with this title: “Embedded Tomcat 9.0.43 : WINDOW_UPDATE not sent when receiving http2 requests over unknown url”

That looks exactly like our issue, but with slightly different numbers.

From: Doug Whitfield <[hidden email]>
Date: Tuesday, March 9, 2021 at 4:02 PM
To: [hidden email] <[hidden email]>
Subject: GOAWAY sent on 404 with large payload in http2
Hi folks,

It is unclear if this is a Tomcat issue, a protocol issue, or something else. I would like some help figuring out if it is a Tomcat issue and then resolving the issue if it is. We have seen this issue in Tomcat 9.0.38 through 9.0.43.

For a handful of requests, Tomcat sends GOAWAY with below reason :
GOAWAY with FRAME_SIZE_ERROR : The payload is [2105376] bytes long but the maximum frame size is [16384]
The payload size here from the user point of view is around 55 KB, but we’ve tested similar payload sizes with similar results.



Steps to reproduce:
1. client sends an request of size 90KB to an unknown URL (means servlet-context was not deployed) towards tomcat.
2. Tomcat sends 404 page not found which is expected.
3. Tomcats sends RST_STREAM with CANCEL as reason.
4. Tomcat doesn't send WINDOW_UPDATE after this.
5. Client was not able to send further DATA frames towards Tomcat.
6. Client is able to send only HEADERS (to an deployed URL), but no DATA, all requests on that connection fails.
The above flow works properly when client sends an request to an correct deployed URL. The obvious response is to just use the correct URLs in applications, but we want to make sure this is not a security vulnerability.

All properties are the tomcat defaults.
Here are the connector details:
Connector connector = new Connector();
connector.setPort(1080);
Http2Protocol http2Protocol = new Http2Protocol();
connector.addUpgradeProtocol( http2Protocol );
tomcat.setConnector(connector);


Windows 10 (but also reproduced on Red Hat 7.4)
Processor: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz 1.90GHz
RAM:16 GB
System Type : 64 bit

How we reproduced:
With JMeter as simulation client, on configuring 700 threads (700 connections) to connect towards Tomcat Server 9.0.43 embedded in the application and on sending 20 requests per second with payload of 55KB (same request with just one json value sent uniquely via the Random number generator) and on running the test in an infinite loop , few requests are getting timed out . On analyzing the PCAP for the requests that timed-out we see that tomcat sends GOAWAY with  PROTOCOL ERROR / FRAME_SIZE_ERROR .

Client:
JMeter 5.3 with additional HTTP2 sampler and Xmx 5g.
No of threads: 700
Ramp-up period: 10 seconds
Loop: Infinite
Payload size: around 55KB
Constant Throughput Timer added to limit the tps to 20.
Random Variable Generator added to the JSON request to uniquely identify for which request the exception is printed and to map it in the PCAP collected .
Response Timeout : 5000 ms

SERVER:
The input requests are processed asynchronously with 40 threads.




Are there any logs I should look at, and at what log level? There wasn’t anything obviously useful at FINER.

Please let me know if there is any additional information that would be useful.


Best Regards,
--

Doug Whitfield | Enterprise Architect, OpenLogic<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.openlogic.com%2F%3Futm_leadsource%3Demail-signature%26utm_source%3Doutlook-direct-email%26utm_medium%3Demail%26utm_campaign%3D2019-common%26utm_content%3Demail-signature-link&amp;data=04%7C01%7Cdwhitfield%40perforce.com%7Cee15754dd3ae42fc792408d8e34709a1%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637509241697963952%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=LAwM%2FysWiVMfHqiZ4OQ4bw7YB8gq4CZepoIz3mkqduQ%3D&amp;reserved=0>
Perforce Software<http://www.perforce.com/?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link>
Visit us on: LinkedIn<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fperforce%3Futm_leadsource%3Demail-signature%26utm_source%3Doutlook-direct-email%26utm_medium%3Demail%26utm_campaign%3D2019-common%26utm_content%3Demail-signature-link&amp;data=04%7C01%7Cdwhitfield%40perforce.com%7Cee15754dd3ae42fc792408d8e34709a1%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637509241697973946%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=8%2Fw8RczUp8k4glOsUxCOB5wg8jejdb11wk1xCY%2FtRuA%3D&amp;reserved=0> | Twitter<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fperforce%3Futm_leadsource%3Demail-signature%26utm_source%3Doutlook-direct-email%26utm_medium%3Demail%26utm_campaign%3D2019-common%26utm_content%3Demail-signature-link&amp;data=04%7C01%7Cdwhitfield%40perforce.com%7Cee15754dd3ae42fc792408d8e34709a1%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637509241697973946%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=5iZYDq8b4W2rvDAyPcrolSbC55VYNugP4tgVz5mAeoQ%3D&amp;reserved=0> | Facebook<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.facebook.com%2Fperforce%2F%3Futm_leadsource%3Demail-signature%26utm_source%3Doutlook-direct-email%26utm_medium%3Demail%26utm_campaign%3D2019-common%26utm_content%3Demail-signature-link&amp;data=04%7C01%7Cdwhitfield%40perforce.com%7Cee15754dd3ae42fc792408d8e34709a1%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637509241697973946%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=gqt7qUO9%2BQ3wGu3PQXUB4OhAH2lhdKyjj%2F1wCjMdP5o%3D&amp;reserved=0> | YouTube<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fuser%2Fperforcesoftware%3Futm_leadsource%3Demail-signature%26utm_source%3Doutlook-direct-email%26utm_medium%3Demail%26utm_campaign%3D2019-common%26utm_content%3Demail-signature-link&amp;data=04%7C01%7Cdwhitfield%40perforce.com%7Cee15754dd3ae42fc792408d8e34709a1%7C95b666d19a7549ab95a38969fbcdc08c%7C0%7C0%7C637509241697973946%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=6WRjwN%2Ff7nrohgMa2G%2F5PMx5Uw5m1O6xUbsNo3MAvK4%3D&amp;reserved=0>




This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.


This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.