https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
Bug ID: 65131 Summary: OpenSSLEngine errors on a connection affect other connections Product: Tomcat 9 Version: 9.0.34 Hardware: PC OS: Linux Status: NEW Severity: normal Priority: P2 Component: Connectors Assignee: [hidden email] Reporter: [hidden email] Target Milestone: ----- This issue was previously reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=62054 but was discarded because it was not reproducible. However, I have been able to reproduce it. The setup is: tomcat 9.0.34 HTTPNIO connector, with OpenSSLEngine implementation openssl 1.0.2u To reproduce, configure a secure connector with a self-signed certificate. Then, in a loop, access the secure connector using curl, but ignore the self-signed certificate errors (-k), like this: for i in {1..1000} ; do curl -q -k https://hostname >/dev/null 2>&1; echo $?; done The exit code should always be zero if the connection is successful. While the test is running, access the same interface, but let curl fail the connection by letting it verify the certificate: curl https://hostname curl: (60) Peer's certificate issuer has been marked as not trusted by the user. The test will start reporting connection errors (exit code 52). Run both in a tight loop to get many errors. The following stack trace is visible in tomcat logs: FINE: OpenSSL error: [336462231] message: [error:140E0197:SSL routines:SSL_shutdown:shutdown while in init] Feb 08, 2021 11:41:19 AM org.apache.coyote.http11.Http11Processor service FINE: Error parsing HTTP request header javax.net.ssl.SSLException: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init at org.apache.tomcat.util.net.openssl.OpenSSLEngine.checkLastError(OpenSSLEngine.java:946) at org.apache.tomcat.util.net.openssl.OpenSSLEngine.pendingReadableBytesInSSL(OpenSSLEngine.java:631) at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:558) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:629) at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:617) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1229) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1141) at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:761) at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:356) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at com.broadsoft.xsp.platform.InstrumentedExecutor$Tracker.run(InstrumentedExecutor.java:413) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) My feeling is that this is related to the same problem that led to this fix: https://github.com/apache/tomcat/commit/ebab25a3576a315ca676db7ce2366e53f9dcf311 It looks like, in this case, there are multiple errors waiting in the openssl error queue, and just reading one error is not sufficient to clear the queue. Therefore, SSL errors on a connection are permitted to propagate to other connections as long as they are processed on the same thread. As an aside, I also noticed the following NPE stacktrace in the logs, but I'm not sure if it's related. It may be a different scenario that caused it. I think it's related to SecureNioChannel.processSNI returning/throwing before creating the SSLEngine. Feb 08, 2021 11:48:00 AM org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper doClose SEVERE: Failed to close channel java.lang.NullPointerException at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:534) at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:560) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1172) at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:396) at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:684) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at com.broadsoft.xsp.platform.InstrumentedExecutor$Tracker.run(InstrumentedExecutor.java:413) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
--- Comment #1 from Remy Maucherat <[hidden email]> --- Ok, I can see the error check could still be a bit shaky, a loop could possibly be used in clearLastError (documentation of the OpenSSL API is here: https://www.openssl.org/docs/manmaster/man3/ERR_get_error.html ). Not sure why clearLastError is called in unwrap however, maybe Mark can comment on that one. The TODO is accurate: ideally every SSL call should check the error. BTW, I doubt this reproduces anything, ab works just fine, so curl isn't going to be doing anything special beyond that. The NPE in SecureNioChannel.close is possible, it would be a cosmetic error but the code could be more defensive. -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
Remy Maucherat <[hidden email]> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO --- Comment #2 from Remy Maucherat <[hidden email]> --- I cannot reproduce the issue, please test with the most recent 9.0 release. Since this better matches the OpenSSL documentation, I am ok conceptually to modify clearLastError to do: while (SSL.getLastErrorNumber() != SSL.SSL_ERROR_NONE); -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
--- Comment #3 from [hidden email] --- I have reproduced the problem with tomcat version 9.0.43: Feb 11, 2021 11:49:43 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: Loaded Apache Tomcat Native library [1.2.26] using APR version [1.4.8]. Feb 11, 2021 11:49:43 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Feb 11, 2021 11:49:43 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] Feb 11, 2021 11:49:43 AM org.apache.catalina.core.AprLifecycleListener initializeSSL INFO: OpenSSL successfully initialized [OpenSSL 1.0.2u 20 Dec 2019] .... Feb 11, 2021 12:05:01 PM org.apache.tomcat.util.net.openssl.OpenSSLEngine checkLastError FINE: OpenSSL error: [336462231] message: [error:140E0197:SSL routines:SSL_shutdown:shutdown while in init] Feb 11, 2021 12:05:01 PM org.apache.coyote.http11.Http11Processor service FINE: Error parsing HTTP request header javax.net.ssl.SSLException: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init at org.apache.tomcat.util.net.openssl.OpenSSLEngine.checkLastError(OpenSSLEngine.java:951) at org.apache.tomcat.util.net.openssl.OpenSSLEngine.pendingReadableBytesInSSL(OpenSSLEngine.java:636) at org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:558) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:629) at org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:637) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1320) at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1232) at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:791) at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:359) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:261) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:887) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1684) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at com.broadsoft.xsp.platform.InstrumentedExecutor$Tracker.run(InstrumentedExecutor.java:413) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
--- Comment #4 from Mark Thomas <[hidden email]> --- I can't recreate this using the provided test case, Tomcat Native 1.2.26 and the latest OpenSSL 1.1.1 source code. I'll note that Tomcat 1.2.26 won't compile with OpenSSL 1.0.2 so I do wonder exactly what versions were in used in the tests described here. I'll also note that OpenSSL 1.0.2 is no longer supported. Please retest with the latest OpenSSL 1.1.1 release. -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
--- Comment #5 from Remy Maucherat <[hidden email]> --- (In reply to Mark Thomas from comment #4) > I can't recreate this using the provided test case, Tomcat Native 1.2.26 and > the latest OpenSSL 1.1.1 source code. > > I'll note that Tomcat 1.2.26 won't compile with OpenSSL 1.0.2 so I do wonder > exactly what versions were in used in the tests described here. > > I'll also note that OpenSSL 1.0.2 is no longer supported. > > Please retest with the latest OpenSSL 1.1.1 release. Should I still replace clearLastError with this ? private static void clearLastError() { while (SSL.getLastErrorNumber() != SSL.SSL_ERROR_NONE); } This does not seem unreasonable given the docs ... -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
--- Comment #6 from Mark Thomas <[hidden email]> --- Yes, the OP is correct that the clearLastError call in unwrap is there is part of this fix: https://github.com/apache/tomcat/commit/ebab25 I agree that using a loop looks to be the right approach here. -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
Remy Maucherat <[hidden email]> changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|NEEDINFO |RESOLVED --- Comment #7 from Remy Maucherat <[hidden email]> --- Ok, so clearLastError will now align with the OpenSSL documentation using the loop. The change will be in 10.0.3, 9.0.44 and 8.5.64 [along with the NPE fix]. We could not reproduce, so you will have to verify this is fixed. -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=65131
--- Comment #8 from [hidden email] --- Thanks, Remy. I'll update this bug report when I get the chance to test the latest tomcat version. -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
Free forum by Nabble | Edit this page |