[Bug 64771] New: Windows CPU processor getting locked by a thread reading request body from https connection

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 64771] New: Windows CPU processor getting locked by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

            Bug ID: 64771
           Summary: Windows CPU processor getting locked by a thread
                    reading request body from https connection
           Product: Tomcat 9
           Version: 9.0.37
          Hardware: PC
            Status: NEW
          Severity: major
          Priority: P2
         Component: Util
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: -----

Created attachment 37472
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=37472&action=edit
Java Thread Dump

The problem started with windows CPU always keep at 25%, but it should be
around 2-7% is there is no request.

I use windows ProcessExplorer identified that one thread is reading request
body from an https connection but there is no request at that time based on the
audit logs, and the thread has been running more than 2 hours.

I use windows ProcessExplorer and jdk jstack identified the thread "0xd14" is
always keep running even there is no request to Tomcat.


"boundedElastic-90" #434 daemon prio=5 os_prio=0 tid=0x0000000021055000
nid=0xd14 runnable [0x000000004383d000]
   java.lang.Thread.State: RUNNABLE
        at
org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:603)
        - locked <0x0000000608ad2bf0> (a
org.apache.tomcat.util.net.openssl.OpenSSLEngine)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)


"I/O dispatcher 3" #30 prio=5 os_prio=0 tid=0x00000000208d2800 nid=0x17e8
waiting for monitor entry [0x0000000024ace000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at
org.apache.tomcat.util.net.openssl.OpenSSLEngine.wrap(OpenSSLEngine.java:391)
        - waiting to lock <0x0000000608ad2bf0> (a
org.apache.tomcat.util.net.openssl.OpenSSLEngine)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
        at
org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:802)



I try to understand the source "OpenSSLEngine" based on the thread logs, it
seems that the reading request body is not yet fully completed but the request
connection is closed by the caller.

I need help to have a fix to prevent this kind of problem happen in Tomcat to
prevent such problem happen in production.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor getting locked by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

Jerry Hou <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 OS|                            |All

--- Comment #1 from Jerry Hou <[hidden email]> ---
It is starting from     at
com.ncs.secureconnect.webflux.common.gateway.context.DefaultContext.bufferBody(DefaultContext.java:173)

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

Jerry Hou <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Windows CPU processor       |Windows CPU processor
                   |getting locked by a thread  |always running by a thread
                   |reading request body from   |reading request body from
                   |https connection            |https connection

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #2 from mgrigorov <[hidden email]> ---
Could you please provide the full stack traces for these two threads ?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #3 from Jerry Hou <[hidden email]> ---
"boundedElastic-90" #434 daemon prio=5 os_prio=0 tid=0x0000000021055000
nid=0xd14 runnable [0x000000004383d000]
   java.lang.Thread.State: RUNNABLE
        at
org.apache.tomcat.util.net.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:603)
        - locked <0x0000000608ad2bf0> (a
org.apache.tomcat.util.net.openssl.OpenSSLEngine)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at
org.apache.tomcat.util.net.SecureNioChannel.read(SecureNioChannel.java:617)
        at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1228)
        at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1140)
        at
org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:780)
        at
org.apache.coyote.http11.Http11InputBuffer.available(Http11InputBuffer.java:668)
        at
org.apache.coyote.http11.Http11Processor.available(Http11Processor.java:1185)
        at
org.apache.coyote.AbstractProcessor.isReadyForRead(AbstractProcessor.java:825)
        at
org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:574)
        at org.apache.coyote.Request.action(Request.java:432)
        at
org.apache.catalina.connector.InputBuffer.isReady(InputBuffer.java:305)
        at
org.apache.catalina.connector.CoyoteInputStream.isReady(CoyoteInputStream.java:201)
        at
org.springframework.http.server.reactive.ServletServerHttpRequest$RequestBodyPublisher.checkOnDataAvailable(ServletServerHttpRequest.java:290)
        at
org.springframework.http.server.reactive.AbstractListenerReadPublisher.changeToDemandState(AbstractListenerReadPublisher.java:222)
        at
org.springframework.http.server.reactive.AbstractListenerReadPublisher.access$1000(AbstractListenerReadPublisher.java:48)
        at
org.springframework.http.server.reactive.AbstractListenerReadPublisher$State$2.request(AbstractListenerReadPublisher.java:333)
        at
org.springframework.http.server.reactive.AbstractListenerReadPublisher$ReadSubscription.request(AbstractListenerReadPublisher.java:260)
        at
reactor.core.publisher.MonoReduce$ReduceSubscriber.onSubscribe(MonoReduce.java:78)
        at
org.springframework.http.server.reactive.AbstractListenerReadPublisher$State$1.subscribe(AbstractListenerReadPublisher.java:301)
        at
org.springframework.http.server.reactive.AbstractListenerReadPublisher.subscribe(AbstractListenerReadPublisher.java:105)
        at reactor.core.publisher.FluxSource.subscribe(FluxSource.java:65)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at reactor.core.publisher.Mono.subscribeWith(Mono.java:4324)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4184)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4120)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4067)
        at
com.bac.bacconnect.webflux.common.gateway.context.DefaultContext.bufferBody(DefaultContext.java:173)
        at
com.bac.bacconnect.commons.filters.SecurityFilter.e(SecurityFilter.java:205)
        at
com.bac.bacconnect.commons.filters.SecurityFilter.doFilterInternally(SecurityFilter.java:132)
        at
com.bac.bacconnect.commons.filters.SecurityFilter.doFilter(SecurityFilter.java:106)
        at
com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:42)
        at com.bac.bacconnect.igw.webflux.controller.Api.a(Api.java:102)
        at
com.bac.bacconnect.igw.webflux.controller.Api$$Lambda$417/1031507663.accept(Unknown
Source)
        at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
        at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
        at
reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
        at
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
        at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
        at
reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296)
        at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
        at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
        at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
        at
reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:247)
        at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:329)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
        at
reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132)
        at
reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:318)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145)
        at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153)
        at
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
        at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
        at
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851)
        at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173)
        at
reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2344)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:132)
        at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162)
        at
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152)
        at
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2026)
        at
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:145)
        at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
        at
reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
        at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:141)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
        at
reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
        at
reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:92)
        at
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851)
        at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
        at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:144)
        at
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
        at
reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
        at
reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:845)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
        at
reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:76)
        at
reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:107)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmitScalar(FluxFlatMap.java:480)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:413)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203)
        at
reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:267)
        at
reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:137)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:171)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:172)
        at
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
        at
reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160)
        at
reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onComplete(FluxPeekFuseable.java:931)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:78)
        at
reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2346)
        at
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2152)
        at
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2026)
        at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75)
        at
reactor.core.publisher.MonoNext$NextSubscriber.onComplete(MonoNext.java:96)
        at
reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:838)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:600)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:580)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:457)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onComplete(FluxPeekFuseable.java:270)
        at
reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:289)
        at
reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:137)
        at
reactor.core.publisher.FluxFlatMap$FlatMapMain.onSubscribe(FluxFlatMap.java:363)
        at
reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:171)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161)
        at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
        at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)
        at
reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onComplete(FluxDefaultIfEmpty.java:100)
        at
reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
        at
reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
        at
reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160)
        at
reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:262)
        at
reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1783)
        at
reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:320)
        at
reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:337)
        at
reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
        at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)
        at
reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:249)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)




"I/O dispatcher 3" #30 prio=5 os_prio=0 tid=0x00000000208d2800 nid=0x17e8
waiting for monitor entry [0x0000000024ace000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at
org.apache.tomcat.util.net.openssl.OpenSSLEngine.wrap(OpenSSLEngine.java:391)
        - waiting to lock <0x0000000608ad2bf0> (a
org.apache.tomcat.util.net.openssl.OpenSSLEngine)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
        at
org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:802)
        at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1268)
        at
org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:764)
        at
org.apache.tomcat.util.net.SocketWrapperBase.flushNonBlocking(SocketWrapperBase.java:735)
        at
org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:709)
        at
org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:564)
        at
org.apache.coyote.http11.filters.IdentityOutputFilter.flush(IdentityOutputFilter.java:117)
        at
org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:217)
        at
org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1179)
        at
org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:399)
        at org.apache.coyote.Response.action(Response.java:209)
        at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305)
        at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:272)
        at
org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
        at
org.springframework.http.server.reactive.ServletServerHttpResponse.flush(ServletServerHttpResponse.java:198)
        at
org.springframework.http.server.reactive.ServletServerHttpResponse.access$500(ServletServerHttpResponse.java:50)
        at
org.springframework.http.server.reactive.ServletServerHttpResponse$ResponseBodyFlushProcessor.flush(ServletServerHttpResponse.java:316)
        at
org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$3.writeComplete(AbstractListenerWriteFlushProcessor.java:288)
        at
org.springframework.http.server.reactive.AbstractListenerWriteFlushProcessor$State$WriteResultSubscriber.onComplete(AbstractListenerWriteFlushProcessor.java:434)
        at
org.springframework.http.server.reactive.WriteResultPublisher$State.publishComplete(WriteResultPublisher.java:256)
        at
org.springframework.http.server.reactive.WriteResultPublisher.publishComplete(WriteResultPublisher.java:84)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor.changeStateToComplete(AbstractListenerWriteProcessor.java:280)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor.access$300(AbstractListenerWriteProcessor.java:46)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor$State$3.onWritePossible(AbstractListenerWriteProcessor.java:368)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor.onWritePossible(AbstractListenerWriteProcessor.java:153)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor.writeIfPossible(AbstractListenerWriteProcessor.java:293)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor.changeStateToReceived(AbstractListenerWriteProcessor.java:272)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor.access$400(AbstractListenerWriteProcessor.java:46)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor$State$2.onComplete(AbstractListenerWriteProcessor.java:359)
        at
org.springframework.http.server.reactive.AbstractListenerWriteProcessor.onComplete(AbstractListenerWriteProcessor.java:141)
        at
org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onComplete(ChannelSendOperator.java:231)
        at
reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onComplete(FluxContextStart.java:115)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:359)
        at
reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:211)
        at
reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onSubscribe(FluxTake.java:370)
        at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70)
        at
reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
        at
org.springframework.http.server.reactive.ChannelSendOperator.subscribe(ChannelSendOperator.java:77)
        at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
        at
reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:199)
        at
reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
        at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
        at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at
reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
        at com.bac.bacconnect.igw.webflux.controller.Api.a(Api.java:113)
        at
com.bac.bacconnect.igw.webflux.controller.Api$$Lambda$424/723410695.handle(Unknown
Source)
        at
com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:36)
        at
com.bac.bacconnect.commons.common.filter.FilterChain.a(FilterChain.java:48)
        at
com.bac.bacconnect.commons.common.filter.FilterChain$$Lambda$429/662427353.handle(Unknown
Source)
        at
com.bac.bacconnect.igw.services.proxy.DefaultProxyService.decrypt(DefaultProxyService.java:48)
        at
com.bac.bacconnect.commons.filters.DecryptionFilter.doFilter(DecryptionFilter.java:31)
        at
com.bac.bacconnect.commons.common.filter.FilterChain.doFilter(FilterChain.java:42)
        at
com.bac.bacconnect.commons.common.filter.FilterChain.a(FilterChain.java:48)
        at
com.bac.bacconnect.commons.common.filter.FilterChain$$Lambda$429/662427353.handle(Unknown
Source)
        at
com.bac.bacconnect.commons.services.common.agent.async.AsyncClientAgent.a(AsyncClientAgent.java:220)
        at
com.bac.bacconnect.commons.services.common.agent.async.AsyncClientAgent$$Lambda$441/412883409.accept(Unknown
Source)
        at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
        at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
        at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
        at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
        at
com.bac.bacconnect.commons.services.common.agent.async.d.a(AsyncResponseCallback.java:118)
        at
com.bac.bacconnect.commons.services.common.agent.async.d.completed(AsyncResponseCallback.java:24)
        at
org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
        at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
        at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
        at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
        at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
        at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
        at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
        at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
        at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.lang.Thread.run(Thread.java:748)

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

Mark Thomas <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO

--- Comment #4 from Mark Thomas <[hidden email]> ---
There is insufficient information in the stack trace to identify a bug in
Tomcat.

I suspect that the application is retaining references to request/response
objects beyond the completion of the response.

I'd recommend re-testing without the Tomcat Native library. If you can
reproduce the issue with a pure Java stack it will be easier to debug.

What we really need is a *minimal* test case that reproduces this issue on a
clean install of the latest 9.0.x release.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #5 from Mark Thomas <[hidden email]> ---
Two months have passed and we still don't have a minimal WAR that reproduces
this issue. I'll leave this open another month but without a test case to
investigate this will get closed.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

Mark Thomas <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WORKSFORME
             Status|NEEDINFO                    |RESOLVED

--- Comment #6 from Mark Thomas <[hidden email]> ---
Resolving as per previous comment.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #7 from [hidden email] ---
I believe I have hit this same issue.

I reported a bug against Spring as
https://github.com/spring-projects/spring-framework/issues/26407

I have created a standalone application that can be used to reproduce the
problem.  The app is linked from the description of the spring issue.

There is definitely a case where OpenSSLEngine.unwrap gets into an infinite
loop.  It would be great if someone familiar with OpenSSLEngine can take a
look.  It is not clear if the problem is in tomcat or spring.

Can this tomcat issue be reopened?  Or should I file a new issue?

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #8 from Remy Maucherat <[hidden email]> ---
That doesn't look good, it seems the state of the destination buffer is
changing concurrently (suddenly, no remaining bytes although the capacity is
still there), which then breaks the algorithm. Maybe this loop should be safer,
but it probably means there's missing sync somewhere else [given the report the
threading could be funky, async does not allow to do *anything*]. It could be a
good idea to add a safety check in unwrap, but then the data read is already
corrupted and that request is not going to end well.

--- a/java/org/apache/tomcat/util/net/openssl/OpenSSLEngine.java
+++ b/java/org/apache/tomcat/util/net/openssl/OpenSSLEngine.java
@@ -600,6 +600,10 @@
             }
             if (capacity == 0) {
                 break;
+            } else if (idx == endOffset) {
+                // Avoid loop which could be caused by an unexpected state
+                // change of the destination buffers and report an error
+                throw new IllegalArgumentException("BAAAAAAAD");
             } else if (pendingApp == 0) {
                 pendingApp = pendingReadableBytesInSSL();
             }

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

Mark Thomas <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
         Resolution|WORKSFORME                  |---

--- Comment #9 from Mark Thomas <[hidden email]> ---
RE-opening this issue while I look at the test case...

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #10 from Remy Maucherat <[hidden email]> ---
(In reply to Mark Thomas from comment #9)
> RE-opening this issue while I look at the test case...

The use of non container threads doing ??? seems to be the root cause of the
problems here. Also the behavior of available() could be a contributor, and you
did change that for the better very recently. So you might not be able to
reproduce.

OTOH, I would still like to add a ISE exit for the loop in
OpenSSLEngine.unwrap, in case the state becomes inconsistent.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #11 from Mark Thomas <[hidden email]> ---
The reproducer provided in the Spring issue uses Tomcat 9.0.41 and that has the
available() fixes.

Bug 65001 might be a factor and that is a post 9.0.41 fix but since I can't
reproduce the issue, I can't test that. I'll point the OP of the Spring issue
to the snapshots to see if that helps.

My current thoughts are this is not a Tomcat issue as per my comment on the
Spring issue.

I agree with your ISE suggestion to catch his state. Async code is easy to get
wrong and we should be defensive where practical.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

Mark Thomas <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |NEEDINFO

--- Comment #12 from Mark Thomas <[hidden email]> ---
Rémy has added the defensive code.

I'm moving this to NEEDINFO while the Spring team continue to investigate.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 64771] Windows CPU processor always running by a thread reading request body from https connection

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=64771

--- Comment #13 from Mark Thomas <[hidden email]> ---
As per the update to the spring issue
(https://github.com/spring-projects/spring-framework/issues/26407) 9.0.x now
contains a fix that may help with this. Currently waiting for user feedback.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]