Websocket semaphore lock on close() blocks all tomcat threads

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Websocket semaphore lock on close() blocks all tomcat threads

Sridhar Rao

We notice a behavior with tomcat where it becomes unresponsive and all http threads go into a timed wait state and the node becomes unresponsive.

Tomcat Version: 8.5.47

Please see the attached sanitized thread dump, which shows all 50 tomcat threads and websocket background thread blocked with a semaphore wait. No thread is currently holding semaphore. It appears that a thread may have acquired the semaphore and failed to release it.

If we do a static code analysis of WSRemoteEndpointImplBase.java, it seems that a semaphore is acquired for 20 seconds in the "if" block, and could potentially not release, if there is an exception in writeMessagePart(mp) method. We do see some broken pipe exceptions in our logs from writeMessagePart(mp) method, which seem to be caused by n/w issues. Please see attachment for the logs.

Please see relevant code below

private static final long DEFAULT_BLOCKING_SEND_TIMEOUT = 20 * 1000;

private final Semaphore messagePartInProgress = new Semaphore(1);

Semaphore Usage

long timeout = timeoutExpiry - System.currentTimeMillis();
try {
if (!messagePartInProgress.tryAcquire(timeout, TimeUnit.MILLISECONDS)) {
String msg = sm.getString("wsRemoteEndpoint.acquireTimeout");
wsSession.doClose(new CloseReason(CloseCodes.GOING_AWAY, msg),
new CloseReason(CloseCodes.CLOSED_ABNORMALLY, msg));
throw new SocketTimeoutException(msg);
}
} catch (InterruptedException e) {
String msg = sm.getString("wsRemoteEndpoint.sendInterrupt");
wsSession.doClose(new CloseReason(CloseCodes.GOING_AWAY, msg),
new CloseReason(CloseCodes.CLOSED_ABNORMALLY, msg));
throw new IOException(msg, e);
}
for (MessagePart mp : messageParts) {
writeMessagePart(mp);
if (!bsh.getSendResult().isOK()) {
messagePartInProgress.release();
Throwable t = bsh.getSendResult().getException();
wsSession.doClose(new CloseReason(CloseCodes.GOING_AWAY, t.getMessage()),
new CloseReason(CloseCodes.CLOSED_ABNORMALLY, t.getMessage()));
throw new IOException (t);
}
// The BlockingSendHandler doesn't call end message so update the
// flags.
fragmented = nextFragmented;
text = nextText;
}
if (payload != null) {
payload.clear();
}
endMessage(null, null);
}
Could this be a tomcat defect? 






---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

sanitized_thread_dump.txt (307K) Download Attachment
brokenPipeExceptions.txt (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Websocket semaphore lock on close() blocks all tomcat threads

Mark Thomas-2
On 14/07/2020 20:57, Sridhar Rao wrote:
>
> We notice a behavior with tomcat where it becomes unresponsive and all
> http threads go into a timed wait state and the node becomes unresponsive.
>
> Tomcat Version: 8.5.47

<snip/>

> Could this be a tomcat defect?

Possibly.

Let me take a look. I don't recall anything along these lines being
fixed since 8.5.47 but there might be.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Websocket semaphore lock on close() blocks all tomcat threads

Mark Thomas-2
On 14/07/2020 21:08, Mark Thomas wrote:

> On 14/07/2020 20:57, Sridhar Rao wrote:
>>
>> We notice a behavior with tomcat where it becomes unresponsive and all
>> http threads go into a timed wait state and the node becomes unresponsive.
>>
>> Tomcat Version: 8.5.47
>
> <snip/>
>
>> Could this be a tomcat defect?
>
> Possibly.
>
> Let me take a look. I don't recall anything along these lines being
> fixed since 8.5.47 but there might be.

Yep. Looks like an issue in all current versions. Fixed for 10.0.x with
back-ports to follow shortly.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]