native connector, server problems with "No data received", what could be causing it?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

native connector, server problems with "No data received", what could be causing it?

Mladen Adamović
Hi,

We have experienced a large number of HTTP requests with the problem "No
data received".
this is the example using curl:
--2020-12-15 15:38:13--
https://www.numbeo.com/admin/DownloadLastDbBackupServlet?numbeoPass=xxx
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.numbeo.com (www.numbeo.com)... 209.126.119.66
Connecting to www.numbeo.com (www.numbeo.com)|209.126.119.66|:443...
connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2020-12-15 15:38:15--  (try: 2)
https://www.numbeo.com/admin/DownloadLastDbBackupServlet?numbeoPass=xxx
Connecting to www.numbeo.com (www.numbeo.com)|209.126.119.66|:443...
connected.
HTTP request sent, awaiting response... No data received.
Retrying.


We have a self-monitoring script which runs on server and when the server
is not working properly it does a log save and the service restart.

We are using currently Apache 8.5.5 and the Connector is set-up as follows
(using Tomcat native):
 <Connector port="8181"
protocol="org.apache.coyote.http11.Http11AprProtocol"
              SSLCertificateFile="/etc/letsencrypt/live/numbeo.com/cert.pem"
              SSLCertificateKeyFile="/etc/letsencrypt/live/
numbeo.com/privkey.pem"
              SSLCertificateChainFile="/etc/letsencrypt/live/
numbeo.com/chain.pem"
              SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
              connectionTimeout="20000" acceptCount="30000"
              acceptorThreadCount="4"
              compression="on" maxConnections="50000" maxThreads="500"

 compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,image/svg+xml,image/svg,image/png,image/jpeg"

              useSendfile="false"
              maxHttpHeaderSize="16392" SSLEnabled="true"
              enableLookups="false"
              scheme="https" secure="true"   clientAuth="false"
 useBodyEncodingForURI="true"
              URIEncoding="UTF-8"
              />


*What would be the next steps how to identify the problem and perhaps solve
it?*
If we should migrate to the new version, should we try first the last 8.5
version or the last 9 Tomcat version?

Our JSTACK when the last problem occurred is as follows:

root@condor1796 ~ # cat jstack.log
Attaching to process ID 27753, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.101-b13
Deadlock Detection:

No deadlocks found.

Thread 1237: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 - common.MemoryCleanerAndLogger.run() @bci=3, line=20 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 516: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 515: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 514: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 512: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 511: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 510: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 509: (state = IN_NATIVE)
 - org.apache.tomcat.jni.SSLSocket.handshake(long) @bci=0 (Compiled frame;
information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=119, line=726 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
@bci=102, line=2204 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 508: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 507: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 456: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 455: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 450: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 412: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 332: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 300: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32691: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32690: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32621: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32608: (state = IN_NATIVE)
 - org.apache.tomcat.jni.SSLSocket.handshake(long) @bci=0 (Compiled frame;
information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=119, line=726 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
@bci=102, line=2204 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32607: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32606: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32581: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32488: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32487: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32310: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32309: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32308: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32307: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32306: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32305: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32304: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32303: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32247: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 32056: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - sun.net.www.http.KeepAliveStreamCleaner.run() @bci=30, line=101
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 30294: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 30293: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 30292: (state = IN_NATIVE)
 - org.apache.tomcat.jni.Socket.sendb(long, java.nio.ByteBuffer, int, int)
@bci=0 (Compiled frame; information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWriteInternal()
@bci=89, line=2517 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(boolean)
@bci=85, line=2464 (Compiled frame)
 - org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(byte[], int,
int) @bci=47, line=378 (Compiled frame)
 - org.apache.tomcat.util.net.SocketWrapperBase.write(boolean, byte[], int,
int) @bci=19, line=347 (Compiled frame)
 -
org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk)
@bci=34, line=561 (Compiled frame)
 -
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(org.apache.tomcat.util.buf.ByteChunk)
@bci=90, line=112 (Compiled frame)
 -
org.apache.coyote.http11.filters.GzipOutputFilter$FakeOutputStream.write(byte[],
int, int) @bci=21, line=168 (Compiled frame)
 - java.util.zip.DeflaterOutputStream.deflate() @bci=32, line=253 (Compiled
frame)
 - java.util.zip.DeflaterOutputStream.write(byte[], int, int) @bci=81,
line=211 (Compiled frame)
 - java.util.zip.GZIPOutputStream.write(byte[], int, int) @bci=4, line=145
(Compiled frame)
 -
org.apache.coyote.http11.filters.GzipOutputFilter.doWrite(org.apache.tomcat.util.buf.ByteChunk)
@bci=39, line=70 (Compiled frame)
 -
org.apache.coyote.http11.Http11OutputBuffer.doWrite(org.apache.tomcat.util.buf.ByteChunk)
@bci=50, line=206 (Compiled frame)
 - org.apache.coyote.Response.doWrite(org.apache.tomcat.util.buf.ByteChunk)
@bci=5, line=499 (Compiled frame)
 - org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int,
int) @bci=38, line=375 (Compiled frame)
 - org.apache.tomcat.util.buf.ByteChunk.flushBuffer() @bci=71, line=420
(Compiled frame)
 - org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int) @bci=152,
line=345 (Compiled frame)
 - org.apache.catalina.connector.OutputBuffer.writeBytes(byte[], int, int)
@bci=15, line=405 (Compiled frame)
 - org.apache.catalina.connector.OutputBuffer.write(byte[], int, int)
@bci=12, line=393 (Compiled frame)
 - org.apache.catalina.connector.CoyoteOutputStream.write(byte[], int, int)
@bci=13, line=96 (Compiled frame)
 -
org.apache.catalina.servlets.DefaultServlet.copyRange(java.io.InputStream,
javax.servlet.ServletOutputStream) @bci=38, line=2281 (Compiled frame)
 -
org.apache.catalina.servlets.DefaultServlet.copy(org.apache.catalina.WebResource,
java.io.InputStream, javax.servlet.ServletOutputStream) @bci=21, line=2124
(Compiled frame)
 -
org.apache.catalina.servlets.DefaultServlet.serveResource(javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse, boolean, java.lang.String)
@bci=1277, line=1006 (Compiled frame)
 -
org.apache.catalina.servlets.DefaultServlet.doGet(javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=8, line=425 (Compiled frame)
 -
javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=35, line=622 (Compiled frame)
 - javax.servlet.http.HttpServlet.service(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=30, line=729 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=304, line=230 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=165 (Compiled frame)
 -
org.apache.tomcat.websocket.server.WsFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=21, line=52
(Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=135, line=192 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=165 (Compiled frame)
 -
org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=126,
line=120 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=135, line=192 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=165 (Compiled frame)
 - common.ExpiresFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=90, line=44
(Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=135, line=192 (Compiled frame)
 -
org.apache.catalina.core.ApplicationFilterChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=101, line=165 (Compiled frame)
 -
org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=688, line=198 (Compiled frame)
 -
org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=166, line=108 (Compiled frame)
 -
org.apache.catalina.authenticator.AuthenticatorBase.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=275, line=472 (Compiled frame)
 -
org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=135, line=140 (Compiled frame)
 -
org.apache.catalina.valves.ErrorReportValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=6, line=79 (Compiled frame)
 -
org.apache.catalina.valves.AbstractAccessLogValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=6, line=620 (Compiled frame)
 -
org.apache.catalina.core.StandardEngineValve.invoke(org.apache.catalina.connector.Request,
org.apache.catalina.connector.Response) @bci=71, line=87 (Compiled frame)
 -
org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request,
org.apache.coyote.Response) @bci=199, line=349 (Compiled frame)
 -
org.apache.coyote.http11.Http11Processor.service(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=806, line=784 (Compiled frame)
 -
org.apache.coyote.AbstractProcessorLight.process(org.apache.tomcat.util.net.SocketWrapperBase,
org.apache.tomcat.util.net.SocketEvent) @bci=106, line=66 (Compiled frame)
 -
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(org.apache.tomcat.util.net.SocketWrapperBase,
org.apache.tomcat.util.net.SocketEvent) @bci=295, line=802 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun() @bci=15,
line=2241 (Compiled frame)
 - org.apache.tomcat.util.net.SocketProcessorBase.run() @bci=21, line=49
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 30291: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 30290: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 30289: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27951: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27950: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27929: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27927: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled
frame)
 - com.mysql.jdbc.AbandonedConnectionCleanupThread.run() @bci=23, line=41
(Compiled frame)


Thread 27919: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27918: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27915: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27914: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27913: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27910: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27909: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27908: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27904: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27903: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27889: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=2039 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103
(Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31
(Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27886: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 - org.apache.coyote.AbstractProtocol$AsyncTimeout.run() @bci=10, line=1137
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27885: (state = IN_NATIVE)
 - sun.nio.ch.ServerSocketChannelImpl.accept0(java.io.FileDescriptor,
java.io.FileDescriptor, java.net.InetSocketAddress[]) @bci=0 (Compiled
frame; information may be imprecise)
 - sun.nio.ch.ServerSocketChannelImpl.accept(java.io.FileDescriptor,
java.io.FileDescriptor, java.net.InetSocketAddress[]) @bci=4, line=422
(Compiled frame)
 - sun.nio.ch.ServerSocketChannelImpl.accept() @bci=130, line=250 (Compiled
frame)
 - org.apache.tomcat.util.net.NioEndpoint$Acceptor.run() @bci=88, line=456
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27883: (state = BLOCKED)
 - sun.nio.ch.ServerSocketChannelImpl.accept() @bci=6, line=233 (Compiled
frame)
 - org.apache.tomcat.util.net.NioEndpoint$Acceptor.run() @bci=88, line=456
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27882: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
(Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.tomcat.util.net.NioEndpoint$Poller.run() @bci=55, line=790
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27881: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
(Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.tomcat.util.net.NioEndpoint$Poller.run() @bci=55, line=790
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27880: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 -
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run()
@bci=46, line=1339 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27877: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27876: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27875: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27874: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27873: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27872: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27871: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27870: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27869: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27868: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27867: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27866: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27865: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27864: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27863: (state = IN_NATIVE)
 - org.apache.tomcat.jni.SSLSocket.handshake(long) @bci=0 (Compiled frame;
information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=119, line=726 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
@bci=102, line=2204 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27862: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27861: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27860: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27859: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27858: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27857: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27856: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27855: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27854: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27853: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27852: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27851: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27850: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27849: (state = IN_NATIVE)
 - org.apache.tomcat.jni.SSLSocket.handshake(long) @bci=0 (Compiled frame;
information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=119, line=726 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
@bci=102, line=2204 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27848: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27847: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27846: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27845: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27844: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27843: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27842: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27841: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27840: (state = IN_NATIVE)
 - org.apache.tomcat.jni.SSLSocket.handshake(long) @bci=0 (Compiled frame;
information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=119, line=726 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
@bci=102, line=2204 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27839: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27838: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27837: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27836: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27835: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27834: (state = IN_NATIVE)
 - org.apache.tomcat.jni.SSLSocket.handshake(long) @bci=0 (Compiled frame;
information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=119, line=726 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
@bci=102, line=2204 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27833: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27832: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27831: (state = IN_NATIVE)
 - org.apache.tomcat.jni.SSLSocket.handshake(long) @bci=0 (Compiled frame;
information may be imprecise)
 -
org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(org.apache.tomcat.util.net.SocketWrapperBase)
@bci=119, line=726 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
@bci=102, line=2204 (Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27830: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27829: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27828: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27827: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27826: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27825: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27824: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27823: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27822: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27821: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27820: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27819: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27818: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27817: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27816: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27815: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27814: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27812: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27811: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27810: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27809: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27808: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27807: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27806: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27804: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 - org.apache.coyote.AbstractProtocol$AsyncTimeout.run() @bci=10, line=1137
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27803: (state = IN_NATIVE)
 - sun.nio.ch.ServerSocketChannelImpl.accept0(java.io.FileDescriptor,
java.io.FileDescriptor, java.net.InetSocketAddress[]) @bci=0 (Compiled
frame; information may be imprecise)
 - sun.nio.ch.ServerSocketChannelImpl.accept(java.io.FileDescriptor,
java.io.FileDescriptor, java.net.InetSocketAddress[]) @bci=4, line=422
(Compiled frame)
 - sun.nio.ch.ServerSocketChannelImpl.accept() @bci=130, line=250 (Compiled
frame)
 - org.apache.tomcat.util.net.NioEndpoint$Acceptor.run() @bci=88, line=456
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27802: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27801: (state = BLOCKED)
 - sun.nio.ch.ServerSocketChannelImpl.accept() @bci=6, line=233 (Compiled
frame)
 - org.apache.tomcat.util.net.NioEndpoint$Acceptor.run() @bci=88, line=456
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27800: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27799: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27798: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27797: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27796: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=836 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(int)
@bci=66, line=897 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(int)
@bci=24, line=1222 (Compiled frame)
 - java.util.concurrent.locks.ReentrantLock.lockInterruptibly() @bci=5,
line=335 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=27, line=462 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27795: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)
 - org.apache.tomcat.util.threads.TaskQueue.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=26, line=1127 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27794: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
(Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.tomcat.util.net.NioEndpoint$Poller.run() @bci=55, line=790
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27793: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
(Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.tomcat.util.net.NioEndpoint$Poller.run() @bci=55, line=790
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27792: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 - org.apache.coyote.AbstractProtocol$AsyncTimeout.run() @bci=10, line=1137
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27791: (state = IN_NATIVE)
 - org.apache.tomcat.jni.Socket.accept(long) @bci=0 (Compiled frame;
information may be imprecise)
 - org.apache.tomcat.util.net.AprEndpoint$Acceptor.run() @bci=88, line=925
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27790: (state = IN_NATIVE)
 - org.apache.tomcat.jni.Socket.accept(long) @bci=0 (Compiled frame;
information may be imprecise)
 - org.apache.tomcat.util.net.AprEndpoint$Acceptor.run() @bci=88, line=925
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27789: (state = IN_NATIVE)
 - org.apache.tomcat.jni.Socket.accept(long) @bci=0 (Compiled frame;
information may be imprecise)
 - org.apache.tomcat.util.net.AprEndpoint$Acceptor.run() @bci=88, line=925
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27788: (state = IN_NATIVE)
 - org.apache.tomcat.jni.Socket.accept(long) @bci=0 (Compiled frame;
information may be imprecise)
 - org.apache.tomcat.util.net.AprEndpoint$Acceptor.run() @bci=88, line=925
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27787: (state = IN_NATIVE)
 - org.apache.tomcat.jni.Poll.add(long, long, int) @bci=0 (Compiled frame;
information may be imprecise)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.addToPoller(long, int)
@bci=35, line=1416 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.run() @bci=506, line=1616
(Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27786: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 -
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run()
@bci=46, line=1339 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27784: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - jobs.AlertSender.run() @bci=19, line=39 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27782: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled
frame)
 - com.mysql.jdbc.AbandonedConnectionCleanupThread.run() @bci=23, line=41
(Compiled frame)


Thread 27781: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 - livingcost.PricesAnalysisReader.run() @bci=13, line=52 (Compiled frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27780: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - wikilibrary.EmailQueueWorker.run() @bci=88, line=38 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27779: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be
imprecise)
 - wikilibrary.CurrentRankingsUpdater.run() @bci=49, line=93 (Interpreted
frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)


Thread 27776: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
(Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run()
@bci=52, line=339 (Interpreted frame)


Thread 27775: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
(Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled
frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run()
@bci=52, line=339 (Interpreted frame)


Thread 27774: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - sun.misc.GC$Daemon.run() @bci=51, line=117 (Interpreted frame)


Thread 27773: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long)
@bci=20, line=215 (Compiled frame)
 -
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.LinkedBlockingDeque.pollFirst(long,
java.util.concurrent.TimeUnit) @bci=52, line=522 (Compiled frame)
 - java.util.concurrent.LinkedBlockingDeque.poll(long,
java.util.concurrent.TimeUnit) @bci=3, line=684 (Compiled frame)
 - org.apache.juli.AsyncFileHandler$LoggerThread.run() @bci=10, line=152
(Interpreted frame)


Thread 27766: (state = BLOCKED)


Thread 27765: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled
frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209
(Compiled frame)


Thread 27764: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
imprecise)
 - java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
 - java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191
(Compiled frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153
(Compiled frame)


Thread 27754: (state = IN_NATIVE)
 - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
(Interpreted frame)
 - java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7,
line=409 (Interpreted frame)
 - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=545
(Interpreted frame)
 - java.net.ServerSocket.accept() @bci=48, line=513 (Interpreted frame)
 - org.apache.catalina.core.StandardServer.await() @bci=180, line=466
(Interpreted frame)
 - org.apache.catalina.startup.Catalina.await() @bci=4, line=743
(Interpreted frame)
 - org.apache.catalina.startup.Catalina.start() @bci=209, line=689
(Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method,
java.lang.Object, java.lang.Object[]) @bci=0 (Compiled frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=100, line=62 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=6, line=43 (Compiled frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[])
@bci=56, line=498 (Compiled frame)
 - org.apache.catalina.startup.Bootstrap.start() @bci=37, line=355
(Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=150,
line=495 (Interpreted frame)


root@condor1796 ~ #
Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Christopher Schultz-2
Mladen,

On 12/16/20 04:10, Mladen Adamović wrote:

> We have experienced a large number of HTTP requests with the problem "No
> data received".
> this is the example using curl:
> --2020-12-15 15:38:13--
> https://www.numbeo.com/admin/DownloadLastDbBackupServlet?numbeoPass=xxx
> Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
> Resolving www.numbeo.com (www.numbeo.com)... 209.126.119.66
> Connecting to www.numbeo.com (www.numbeo.com)|209.126.119.66|:443...
> connected.
> HTTP request sent, awaiting response... No data received.
> Retrying.
>
> --2020-12-15 15:38:15--  (try: 2)
> https://www.numbeo.com/admin/DownloadLastDbBackupServlet?numbeoPass=xxx
> Connecting to www.numbeo.com (www.numbeo.com)|209.126.119.66|:443...
> connected.
> HTTP request sent, awaiting response... No data received.
> Retrying.
>
>
> We have a self-monitoring script which runs on server and when the server
> is not working properly it does a log save and the service restart.

How do you detect this state? Just make a request and if you get "No
data received" from curl, you restart the server?

> We are using currently Apache 8.5.5 and the Connector is set-up as follows
> (using Tomcat native):
>   <Connector port="8181"
> protocol="org.apache.coyote.http11.Http11AprProtocol"
>                SSLCertificateFile="/etc/letsencrypt/live/numbeo.com/cert.pem"
>                SSLCertificateKeyFile="/etc/letsencrypt/live/
> numbeo.com/privkey.pem"
>                SSLCertificateChainFile="/etc/letsencrypt/live/
> numbeo.com/chain.pem"
>                SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
>                connectionTimeout="20000" acceptCount="30000"
>                acceptorThreadCount="4"
>                compression="on" maxConnections="50000" maxThreads="500"
>
>   compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,image/svg+xml,image/svg,image/png,image/jpeg"
>
>                useSendfile="false"
>                maxHttpHeaderSize="16392" SSLEnabled="true"
>                enableLookups="false"
>                scheme="https" secure="true"   clientAuth="false"
>   useBodyEncodingForURI="true"
>                URIEncoding="UTF-8"
>                />

I see you are using Let's Encrypt. How are you managing the rotating of
the keys and certificates?

> *What would be the next steps how to identify the problem and perhaps
> solve it?*
What have you done so far?

> If we should migrate to the new version, should we try first the last
> 8.5 version or the last 9 Tomcat version?
I think in general you should upgrade to the latest version which makes
sense for you. That's probably 8.5.61 for now, but maybe you should make
a plan to migrate to Tomcat 9.x over the next few months or years.

> Our JSTACK when the last problem occurred is as follows:
>
> root@condor1796 ~ # cat jstack.log
 > [...]

I don't see anything that sticks out in your thread dump.

> Attaching to process ID 27753, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 25.101-b13
> Deadlock Detection:

That JVM seems fairly old, too. Consider upgrading to latest Java 8 VM
(or beyond, if appropriate).

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Mladen Adamović
On Wed, Dec 16, 2020 at 3:27 PM Christopher Schultz <
[hidden email]> wrote:

> > We have a self-monitoring script which runs on server and when the server
> > is not working properly it does a log save and the service restart.
>
> How do you detect this state? Just make a request and if you get "No
> data received" from curl, you restart the server?
>

If there is an error code or the specific text doesn't appear on the
response we monitor the state and do /etc/init.d/tomcat restart.
The full script is:
#!/bin/bash
serverFailure=0
cd /root
rm /root/numbeo_test.out
#wget -t 1 -T 5 --no-proxy --no-cache --cache=off -q
'localhost:8080/cost-of-living/city_result.jsp?country=Ireland&city=Dublin'
-O /root/numbeo_test.out
#curl -L -m 2 -v  -o /root/numbeo_test.out --trace curl.log
'localhost:8008/cost-of-living/in/Dublin'
curl -L -m 2 -v --insecure -o /root/numbeo_test.out --trace curl.log '
https://localhost:8181/cost-of-living/in/Dublin'
wgetOutput=$?

grep -q "entries in the past" /root/numbeo_test.out
if [ $? != 0 ]; then
cd /root
rm /root/numbeo_test.out
sleep 10s
#wget -t 2 -T 2 --no-proxy --no-cache --cache=off -q
'localhost:8080/cost-of-living/city_result.jsp?country=Ireland&city=Dublin'
-O /root/numbeo_test.out
  #curl -L -m 2 --retry 1 -v  -o /root/numbeo_test.out --trace curl.log
'localhost:8008/cost-of-living/in/Dublin'
  curl -L -m 2 -v --insecure -o /root/numbeo_test.out --trace curl.log '
https://localhost:8181/cost-of-living/in/Dublin'
  wgetOutput=$?
grep -q "entries in the past" /root/numbeo_test.out

if [ $? != 0 ]; then
#echo 'server is down!';
ps -eo pid,comm | while read pid command
do
   if [[ "$command" = "java" ]]
       then
               echo $pid
               DATE=`date +%Y-%m-%d`
               echo ${wgetOutput} > ~/wget_${DATE}_${pid}.log
               cp /root/numbeo_test.out >
~/numbeo_test_out_${DATE}_${pid}.log
               jstack -J-d64 -F $pid > ~/jstack_${DATE}_${pid}.log
               iostat > ~/iostat_${DATE}_${pid}.log
               vmstat > ~/vmstat_${DATE}_${pid}.log
               netstat -tnp > ~/netstat_${DATE}_${pid}.log
               netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d:
-f1 | sort | uniq -c | sort -n > ~/netstat_anp_outline_${DATE}_${pid}.log
               ps aux > ~/ps_aux_${DATE}_${pid}.log
               tail -n 5000
~glassfish/apache-tomcat-8.5.5/logs/catalina.out >
~/catalina_out_${DATE}_${pid}.log
               break
   fi
done
echo 'too many server failures... going to rebootsoftly' >> ~/reboot.log ;
date | mail -s "Numbeo soft reset" [hidden email]
date >> ~/reboot.log
killall -9 java
/root/fix_letsencrypt_chmod.sh
#/etc/init.d/glassfish start
/etc/init.d/tomcat start
#reboot
fi
fi


I see you are using Let's Encrypt. How are you managing the rotating of
> the keys and certificates?
>

Crontab: 5   1  1   *   *     /root/renew_cert_numbeo.sh
root@condor1796 ~ # cat renew_cert_numbeo.sh
#!/bin/bash

mkdir -p /tmp/letsencrypt/public_html
certbot certonly -n --force-renewal --webroot --webroot-path
/tmp/letsencrypt/public_html -d numbeo.com -d www.numbeo.com \
        -d es.numbeo.com -d  pt.numbeo.com -d  fr.numbeo.com -d
ru.numbeo.com -d  ja.numbeo.com -d  de.numbeo.com -d nl.numbeo.com \
        -d it.numbeo.com -d zh.numbeo.com -d ar.numbeo.com -d
jobs.numbeo.com \
     --agree-tos --email [hidden email]

/root/fix_letsencrypt_chmod.sh
if [ $? != 0 ]; then
   date | mail -s "Lets encrypt renew certificate fails for numbeo.com"
[hidden email]
else
   /etc/init.d/tomcat restart
fi

root@condor1796 ~ # cat fix_letsencrypt_chmod.sh
#!/bin/bash
chmod o+rx /etc/letsencrypt
chmod -R o+rx /etc/letsencrypt/*

root@condor1796 ~ #



> > *What would be the next steps how to identify the problem and perhaps
> > solve it?*
> What have you done so far?
>

aaah... reading the Tomcat source to try to understand the state of Threads.

I don't see anything that sticks out in your thread dump.
>

There are several threads which are trying to get monitor in
AprEndpoint$Poller.add and no thread seems to be blocking it. Don't you
find it weird:

root@condor1796 ~ # grep Poller jstack_2020-12-16_31415.log  | grep "Apr"
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.run() @bci=1713, line=1799
(Compiled frame; information may be imprecise)


Thread 22685: (state = BLOCKED)
 - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
 -
org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.registerReadInterest()
@bci=48, line=2560 (Compiled frame)
 -
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(org.apache.tomcat.util.net.SocketWrapperBase,
org.apache.tomcat.util.net.SocketEvent) @bci=643, line=870 (Compiled frame)
 - org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun() @bci=15,
line=2241 (Compiled frame)
 - org.apache.tomcat.util.net.SocketProcessorBase.run() @bci=21, line=49
(Compiled frame)
 -
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4,
line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)

The Poller add(...) method is as follows:

        private void add(long socket, long timeout, int flags) {
            if (log.isDebugEnabled()) {
                String msg = sm.getString("endpoint.debug.pollerAdd",
                        Long.valueOf(socket), Long.valueOf(timeout),
                        Integer.valueOf(flags));
                if (log.isTraceEnabled()) {
                    log.trace(msg, new Exception());
                } else {
                    log.debug(msg);
                }
            }
            if (timeout <= 0) {
                // Always put a timeout in
                timeout = Integer.MAX_VALUE;
            }
            synchronized (this) {
                // Add socket to the list. Newly added sockets will wait
                // at most for pollTime before being polled.
                if (addList.add(socket, timeout, flags)) {
                    this.notify();
                }
            }
        }


Don't you find it weird that all threads are trying to get synchronized on
a Poller instance and no one is in this block or another synchronized
block/method?





>
> > Attaching to process ID 27753, please wait...
> > Debugger attached successfully.
> > Server compiler detected.
> > JVM version is 25.101-b13
> > Deadlock Detection:
>
> That JVM seems fairly old, too. Consider upgrading to latest Java 8 VM
> (or beyond, if appropriate).
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Christopher Schultz-2
Mladen,

On 12/16/20 10:12, Mladen Adamović wrote:

> On Wed, Dec 16, 2020 at 3:27 PM Christopher Schultz
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>      > We have a self-monitoring script which runs on server and when
>     the server
>      > is not working properly it does a log save and the service restart.
>
>     How do you detect this state? Just make a request and if you get "No
>     data received" from curl, you restart the server?
>
>
> If there is an error code or the specific text doesn't appear on the
> response we monitor the state and do /etc/init.d/tomcat restart.
> The full script is:
> #!/bin/bash
> serverFailure=0
> cd /root
> rm /root/numbeo_test.out
> #wget -t 1 -T 5 --no-proxy --no-cache --cache=off -q
> 'localhost:8080/cost-of-living/city_result.jsp?country=Ireland&city=Dublin'
> -O /root/numbeo_test.out
> #curl -L -m 2 -v  -o /root/numbeo_test.out --trace curl.log
> 'localhost:8008/cost-of-living/in/Dublin'
> curl -L -m 2 -v --insecure -o /root/numbeo_test.out --trace curl.log
> 'https://localhost:8181/cost-of-living/in/Dublin 
> <https://localhost:8181/cost-of-living/in/Dublin>'
> wgetOutput=$?
>
> grep -q "entries in the past" /root/numbeo_test.out
> if [ $? != 0 ]; then
> cd /root
> rm /root/numbeo_test.out
> sleep 10s
> #wget -t 2 -T 2 --no-proxy --no-cache --cache=off -q
> 'localhost:8080/cost-of-living/city_result.jsp?country=Ireland&city=Dublin'
> -O /root/numbeo_test.out
>    #curl -L -m 2 --retry 1 -v  -o /root/numbeo_test.out --trace curl.log
> 'localhost:8008/cost-of-living/in/Dublin'
>    curl -L -m 2 -v --insecure -o /root/numbeo_test.out --trace curl.log
> 'https://localhost:8181/cost-of-living/in/Dublin 
> <https://localhost:8181/cost-of-living/in/Dublin>'
>    wgetOutput=$?
> grep -q "entries in the past" /root/numbeo_test.out
>
> if [ $? != 0 ]; then
> #echo 'server is down!';
> ps -eo pid,comm | while read pid command
> do
>     if [[ "$command" = "java" ]]
>         then
>                 echo $pid
>                 DATE=`date +%Y-%m-%d`
>                 echo ${wgetOutput} > ~/wget_${DATE}_${pid}.log
>                 cp /root/numbeo_test.out >
> ~/numbeo_test_out_${DATE}_${pid}.log
>                 jstack -J-d64 -F $pid > ~/jstack_${DATE}_${pid}.log
>                 iostat > ~/iostat_${DATE}_${pid}.log
>                 vmstat > ~/vmstat_${DATE}_${pid}.log
>                 netstat -tnp > ~/netstat_${DATE}_${pid}.log
>                 netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut
> -d: -f1 | sort | uniq -c | sort -n >
> ~/netstat_anp_outline_${DATE}_${pid}.log
>                 ps aux > ~/ps_aux_${DATE}_${pid}.log
>                 tail -n 5000
> ~glassfish/apache-tomcat-8.5.5/logs/catalina.out >
> ~/catalina_out_${DATE}_${pid}.log
>                 break
>     fi
> done
> echo 'too many server failures... going to rebootsoftly' >> ~/reboot.log ;
> date | mail -s "Numbeo soft reset" [hidden email]
> <mailto:[hidden email]>
> date >> ~/reboot.log
> killall -9 java
> /root/fix_letsencrypt_chmod.sh
> #/etc/init.d/glassfish start
> /etc/init.d/tomcat start
> #reboot
> fi
> fi

That seems a little fragile, but it's your server so I guess you can do
what you want.

>     I see you are using Let's Encrypt. How are you managing the rotating of
>     the keys and certificates?
>
>
> Crontab: 5   1  1   *   *     /root/renew_cert_numbeo.sh
> root@condor1796 ~ # cat renew_cert_numbeo.sh
> #!/bin/bash
>
> mkdir -p /tmp/letsencrypt/public_html
> certbot certonly -n --force-renewal --webroot --webroot-path
> /tmp/letsencrypt/public_html -d numbeo.com <http://numbeo.com> -d
> www.numbeo.com <http://www.numbeo.com> \
>          -d es.numbeo.com <http://es.numbeo.com> -d pt.numbeo.com
> <http://pt.numbeo.com> -d fr.numbeo.com <http://fr.numbeo.com> -d
> ru.numbeo.com <http://ru.numbeo.com> -d ja.numbeo.com
> <http://ja.numbeo.com> -d de.numbeo.com <http://de.numbeo.com> -d
> nl.numbeo.com <http://nl.numbeo.com> \
>          -d it.numbeo.com <http://it.numbeo.com> -d zh.numbeo.com
> <http://zh.numbeo.com> -d ar.numbeo.com <http://ar.numbeo.com> -d
> jobs.numbeo.com <http://jobs.numbeo.com> \
>       --agree-tos --email [hidden email]
> <mailto:[hidden email]>
>
> /root/fix_letsencrypt_chmod.sh
> if [ $? != 0 ]; then
>     date | mail -s "Lets encrypt renew certificate fails for numbeo.com
> <http://numbeo.com>" [hidden email]
> <mailto:[hidden email]>
> else
>     /etc/init.d/tomcat restart
> fi
>
> root@condor1796 ~ # cat fix_letsencrypt_chmod.sh
> #!/bin/bash
> chmod o+rx /etc/letsencrypt
> chmod -R o+rx /etc/letsencrypt/*
>
> root@condor1796 ~ #

I think your scripts will restart Tomcat even when it's not necessary.
The $? check before sending the email message looks like it should be
checking the result of the certbot command, but it's checking the result
of the chmod command instead. (Or maybe the result of the .sh script,
which will proably be 0.)

>      > *What would be the next steps how to identify the problem and perhaps
>      > solve it?*
>     What have you done so far?
>
>
> aaah... reading the Tomcat source to try to understand the state of Threads.
>
>     I don't see anything that sticks out in your thread dump.
>
>
> There are several threads which are trying to get monitor in
> AprEndpoint$Poller.add and no thread seems to be blocking it. Don't you
> find it weird:
>
> root@condor1796 ~ # grep Poller jstack_2020-12-16_31415.log  | grep "Apr"
>   - org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
> @bci=102, line=1398 (Compiled frame)

I might have found that odd had you posted that in your original
message, but you did not.

You need to show the full stack trace for that thread to make it
meaningful. Sockets are added to the poller all the time. It's not
unusual to see that happening. It they are getting *stuck*, that would
be bad, of course.

> Don't you find it weird that all threads are trying to get synchronized
> on a Poller instance and no one is in this block or another synchronized
> block/method?

I would find it weird if no threads were making any progress. Lots of
threads adding sockets to the poller is not out of the ordinary.

If you suspect a bug in Tomcat's socket handling, upgrading to the
latest 8.5.x release and re-trying would be the best move. There have
been many fixes since your 8.5.5 release which is now 4+ years old.

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Mladen Adamović
On Wed, Dec 16, 2020 at 7:07 PM Christopher Schultz <
[hidden email]> wrote:

> I think your scripts will restart Tomcat even when it's not necessary.
>

Hm, is this
https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates
the way to reload the certificates or is there another suggestion?


> The $? check before sending the email message looks like it should be
> checking the result of the certbot command, but it's checking the result
> of the chmod command instead. (Or maybe the result of the .sh script,
> which will proably be 0.)
>

Correct, thank you this is fixed now.


> I might have found that odd had you posted that in your original
> message, but you did not.
>
> You need to show the full stack trace for that thread to make it
> meaningful. Sockets are added to the poller all the time. It's not
> unusual to see that happening. It they are getting *stuck*, that would
> be bad, of course.
>

I did post full thread dump.


> > Don't you find it weird that all threads are trying to get synchronized
> > on a Poller instance and no one is in this block or another synchronized
> > block/method?
>
> I would find it weird if no threads were making any progress. Lots of
> threads adding sockets to the poller is not out of the ordinary.
>
> If you suspect a bug in Tomcat's socket handling, upgrading to the
> latest 8.5.x release and re-trying would be the best move. There have
> been many fixes since your 8.5.5 release which is now 4+ years old.
>

I have switched today from Apr connector to  Nio2 connector and no problem
yet.
What I have found strange is that processor usage is lower when using Nio2,
I have never tested it or tried it before, I have setup APR for
performances reason back in 2016. But oddly it seems with Nio2 processor
usage is lower.
I have tried the last OpenJDK but jstack is not working for me, so switched
back to the previous old JRM.
Let's see what will happen...




> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Mladen Adamović
I have these problems again. To narrow it down I have done the following:
- tested with the latest JDK8 - problem exists
- it exists in both Apr connector and Nio2 connector
- did log JVM Garbage Collector details - it's not due to Garbage collector

Christopher (or anyone else), if we had 1570 requests per minute where this
problem happened (approx. 26 per second), what do you think should our
Connector params be, it's currently:

   <Connector port="8181"
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
              SSLCertificateFile="/etc/letsencrypt/live/numbeo.com/cert.pem"
              SSLCertificateKeyFile="/etc/letsencrypt/live/
numbeo.com/privkey.pem"
              SSLCertificateChainFile="/etc/letsencrypt/live/
numbeo.com/chain.pem"
              SSLVerifyClient="optional" SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
              connectionTimeout="20000" acceptCount="30000"
              acceptorThreadCount="4"
              compression="on" maxConnections="50000" maxThreads="500"

compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,image/svg+xml,image/svg,image/png,image/jpeg"

              useSendfile="false"
              maxHttpHeaderSize="16392" SSLEnabled="true"
              enableLookups="false"
              scheme="https" secure="true"   clientAuth="false"
 useBodyEncodingForURI="true"
              URIEncoding="UTF-8"
              />





On Wed, Dec 16, 2020 at 7:32 PM Mladen Adamović <[hidden email]>
wrote:

> On Wed, Dec 16, 2020 at 7:07 PM Christopher Schultz <
> [hidden email]> wrote:
>
>> I think your scripts will restart Tomcat even when it's not necessary.
>>
>
> Hm, is this
> https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates
> the way to reload the certificates or is there another suggestion?
>
>
>> The $? check before sending the email message looks like it should be
>> checking the result of the certbot command, but it's checking the result
>> of the chmod command instead. (Or maybe the result of the .sh script,
>> which will proably be 0.)
>>
>
> Correct, thank you this is fixed now.
>
>
>> I might have found that odd had you posted that in your original
>> message, but you did not.
>>
>> You need to show the full stack trace for that thread to make it
>> meaningful. Sockets are added to the poller all the time. It's not
>> unusual to see that happening. It they are getting *stuck*, that would
>> be bad, of course.
>>
>
> I did post full thread dump.
>
>
>> > Don't you find it weird that all threads are trying to get synchronized
>> > on a Poller instance and no one is in this block or another
>> synchronized
>> > block/method?
>>
>> I would find it weird if no threads were making any progress. Lots of
>> threads adding sockets to the poller is not out of the ordinary.
>>
>> If you suspect a bug in Tomcat's socket handling, upgrading to the
>> latest 8.5.x release and re-trying would be the best move. There have
>> been many fixes since your 8.5.5 release which is now 4+ years old.
>>
>
> I have switched today from Apr connector to  Nio2 connector and no problem
> yet.
> What I have found strange is that processor usage is lower when using
> Nio2, I have never tested it or tried it before, I have setup APR for
> performances reason back in 2016. But oddly it seems with Nio2 processor
> usage is lower.
> I have tried the last OpenJDK but jstack is not working for me, so
> switched back to the previous old JRM.
> Let's see what will happen...
>
>
>
>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Mladen Adamović
I have just updated to the latest Tomcat 9.0.41

It seems that the problem doesn't appear anymore. If I get it reappeared,
I'll post new details to the Tomcat dev mailing list.


On Thu, Dec 17, 2020 at 8:56 AM Mladen Adamović <[hidden email]>
wrote:

> I have these problems again. To narrow it down I have done the following:
> - tested with the latest JDK8 - problem exists
> - it exists in both Apr connector and Nio2 connector
> - did log JVM Garbage Collector details - it's not due to Garbage collector
>
> Christopher (or anyone else), if we had 1570 requests per minute where
> this problem happened (approx. 26 per second), what do you think should our
> Connector params be, it's currently:
>
>    <Connector port="8181"
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>               SSLCertificateFile="/etc/letsencrypt/live/
> numbeo.com/cert.pem"
>               SSLCertificateKeyFile="/etc/letsencrypt/live/
> numbeo.com/privkey.pem"
>               SSLCertificateChainFile="/etc/letsencrypt/live/
> numbeo.com/chain.pem"
>               SSLVerifyClient="optional"
> SSLProtocol="TLSv1+TLSv1.1+TLSv1.2"
>               connectionTimeout="20000" acceptCount="30000"
>               acceptorThreadCount="4"
>               compression="on" maxConnections="50000" maxThreads="500"
>
> compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,image/svg+xml,image/svg,image/png,image/jpeg"
>
>               useSendfile="false"
>               maxHttpHeaderSize="16392" SSLEnabled="true"
>               enableLookups="false"
>               scheme="https" secure="true"   clientAuth="false"
>  useBodyEncodingForURI="true"
>               URIEncoding="UTF-8"
>               />
>
>
>
>
>
> On Wed, Dec 16, 2020 at 7:32 PM Mladen Adamović <[hidden email]>
> wrote:
>
>> On Wed, Dec 16, 2020 at 7:07 PM Christopher Schultz <
>> [hidden email]> wrote:
>>
>>> I think your scripts will restart Tomcat even when it's not necessary.
>>>
>>
>> Hm, is this
>> https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates
>> the way to reload the certificates or is there another suggestion?
>>
>>
>>> The $? check before sending the email message looks like it should be
>>> checking the result of the certbot command, but it's checking the result
>>> of the chmod command instead. (Or maybe the result of the .sh script,
>>> which will proably be 0.)
>>>
>>
>> Correct, thank you this is fixed now.
>>
>>
>>> I might have found that odd had you posted that in your original
>>> message, but you did not.
>>>
>>> You need to show the full stack trace for that thread to make it
>>> meaningful. Sockets are added to the poller all the time. It's not
>>> unusual to see that happening. It they are getting *stuck*, that would
>>> be bad, of course.
>>>
>>
>> I did post full thread dump.
>>
>>
>>> > Don't you find it weird that all threads are trying to get
>>> synchronized
>>> > on a Poller instance and no one is in this block or another
>>> synchronized
>>> > block/method?
>>>
>>> I would find it weird if no threads were making any progress. Lots of
>>> threads adding sockets to the poller is not out of the ordinary.
>>>
>>> If you suspect a bug in Tomcat's socket handling, upgrading to the
>>> latest 8.5.x release and re-trying would be the best move. There have
>>> been many fixes since your 8.5.5 release which is now 4+ years old.
>>>
>>
>> I have switched today from Apr connector to  Nio2 connector and no
>> problem yet.
>> What I have found strange is that processor usage is lower when using
>> Nio2, I have never tested it or tried it before, I have setup APR for
>> performances reason back in 2016. But oddly it seems with Nio2 processor
>> usage is lower.
>> I have tried the last OpenJDK but jstack is not working for me, so
>> switched back to the previous old JRM.
>> Let's see what will happen...
>>
>>
>>
>>
>>> -chris
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Christopher Schultz-2
In reply to this post by Mladen Adamović
Mladen,

On 12/16/20 13:32, Mladen Adamović wrote:

> On Wed, Dec 16, 2020 at 7:07 PM Christopher Schultz
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     I think your scripts will restart Tomcat even when it's not necessary.
>
>
> Hm, is this
> https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates 
> <https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates>
> the way to reload the certificates or is there another suggestion?

The technique showed there seems perfectly fine. Your script completely
restarts Tomcat instead of just pinging the connector to reload its
configuration.

>     I might have found that odd had you posted that in your original
>     message, but you did not.
>
>     You need to show the full stack trace for that thread to make it
>     meaningful. Sockets are added to the poller all the time. It's not
>     unusual to see that happening. It they are getting *stuck*, that would
>     be bad, of course.
>
>
> I did post full thread dump.

I don't see it in any of your previous posts.

>      > Don't you find it weird that all threads are trying to get
>     synchronized
>      > on a Poller instance and no one is in this block or another
>     synchronized
>      > block/method?
>
>     I would find it weird if no threads were making any progress. Lots of
>     threads adding sockets to the poller is not out of the ordinary.
>
>     If you suspect a bug in Tomcat's socket handling, upgrading to the
>     latest 8.5.x release and re-trying would be the best move. There have
>     been many fixes since your 8.5.5 release which is now 4+ years old.
>
>
> I have switched today from Apr connector to  Nio2 connector and no
> problem yet.
> What I have found strange is that processor usage is lower when using
> Nio2, I have never tested it or tried it before, I have setup APR for
> performances reason back in 2016. But oddly it seems with Nio2 processor
> usage is lower.

Theoretically, NIO2 should be the lowest CPU usage for the same
workload. APR does real blocking for certain operations while NIO/NIO2
does "simluated" blocking which usually ends up with slightly higher CPU
usage (because there is work being actively done wheras blocking usually
just uses IO interrupts).

> I have tried the last OpenJDK but jstack is not working for me, so
> switched back to the previous old JRM.

jstack isn't working? Hmm. Make sure that your jstack and Java binary
are the same version. If you are a different user you might have to
elevate privileges and/or use "jstack -F".

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: native connector, server problems with "No data received", what could be causing it?

Mladen Adamović
On Thu, Dec 17, 2020 at 7:57 PM Christopher Schultz <
[hidden email]> wrote:

> jstack isn't working? Hmm. Make sure that your jstack and Java binary
> are the same version. If you are a different user you might have to
> elevate privileges and/or use "jstack -F".
>

I have tried all 8 variations with sudo, -F , -J-d64 parameters and finally
it worked only with simply
sudo -u runninguser jstack
without -F and -J-d64 params



> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>