HTTP thread pool size remained high after database failover

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

HTTP thread pool size remained high after database failover

Suvendu Sekhar Mondal
Hello Everyone,

During database failover resiliency testing we noticed something
unusual which we still cannot explain. So, reaching out to the
community for help.

Application Setup:
AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016

Observations:
After database failover, we noticed that HTTP thread pool size jumped
from 10 to 50. No. of busy threads jumped to 40 for a minute and then
came down to normal range(1-5). But thread pool size remained 50 for
the next one hour. When we collected the heap dump after one hour,
thread pool size jumped again to ~90. Thread dump analysis is showing
most of the HTTP threads were just waiting to get some work[1]. That
looked surprising to me. Few of them are runnable and doing some work
- which matches "busy thread" count. During all this timeframe load on
the system remained the same.

We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
do not have any keepAliveTimeout set in Tomcat. App uses session
affinity. We were looking for some connector properties which deal
with idle threads but found none[3].

We are not seeing any "problem" in our app after DB failover but this
is something also newer to us and we are trying to explain it.
Tomcat's memory usage increased by a few MB as new threads were
spawned. My expectation was that if threads in the thread pool are not
getting used and idle, after sometime thread pool size will reduce to
"minSpareThreads" size which is 10 in our case. Is that expectation
wrong? Can you please share your thoughts?

[1]
"http-apr-18110-exec-63" #2170 daemon prio=5 os_prio=0
tid=0x0000000045f03000 nid=0x30d4 waiting on condition
[0x000000005a4cf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000006a5400178> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86)
at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
- None

[2]
<Connector port="18110" protocol="HTTP/1.1" maxThreads="500"
acceptCount="300" URIEncoding="UTF-8" redirectPort="18113"
connectionTimeout="30000"/>

<Connector port="18113" threadPriority="6" compression="force"
compressionMinSize="2048"
compressableMimeTypes="text/html,text/xml,text/plain,application/json"
sslProtocol="TLS" secure="true" scheme="https" protocol="HTTP/1.1"
maxThreads="150" maxSavePostSize="-1" maxHttpHeaderSize="8192"
keystoreFile="conf/.keystore" enableLookups="false"
disableUploadTimeout="true" clientAuth="false" acceptCount="100"
SSLPassword="" SSLEnabled="true" SSLCertificateKeyFile="\app\my.key"
SSLCertificateFile="\app\mycert.cer"/>

[3]
https://tomcat.apache.org/tomcat-7.0-doc/config/http.html

Thanks & Regards,
Suvendu

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: HTTP thread pool size remained high after database failover

Michael Osipov
Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal:
> Hello Everyone,
>
> During database failover resiliency testing we noticed something
> unusual which we still cannot explain. So, reaching out to the
> community for help.
>
> Application Setup:
> AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016

Almost of them them are outdated, you should upgrade first.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: HTTP thread pool size remained high after database failover

Suvendu Sekhar Mondal
Hi Michael,

On Wed, Nov 11, 2020 at 6:03 PM Michael Osipov <[hidden email]> wrote:

>
> Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal:
> > Hello Everyone,
> >
> > During database failover resiliency testing we noticed something
> > unusual which we still cannot explain. So, reaching out to the
> > community for help.
> >
> > Application Setup:
> > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016
>
> Almost of them them are outdated, you should upgrade first.

I know. We will be upgrading them in the next release. For this
release we have to use aforementioned softwares.

For the last three releases we used the same softwares but had not
noticed this issue - maybe we were not looking that closely. Now, we
are seeing this and can't let it go without explaining it. :)

So, any thoughts on this?

>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: HTTP thread pool size remained high after database failover

Christopher Schultz-2
In reply to this post by Suvendu Sekhar Mondal
Suvendu,

On 11/11/20 06:40, Suvendu Sekhar Mondal wrote:
> Application Setup:
> AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016

As Michael pointed-out, these all need to be upgraded, but they are not
affecting your experience, here.

> Observations:
> After database failover, we noticed that HTTP thread pool size jumped
> from 10 to 50. No. of busy threads jumped to 40 for a minute and then
> came down to normal range(1-5). But thread pool size remained 50 for
> the next one hour. When we collected the heap dump after one hour,
> thread pool size jumped again to ~90. Thread dump analysis is showing
> most of the HTTP threads were just waiting to get some work[1]. That
> looked surprising to me. Few of them are runnable and doing some work
> - which matches "busy thread" count. During all this timeframe load on
> the system remained the same.
>
> We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
> do not have any keepAliveTimeout set in Tomcat. App uses session
> affinity. We were looking for some connector properties which deal
> with idle threads but found none[3].

You won't find them.

> We are not seeing any "problem" in our app after DB failover but this
> is something also newer to us and we are trying to explain it.

What are you trying to explain?

> Tomcat's memory usage increased by a few MB as new threads were
> spawned. My expectation was that if threads in the thread pool are not
> getting used and idle, after sometime thread pool size will reduce to
> "minSpareThreads" size which is 10 in our case. Is that expectation
> wrong? Can you please share your thoughts?

This is a common misconception. If you do not configure an "executor"
then Tomcat will create one for you and it will NOT remove idle threads
from the thread pool.

The solution is to manually-specify an Executor:

<Executor name="myExecutor" namePrefix="myExecutor-" maxThreads="X"
minSpareThreads="Y" />

<Connector executor="myExecutor" ... />

With this configuration, you should see your thread count stay between Y
and X, but when X are not necessary, your thread count should drop
towards Y as appropriate.

Hope that helps,
-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: HTTP thread pool size remained high after database failover

Suvendu Sekhar Mondal
Thank you Chris for your response.

On Wed, Nov 11, 2020 at 10:02 PM Christopher Schultz
<[hidden email]> wrote:

>
> Suvendu,
>
> On 11/11/20 06:40, Suvendu Sekhar Mondal wrote:
> > Application Setup:
> > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016
>
> As Michael pointed-out, these all need to be upgraded, but they are not
> affecting your experience, here.
>
> > Observations:
> > After database failover, we noticed that HTTP thread pool size jumped
> > from 10 to 50. No. of busy threads jumped to 40 for a minute and then
> > came down to normal range(1-5). But thread pool size remained 50 for
> > the next one hour. When we collected the heap dump after one hour,
> > thread pool size jumped again to ~90. Thread dump analysis is showing
> > most of the HTTP threads were just waiting to get some work[1]. That
> > looked surprising to me. Few of them are runnable and doing some work
> > - which matches "busy thread" count. During all this timeframe load on
> > the system remained the same.
> >
> > We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We
> > do not have any keepAliveTimeout set in Tomcat. App uses session
> > affinity. We were looking for some connector properties which deal
> > with idle threads but found none[3].
>
> You won't find them.
>
Okay.
> > We are not seeing any "problem" in our app after DB failover but this
> > is something also newer to us and we are trying to explain it.
>
> What are you trying to explain?
>
We are simply trying to understand this behavior as this is something
new we noticed.

> > Tomcat's memory usage increased by a few MB as new threads were
> > spawned. My expectation was that if threads in the thread pool are not
> > getting used and idle, after sometime thread pool size will reduce to
> > "minSpareThreads" size which is 10 in our case. Is that expectation
> > wrong? Can you please share your thoughts?
>
> This is a common misconception. If you do not configure an "executor"
> then Tomcat will create one for you and it will NOT remove idle threads
> from the thread pool.
>
> The solution is to manually-specify an Executor:
>
> <Executor name="myExecutor" namePrefix="myExecutor-" maxThreads="X"
> minSpareThreads="Y" />
>
> <Connector executor="myExecutor" ... />
>
> With this configuration, you should see your thread count stay between Y
> and X, but when X are not necessary, your thread count should drop
> towards Y as appropriate.
>

Thanks for the explanation. This is the kind of information I was
looking for in Tomcat docs but was not able to locate it. We will try
this approach and let you know the outcome.
> Hope that helps,
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]