CurrentThreads not increasing in Apache Tomcat/9.0.37

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

CurrentThreads not increasing in Apache Tomcat/9.0.37

Vicente Perez
Hello,

We have a server with Apache Tomcat/9.0.37 which is dedicated for less than 100 concurrent users. We are using the ARP Connector with the parameter “maxThreads=400”.
 <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"  maxThreads="400" />

We have the next behavior:
Tomcat does not process any new request with currentThreadCount = X and  currentThreadBusy = X, but with X=30 or 50. ConnectionCount is increasing.

A far as I understood, the number of threads should increase if  currentThreadBusy is increasing.
We prevent this behavior by setting minSpareThreads to 100.


  *   Is this behavior correct?
  *   How the Connector should be configured in order to increase the currentThreadCount when currentThreadBusy is equal to currentThreadCount?

Best regards
Vicente Pérez

Reply | Threaded
Open this post in threaded view
|

Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

markt
On 25/09/2020 14:02, Vicente Perez wrote:
> Hello,
>
> We have a server with Apache Tomcat/9.0.37 which is dedicated for less than 100 concurrent users. We are using the ARP Connector with the parameter “maxThreads=400”.

I assume you mean the APR/Native connector here.

>  <Connector port="8080" protocol="HTTP/1.1"
>                connectionTimeout="20000"
>                redirectPort="8443"  maxThreads="400" />
>
> We have the next behavior:
> Tomcat does not process any new request with currentThreadCount = X and  currentThreadBusy = X, but with X=30 or 50. ConnectionCount is increasing.
>
> A far as I understood, the number of threads should increase if  currentThreadBusy is increasing.

Correct.

> We prevent this behavior by setting minSpareThreads to 100.
>
>   *   Is this behavior correct?

Doesn't look right.

>   *   How the Connector should be configured in order to increase the currentThreadCount when currentThreadBusy is equal to currentThreadCount?

This looks like a potential bug that needs further investigation.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

Vicente Perez

On 25-09-20 13:58, "Mark Thomas" <[hidden email]> wrote:

    On 25/09/2020 14:02, Vicente Perez wrote:
    > Hello,
    >
    > We have a server with Apache Tomcat/9.0.37 which is dedicated for less than 100 concurrent users. We are using the ARP Connector with the parameter “maxThreads=400”.

    I assume you mean the APR/Native connector here.

    >  <Connector port="8080" protocol="HTTP/1.1"
    >                connectionTimeout="20000"
    >                redirectPort="8443"  maxThreads="400" />
    >
    > We have the next behavior:
    > Tomcat does not process any new request with currentThreadCount = X and  currentThreadBusy = X, but with X=30 or 50. ConnectionCount is increasing.
    >
    > A far as I understood, the number of threads should increase if  currentThreadBusy is increasing.

    Correct.

    > We prevent this behavior by setting minSpareThreads to 100.
    >
    >   *   Is this behavior correct?

    Doesn't look right.

    >   *   How the Connector should be configured in order to increase the currentThreadCount when currentThreadBusy is equal to currentThreadCount?

    This looks like a potential bug that needs further investigation.

    Mark

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]


Thank you for the answer.

Yes. It is  APR/Native Connector.

Please let me know if you need more information about the configuration/environment.

Regards
Vicente


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

markt
On 25/09/2020 18:07, Vicente Perez wrote:

<snip/>

> Thank you for the answer.
>
> Yes. It is  APR/Native Connector.

Are you sure? It isn't impossible that you are using the APR/Native
connector given your configuration but it would be unusual.

Look in the logs for a line that contains:
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ...

What is the full text of that line?

Also, what version of the Tomcat Native Connector are you using?
Look in the logs for a line that contains:

org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
Apache Tomcat Native library [???] using APR version [???]

What is the full text of that line?

> Please let me know if you need more information about the configuration/environment.

I ran some simple tests with ab and I cannot repeat the problem you
describe.

As I increase the concurrency with ab, currentThreadCount increases.

I even reduced minSpareThreads to 0 and I still could not reproduce the
behaviour.

To investigate this further you'll need to provide the steps to
reproduce this from a clean install of the latest stable 9.0.x release
(9.0.38 as I type this) and the latest stable Tomcat Native (1.2.25).

Ideally the test case would involve nothing more than a clean Tomcat
install and ab to generate the load.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

Vicente Perez

    <snip/>

    > Thank you for the answer.
    >
    > Yes. It is  APR/Native Connector.

    Are you sure? It isn't impossible that you are using the APR/Native
    connector given your configuration but it would be unusual.

    Look in the logs for a line that contains:
    org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ...

    What is the full text of that line?

    Also, what version of the Tomcat Native Connector are you using?
    Look in the logs for a line that contains:

    org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
    Apache Tomcat Native library [???] using APR version [???]

    What is the full text of that line?

    > Please let me know if you need more information about the configuration/environment.

    I ran some simple tests with ab and I cannot repeat the problem you
    describe.

    As I increase the concurrency with ab, currentThreadCount increases.

    I even reduced minSpareThreads to 0 and I still could not reproduce the
    behaviour.

    To investigate this further you'll need to provide the steps to
    reproduce this from a clean install of the latest stable 9.0.x release
    (9.0.38 as I type this) and the latest stable Tomcat Native (1.2.25).

    Ideally the test case would involve nothing more than a clean Tomcat
    install and ab to generate the load.

    Mark

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]


Hi Mark,

Thanks again.

 >Look in the logs for a line that contains:
24-Sep-2020 17:00:13.744 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]

24-Sep-2020 17:00:13.407 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib]

This is the whole startup log before loading web apps.

NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
24-Sep-2020 17:00:13.393 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.37
24-Sep-2020 17:00:13.395 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Jun 30 2020 20:09:49 UTC
24-Sep-2020 17:00:13.395 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.37.0
24-Sep-2020 17:00:13.396 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
24-Sep-2020 17:00:13.396 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.15.0-109-generic
24-Sep-2020 17:00:13.396 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
24-Sep-2020 17:00:13.396 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-11-openjdk-amd64
24-Sep-2020 17:00:13.397 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.7+10-post-Ubuntu-2ubuntu218.04
24-Sep-2020 17:00:13.397 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Ubuntu
24-Sep-2020 17:00:13.397 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /srv/tomcat9/apache-tomcat-9.0.37
24-Sep-2020 17:00:13.397 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /srv/tomcat9/apache-tomcat-9.0.37
24-Sep-2020 17:00:13.401 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
24-Sep-2020 17:00:13.401 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
24-Sep-2020 17:00:13.401 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
24-Sep-2020 17:00:13.402 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/srv/tomcat9/apache-tomcat-9.0.37/conf/logging.properties
24-Sep-2020 17:00:13.402 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
24-Sep-2020 17:00:13.402 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.security.egd=file:///dev/urandom
24-Sep-2020 17:00:13.402 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
24-Sep-2020 17:00:13.403 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
24-Sep-2020 17:00:13.403 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
24-Sep-2020 17:00:13.403 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
24-Sep-2020 17:00:13.403 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -XX:+UseParallelGC
24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote
24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.port=9010
24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.rmi.port=9010
24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.local.only=false
24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.ssl=false
24-Sep-2020 17:00:13.405 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.authenticate=false
24-Sep-2020 17:00:13.405 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Denvironment.name=prod
24-Sep-2020 17:00:13.405 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dzookeeper.url=192.168.6.201:2181,192.168.6.203:2181,192.168.6.204:2181
24-Sep-2020 17:00:13.405 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dtomcat.instance.name=srv-tomcat9-tanatos
24-Sep-2020 17:00:13.406 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dzone.name=zone1
24-Sep-2020 17:00:13.406 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xmx10240M
24-Sep-2020 17:00:13.406 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Xms512M
24-Sep-2020 17:00:13.406 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Duser.timezone=UTC
24-Sep-2020 17:00:13.406 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
24-Sep-2020 17:00:13.407 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/srv/tomcat9/apache-tomcat-9.0.37
24-Sep-2020 17:00:13.407 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/srv/tomcat9/apache-tomcat-9.0.37
24-Sep-2020 17:00:13.407 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/srv/tomcat9/apache-tomcat-9.0.37/temp
24-Sep-2020 17:00:13.407 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib]
24-Sep-2020 17:00:13.744 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
24-Sep-2020 17:00:13.796 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [726] milliseconds
24-Sep-2020 17:00:13.858 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
24-Sep-2020 17:00:13.859 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.37]


>To investigate this further you'll need to provide the steps to
>   reproduce this from a clean install of the latest stable 9.0.x release
>  (9.0.38 as I type this) and the latest stable Tomcat Native (1.2.25).

I will try to reproduce the error in that way.

Regards
Vicente



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: CurrentThreads not increasing in Apache Tomcat/9.0.37

markt
On 25/09/2020 22:29, Vicente Perez wrote:

<snip/>

>  >Look in the logs for a line that contains:
> 24-Sep-2020 17:00:13.744 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
>
> 24-Sep-2020 17:00:13.407 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib]

You are using the NIO HTTP connector. You are not using APR/native.

> 24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote
> 24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.port=9010
> 24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.rmi.port=9010
> 24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.local.only=false
> 24-Sep-2020 17:00:13.404 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.ssl=false
> 24-Sep-2020 17:00:13.405 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcom.sun.management.jmxremote.authenticate=false

Remote JMX enabled without TLS? I'd check that is what was intended.

>> To investigate this further you'll need to provide the steps to
>>   reproduce this from a clean install of the latest stable 9.0.x release
>>  (9.0.38 as I type this) and the latest stable Tomcat Native (1.2.25).
>
> I will try to reproduce the error in that way.

Thanks.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]