Can't get over 100 client connections?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Can't get over 100 client connections?

John.E.Gregg-2
All,

I'm using Tomcat 7.0.75.

I cannot get my connection or thread count over 100 no matter how much load I throw at the server.  Currently I just have a dummy app deployed that doesn't do much except sleep for about 500ms and return a canned response.  Initially I had maxThreads=20 with no explicit maxConnections.  I could get the "current threads busy" metric to 20 easily.  Then I raised it to 100 and was able to reach that number easily.  Additionally, netstat told me I had 100 incoming connections to port 5114.  However raising the maxThreads higher doesn't make any difference, nor does explicitly specifying maxConnections=200.  In either case, I can only get 100 busy threads and 100 active connections.  Once I reach that limit, the response times on the client start going up.  The Tomcat server itself isn't stressed at all.  CPU and GC are low.  The client is on a different server in the same data center.

I'm not going through a load balancer.  I'm going directly to the connector below:


    <Connector port="5114"
        protocol="HTTP/1.1"
        SSLEnabled="true"
        maxConnections="200"
        maxThreads="200"
        maxKeepAliveRequests="100"
        keepAliveTimeout="10000"
        scheme="https"
        secure="true"
        clientAuth="true"
        sslProtocol="TLS"
        keystoreFile="/app/comp/myapp/certs/appcerttestkeystore"
        keystorePass="${keystore.password}"
        keyAlias="test"
        truststoreFile="/app/comp/myapp/certs/cacerts"
        truststorePass="${truststore.password}"
        allowUnsafeLegacyRenegotiation="false"
                ciphers="blah blah blah"
        />

Thanks

John

Reply | Threaded
Open this post in threaded view
|

Re: Can't get over 100 client connections?

André Warnier (tomcat)
On 16.05.2017 19:57, [hidden email] wrote:

> All,
>
> I'm using Tomcat 7.0.75.
>
> I cannot get my connection or thread count over 100 no matter how much load I throw at the server.  Currently I just have a dummy app deployed that doesn't do much except sleep for about 500ms and return a canned response.  Initially I had maxThreads=20 with no explicit maxConnections.  I could get the "current threads busy" metric to 20 easily.  Then I raised it to 100 and was able to reach that number easily.  Additionally, netstat told me I had 100 incoming connections to port 5114.  However raising the maxThreads higher doesn't make any difference, nor does explicitly specifying maxConnections=200.  In either case, I can only get 100 busy threads and 100 active connections.  Once I reach that limit, the response times on the client start going up.  The Tomcat server itself isn't stressed at all.  CPU and GC are low.  The client is on a different server in the same data center.
>
> I'm not going through a load balancer.  I'm going directly to the connector below:
>
>
>      <Connector port="5114"
>          protocol="HTTP/1.1"
>          SSLEnabled="true"
>          maxConnections="200"
>          maxThreads="200"
>          maxKeepAliveRequests="100"
>          keepAliveTimeout="10000"
>          scheme="https"
>          secure="true"
>          clientAuth="true"
>          sslProtocol="TLS"
>          keystoreFile="/app/comp/myapp/certs/appcerttestkeystore"
>          keystorePass="${keystore.password}"
>          keyAlias="test"
>          truststoreFile="/app/comp/myapp/certs/cacerts"
>          truststorePass="${truststore.password}"
>          allowUnsafeLegacyRenegotiation="false"
>                  ciphers="blah blah blah"
>          />
>

Hi.
I do not know with what you are testing (as a client).
But be aware of the following :

1) >          keepAliveTimeout="10000"
means 10 seconds.
It means that, after the last request which one particular client sends on its connection
to Tomcat, and Tomcat has responded to it, Tomcat will keep that connection open for an
additional 10 s., just waiting to see if that same client has anything more to request.
Since you are not using an Executor, keeping the connection open will also mean keeping
the corresponding Tomcat thread alive, also waiting.
Only once this time is over, will Tomcat close this connection, and "recycle" the thread
to serve another client connection.
2) there may be a limit in the server OS, as to how many connections a process can have
open at the same time. If that limit is reached at some point, that may either crash the
process that wants an additional one, or put it in some wait queue until one is available
again.
3) when a client opens a connection to a server (or tries to), and the server process does
not immediately respond to the "open connection" request, the TCP/IP stack on the server
will place the connection-open request in a wait queue. The size of that queue is an
adjustable TCP/IP parameter.
 From the client side, if its connection is not accepted immediately (but not rejected
right away), the client will just wait, until it is accepted. There is usually a timeout
for this also, on the client side.

Some combination of the above may explain what you see.



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Can't get over 100 client connections?

John.E.Gregg-2
André,


> -----Original Message-----
> From: André Warnier (tomcat) [mailto:[hidden email]]
> Sent: Tuesday, May 16, 2017 2:00 PM
> To: [hidden email]
> Subject: Re: Can't get over 100 client connections?
>
> On 16.05.2017 19:57, [hidden email] wrote:
> > All,
> >
> > I'm using Tomcat 7.0.75.
> >
> > I cannot get my connection or thread count over 100 no matter how much
> load I throw at the server.  Currently I just have a dummy app deployed that
> doesn't do much except sleep for about 500ms and return a canned
> response.  Initially I had maxThreads=20 with no explicit maxConnections.  I
> could get the "current threads busy" metric to 20 easily.  Then I raised it to
> 100 and was able to reach that number easily.  Additionally, netstat told me I
> had 100 incoming connections to port 5114.  However raising the maxThreads
> higher doesn't make any difference, nor does explicitly specifying
> maxConnections=200.  In either case, I can only get 100 busy threads and 100
> active connections.  Once I reach that limit, the response times on the client
> start going up.  The Tomcat server itself isn't stressed at all.  CPU and GC are
> low.  The client is on a different server in the same data center.
> >
> > I'm not going through a load balancer.  I'm going directly to the connector
> below:
> >
> >
> >      <Connector port="5114"
> >          protocol="HTTP/1.1"
> >          SSLEnabled="true"
> >          maxConnections="200"
> >          maxThreads="200"
> >          maxKeepAliveRequests="100"
> >          keepAliveTimeout="10000"
> >          scheme="https"
> >          secure="true"
> >          clientAuth="true"
> >          sslProtocol="TLS"
> >          keystoreFile="/app/comp/myapp/certs/appcerttestkeystore"
> >          keystorePass="${keystore.password}"
> >          keyAlias="test"
> >          truststoreFile="/app/comp/myapp/certs/cacerts"
> >          truststorePass="${truststore.password}"
> >          allowUnsafeLegacyRenegotiation="false"
> >                  ciphers="blah blah blah"
> >          />
> >
>
> Hi.
> I do not know with what you are testing (as a client).
> But be aware of the following :
>
> 1) >          keepAliveTimeout="10000"
> means 10 seconds.
> It means that, after the last request which one particular client sends on its
> connection to Tomcat, and Tomcat has responded to it, Tomcat will keep that
> connection open for an additional 10 s., just waiting to see if that same client
> has anything more to request.
> Since you are not using an Executor, keeping the connection open will also
> mean keeping the corresponding Tomcat thread alive, also waiting.
> Only once this time is over, will Tomcat close this connection, and "recycle"
> the thread to serve another client connection.
> 2) there may be a limit in the server OS, as to how many connections a
> process can have open at the same time. If that limit is reached at some
> point, that may either crash the process that wants an additional one, or put
> it in some wait queue until one is available again.
> 3) when a client opens a connection to a server (or tries to), and the server
> process does not immediately respond to the "open connection" request,
> the TCP/IP stack on the server will place the connection-open request in a
> wait queue. The size of that queue is an adjustable TCP/IP parameter.
>  From the client side, if its connection is not accepted immediately (but not
> rejected right away), the client will just wait, until it is accepted. There is
> usually a timeout for this also, on the client side.
>
> Some combination of the above may explain what you see.
>

Thanks for the suggestions.  I don't know 100% the cause yet but it appears to be an issue on the client side.  When running multiple clients I can get the concurrent requests over 100 with no trouble.

Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Can't get over 100 client connections?

Christopher Schultz-2
In reply to this post by John.E.Gregg-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

John,

On 5/16/17 1:57 PM, [hidden email] wrote:

> All,
>
> I'm using Tomcat 7.0.75.
>
> I cannot get my connection or thread count over 100 no matter how
> much load I throw at the server.  Currently I just have a dummy app
> deployed that doesn't do much except sleep for about 500ms and
> return a canned response.  Initially I had maxThreads=20 with no
> explicit maxConnections.  I could get the "current threads busy"
> metric to 20 easily.  Then I raised it to 100 and was able to reach
> that number easily.  Additionally, netstat told me I had 100
> incoming connections to port 5114.  However raising the maxThreads
> higher doesn't make any difference, nor does explicitly specifying
> maxConnections=200.  In either case, I can only get 100 busy
> threads and 100 active connections.  Once I reach that limit, the
> response times on the client start going up.  The Tomcat server
> itself isn't stressed at all.  CPU and GC are low.  The client is
> on a different server in the same data center.
>
> I'm not going through a load balancer.  I'm going directly to the
> connector below:
>
>
> <Connector port="5114" protocol="HTTP/1.1" SSLEnabled="true"
> maxConnections="200" maxThreads="200" maxKeepAliveRequests="100"
> keepAliveTimeout="10000" scheme="https" secure="true"
> clientAuth="true" sslProtocol="TLS"
> keystoreFile="/app/comp/myapp/certs/appcerttestkeystore"
> keystorePass="${keystore.password}" keyAlias="test"
> truststoreFile="/app/comp/myapp/certs/cacerts"
> truststorePass="${truststore.password}"
> allowUnsafeLegacyRenegotiation="false" ciphers="blah blah blah" />

What is the benchmark client and how do you invoke it?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlkdK6sACgkQHPApP6U8
pFh1zxAAlYXrL7LVfojOlVFfXizjNCC9ioE1ogn2557ga0lpU34D58FxtnSeP+DK
EywaYVNAcWzckFcoTc796GJJ9DVnmZN+rbGosT5jPR42pLYw9N2M6N3MVp0PQ/Tn
hud1X9giCDIqgC3YsuxtMqfwU4haVS7+qARrfpdVf9s+J0NMjTQK1AYmM78CSLeZ
RRe13eQDZFShl4cVrJhuRilu0Bf8vKSSMXFUCPpaZo1yU6Js6IVfS+YZwTpwjtfO
EYQQjQ6t5cplpv7fvGiiiOEf43/tTH+Cyii0M2VmdYnhDh3/WJUClbVtLE2zkVSY
7TG32sSGrjpUOS5/Zucbfjsc8G5E5y9AFdZctGK9A513yiLr+uB+YEG7TehwNelN
Z3wUGKZ6KXeAtjs2HMfg2tZ+6G2VMKUWAcn2+AC/vhEfq2DjdDn4KNmFma1WMVMW
k3J8khO7zbGyWFuzOyFrZ3dQyjGwWhLsmeiALphLkIKSVcUTD93rIidlvwEu2P3x
yUDeo+sNqwu0tr5j3b+CLPSFT8rClEIuG/yw3SFpeB/6XSOwJt5U5zspVVI7HRgd
7f0q9rVCy4mL9241CF0ntV9XN5EIpRi9EnPL2nXCz2eZTWBEGVjyo0CL+z8m5DZD
skw0vv6f6AlKjYrpk5Kt4c3zJFynUf8nP2B1xJ4NFggIYWQS6Ss=
=luv6
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Can't get over 100 client connections?

John.E.Gregg-2
Hi Chris

> -----Original Message-----
> From: Christopher Schultz [mailto:[hidden email]]
> Sent: Thursday, May 18, 2017 12:06 AM
> To: [hidden email]
> Subject: Re: Can't get over 100 client connections?
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> John,
>
> On 5/16/17 1:57 PM, [hidden email] wrote:
> > All,
> >
> > I'm using Tomcat 7.0.75.
> >
> > I cannot get my connection or thread count over 100 no matter how much
> > load I throw at the server.  Currently I just have a dummy app
> > deployed that doesn't do much except sleep for about 500ms and return
> > a canned response.  Initially I had maxThreads=20 with no explicit
> > maxConnections.  I could get the "current threads busy"
> > metric to 20 easily.  Then I raised it to 100 and was able to reach
> > that number easily.  Additionally, netstat told me I had 100 incoming
> > connections to port 5114.  However raising the maxThreads higher
> > doesn't make any difference, nor does explicitly specifying
> > maxConnections=200.  In either case, I can only get 100 busy threads
> > and 100 active connections.  Once I reach that limit, the response
> > times on the client start going up.  The Tomcat server itself isn't
> > stressed at all.  CPU and GC are low.  The client is on a different
> > server in the same data center.
> >
> > I'm not going through a load balancer.  I'm going directly to the
> > connector below:
> >
> >
> > <Connector port="5114" protocol="HTTP/1.1" SSLEnabled="true"
> > maxConnections="200" maxThreads="200" maxKeepAliveRequests="100"
> > keepAliveTimeout="10000" scheme="https" secure="true"
> > clientAuth="true" sslProtocol="TLS"
> > keystoreFile="/app/comp/myapp/certs/appcerttestkeystore"
> > keystorePass="${keystore.password}" keyAlias="test"
> > truststoreFile="/app/comp/myapp/certs/cacerts"
> > truststorePass="${truststore.password}"
> > allowUnsafeLegacyRenegotiation="false" ciphers="blah blah blah" />
>
> What is the benchmark client and how do you invoke it?
>
> - -chris

LoadRunner --> vendor app using Commons HttpClient 4.x --> my Tomcat server.  Tomcat is running an app I wrote to stub out the backend because we're not allowed to run load tests against it.  I confirmed that Tomcat itself could handle more than 100 connections by running the vendor app under load, as well as a test from JMeter directly against my Tomcat server.  Worked fine.  It's the vendor app that appears to have hit a 100 connection limit.

Interestingly, JMeter with the option to use HttpClient 4.x opens and closes a lot of connections.  The exact same JMeter test with HttpClient 3.x selected reuses the connection the way I expect.

If you're on the HttpClient users list, you might see a question from me!  I don't understand why HttpClient is behaving the way it is.

Thanks



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]