mod_jk endpoints

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

mod_jk endpoints

Jerry Malcolm
Thanks for the responses to my earlier post about websockets/JSPs. 
Http/2 is definitely the right answer for my issue there.

HTTP/2 solved the problem with the throttling the multiple parallel
requests with the browser connection limits.  But now opening that
throttle has now exposed a different problem at the server.  Quick
refresh.... I have an app that does not have many concurrent users at
this time.  However it does periodically have a very large number of
concurrent requests (AJAX/REST) from a single client.

I now realize that with the browser limiting concurrent requests in
HTTP/1.1, mod_jk was not getting overwhelmed.  But now that the
floodgates have opened with HTTP/2, I realize that I have some serious
configuration problems with mod_jk.  The requests are clogging up in
mod_jk, timing out, and logging errors something like "cannot get
endpoint for worker".  It then takes about 5 minutes for the clog to
clear out before mod_jk is functional again for the other virtual hosts
I'm hosting.

Specifics... I'm using Apache 2.4.33, Tomcat 9.0.  I tried to download
the latest mod_jk.so.  The latest mod_jk for httpd on the binaries
download site is from 2016.  All the versions for the past 2 years are
iis only... I assume that's ok???  So I'm using the 2016 1.2.40 version.

My question is.... how do I increase the number of 'endpoints' that
mod_jk has access to?  Is that a TC configuration or workers.properties
setting?  I want to make sure mod_jk has enough resources/endpoints to
pass (i.e.not 'block') all the requests it receives.  Once mod_jk is
passing everything, I realize the flood might hit TC, and I may be back
asking about increasing TC resources.  But for now, I just want to get
mod_jk opened up.

Thanks again.

Jerry


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

Greg Huber
> So I'm using the 2016 1.2.40 version.

Think you have to build mod_jk to use the latest and greatest, this is what
I have in my notes (also from the website docs):

mod_jk.so build
    yum install httpd-devel
    download tomcat-connectors-x.x.xx-src.tar.gz
    extract and cd into the native folder
    cd /opt/apache-tomcat/tomcat-connectors-x.x.xx-src/native
    ./configure --with-apxs=/usr/bin/apxs
    make
    copy file ./apache-2.0/mod_jk.so to /etc/httpd/modules
    make clean

(I only have a pretty much standard workers.properties so can't help much.)

On Sun, 21 Oct 2018 at 02:01, Jerry Malcolm <[hidden email]> wrote:

> Thanks for the responses to my earlier post about websockets/JSPs.
> Http/2 is definitely the right answer for my issue there.
>
> HTTP/2 solved the problem with the throttling the multiple parallel
> requests with the browser connection limits.  But now opening that
> throttle has now exposed a different problem at the server.  Quick
> refresh.... I have an app that does not have many concurrent users at
> this time.  However it does periodically have a very large number of
> concurrent requests (AJAX/REST) from a single client.
>
> I now realize that with the browser limiting concurrent requests in
> HTTP/1.1, mod_jk was not getting overwhelmed.  But now that the
> floodgates have opened with HTTP/2, I realize that I have some serious
> configuration problems with mod_jk.  The requests are clogging up in
> mod_jk, timing out, and logging errors something like "cannot get
> endpoint for worker".  It then takes about 5 minutes for the clog to
> clear out before mod_jk is functional again for the other virtual hosts
> I'm hosting.
>
> Specifics... I'm using Apache 2.4.33, Tomcat 9.0.  I tried to download
> the latest mod_jk.so.  The latest mod_jk for httpd on the binaries
> download site is from 2016.  All the versions for the past 2 years are
> iis only... I assume that's ok???  So I'm using the 2016 1.2.40 version.
>
> My question is.... how do I increase the number of 'endpoints' that
> mod_jk has access to?  Is that a TC configuration or workers.properties
> setting?  I want to make sure mod_jk has enough resources/endpoints to
> pass (i.e.not 'block') all the requests it receives.  Once mod_jk is
> passing everything, I realize the flood might hit TC, and I may be back
> asking about increasing TC resources.  But for now, I just want to get
> mod_jk opened up.
>
> Thanks again.
>
> Jerry
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

markt
On October 21, 2018 7:38:25 AM UTC, Greg Huber <[hidden email]> wrote:
>> So I'm using the 2016 1.2.40 version.
>
>Think you have to build mod_jk to use the latest and greatest,

Correct.

We stopped providing binary builds for anything other than Windows because there were just too many combinations. And it is relatively simple to build on Linux.

We provide the Windows binaries because the build process is more involved, most users don't have the necessary toolchain  easily available and we only need to provide one binary for all Windows platforms (ok we provide several binaries with different build options but it is still a small, manageable number).

> this is
>what
>I have in my notes (also from the website docs):
>
>mod_jk.so build
>    yum install httpd-devel
>    download tomcat-connectors-x.x.xx-src.tar.gz
>    extract and cd into the native folder
>    cd /opt/apache-tomcat/tomcat-connectors-x.x.xx-src/native
>    ./configure --with-apxs=/usr/bin/apxs
>    make
>    copy file ./apache-2.0/mod_jk.so to /etc/httpd/modules
>    make clean
>
>(I only have a pretty much standard workers.properties so can't help
>much.)
>
>On Sun, 21 Oct 2018 at 02:01, Jerry Malcolm <[hidden email]>
>wrote:
>
>> Thanks for the responses to my earlier post about websockets/JSPs.
>> Http/2 is definitely the right answer for my issue there.
>>
>> HTTP/2 solved the problem with the throttling the multiple parallel
>> requests with the browser connection limits.  But now opening that
>> throttle has now exposed a different problem at the server.  Quick
>> refresh.... I have an app that does not have many concurrent users at
>> this time.  However it does periodically have a very large number of
>> concurrent requests (AJAX/REST) from a single client.
>>
>> I now realize that with the browser limiting concurrent requests in
>> HTTP/1.1, mod_jk was not getting overwhelmed.  But now that the
>> floodgates have opened with HTTP/2, I realize that I have some
>serious
>> configuration problems with mod_jk.  The requests are clogging up in
>> mod_jk, timing out, and logging errors something like "cannot get
>> endpoint for worker".  It then takes about 5 minutes for the clog to
>> clear out before mod_jk is functional again for the other virtual
>hosts
>> I'm hosting.
>>
>> Specifics... I'm using Apache 2.4.33, Tomcat 9.0.  I tried to
>download
>> the latest mod_jk.so.  The latest mod_jk for httpd on the binaries
>> download site is from 2016.  All the versions for the past 2 years
>are
>> iis only... I assume that's ok???  So I'm using the 2016 1.2.40
>version.
>>
>> My question is.... how do I increase the number of 'endpoints' that
>> mod_jk has access to?  Is that a TC configuration or
>workers.properties
>> setting?  I want to make sure mod_jk has enough resources/endpoints
>to
>> pass (i.e.not 'block') all the requests it receives.  Once mod_jk is
>> passing everything, I realize the flood might hit TC, and I may be
>back
>> asking about increasing TC resources.  But for now, I just want to
>get
>> mod_jk opened up.

We need to see the configuration for:

- httpd mpm / threads / processes
- workers.proprties
- Tomcat's AJP connector

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

Rainer Jung-3
Am 21.10.2018 um 10:17 schrieb Mark Thomas:

> On October 21, 2018 7:38:25 AM UTC, Greg Huber <[hidden email]> wrote:
>>> So I'm using the 2016 1.2.40 version.
>>
>> Think you have to build mod_jk to use the latest and greatest,
>
> Correct.
>
> We stopped providing binary builds for anything other than Windows because there were just too many combinations. And it is relatively simple to build on Linux.
>
> We provide the Windows binaries because the build process is more involved, most users don't have the necessary toolchain  easily available and we only need to provide one binary for all Windows platforms (ok we provide several binaries with different build options but it is still a small, manageable number).
>
>> this is
>> what
>> I have in my notes (also from the website docs):
>>
>> mod_jk.so build
>>     yum install httpd-devel
>>     download tomcat-connectors-x.x.xx-src.tar.gz
>>     extract and cd into the native folder
>>     cd /opt/apache-tomcat/tomcat-connectors-x.x.xx-src/native
>>     ./configure --with-apxs=/usr/bin/apxs
>>     make
>>     copy file ./apache-2.0/mod_jk.so to /etc/httpd/modules
>>     make clean
>>
>> (I only have a pretty much standard workers.properties so can't help
>> much.)
>>
>> On Sun, 21 Oct 2018 at 02:01, Jerry Malcolm <[hidden email]>
>> wrote:
>>
>>> Thanks for the responses to my earlier post about websockets/JSPs.
>>> Http/2 is definitely the right answer for my issue there.
>>>
>>> HTTP/2 solved the problem with the throttling the multiple parallel
>>> requests with the browser connection limits.  But now opening that
>>> throttle has now exposed a different problem at the server.  Quick
>>> refresh.... I have an app that does not have many concurrent users at
>>> this time.  However it does periodically have a very large number of
>>> concurrent requests (AJAX/REST) from a single client.
>>>
>>> I now realize that with the browser limiting concurrent requests in
>>> HTTP/1.1, mod_jk was not getting overwhelmed.  But now that the
>>> floodgates have opened with HTTP/2, I realize that I have some
>> serious
>>> configuration problems with mod_jk.  The requests are clogging up in
>>> mod_jk, timing out, and logging errors something like "cannot get
>>> endpoint for worker".  It then takes about 5 minutes for the clog to
>>> clear out before mod_jk is functional again for the other virtual
>> hosts
>>> I'm hosting.
>>>
>>> Specifics... I'm using Apache 2.4.33, Tomcat 9.0.  I tried to
>> download
>>> the latest mod_jk.so.  The latest mod_jk for httpd on the binaries
>>> download site is from 2016.  All the versions for the past 2 years
>> are
>>> iis only... I assume that's ok???  So I'm using the 2016 1.2.40
>> version.
>>>
>>> My question is.... how do I increase the number of 'endpoints' that
>>> mod_jk has access to?  Is that a TC configuration or
>> workers.properties
>>> setting?  I want to make sure mod_jk has enough resources/endpoints
>> to
>>> pass (i.e.not 'block') all the requests it receives.  Once mod_jk is
>>> passing everything, I realize the flood might hit TC, and I may be
>> back
>>> asking about increasing TC resources.  But for now, I just want to
>> get
>>> mod_jk opened up.
>
> We need to see the configuration for:
>
> - httpd mpm / threads / processes
> - workers.proprties
> - Tomcat's AJP connector

Plus:

mod_http2 uses additional threads per Apache httpd process. Look for
H2Workers, H2MinWorkers and H2MaxWorkers in the mod_http2 docs. By
default it seems you get ThreadsPerChild normal http worker threads plus
ThreadsPerChild HTTP/2 worker threads.

So the old-style assumption of mod_jk, that it is enough to provide as
many backend connections per httpd process, as normal httpd worker
threads are configured (ThreadsPerChild), does no longer apply for HTTP/2.

You could try increasing the mod_jk pool size (connection_pool_size) to
2*ThreadsPerChild, but you might also want to consider switching to
mod_proxy.

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

Jerry Malcolm
In reply to this post by markt
On 10/21/2018 3:17 AM, Mark Thomas wrote:
> We need to see the configuration for:
> - httpd mpm / threads / processes
> - workers.proprties
> - Tomcat's AJP connector
>
> Mark
>
I'm not sure where to find httpd mpm / threads / processes config.
Nothing like that in my httpd.conf

----------- workers.properties:
worker.list=jk-status
worker.jk-status.type=status
worker.jk-status.read_only=true
worker.list=jk-manager
worker.jk-manager.type=status
worker.list=worker1
# Set properties for worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=50
worker.worker1.socket_keepalive=1
worker.worker1.socket_timeout=300
worker.worker1.reply_timeout=12000000

-------------- httpd (JK section)
LoadModule jk_module modules/mod_jk.so
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkShmFile logs/mod_jk.shm
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

Jerry Malcolm
In reply to this post by Rainer Jung-3
On 10/21/2018 7:27 AM, Rainer Jung wrote:

> Plus:
>
> mod_http2 uses additional threads per Apache httpd process. Look for
> H2Workers, H2MinWorkers and H2MaxWorkers in the mod_http2 docs. By
> default it seems you get ThreadsPerChild normal http worker threads
> plus ThreadsPerChild HTTP/2 worker threads.
>
> So the old-style assumption of mod_jk, that it is enough to provide as
> many backend connections per httpd process, as normal httpd worker
> threads are configured (ThreadsPerChild), does no longer apply for
> HTTP/2.
>
> You could try increasing the mod_jk pool size (connection_pool_size)
> to 2*ThreadsPerChild, but you might also want to consider switching to
> mod_proxy.
>
> Regards,
>
> Rainer
>
Rainer,

I've done some reading up on mpm, http/2, etc.  But it's given me more
questions than answers.  I need a bit of education help sorting this out.

Back to the original problem.  I need to understand these mod_jk log
entries (there are hundreds of them...):

[Sun Oct 21 17:15:19 2018] [10116:5456] [warn]
ajp_get_endpoint::jk_ajp_common.c (3229): Unable to get the free
endpoint for worker worker1 from 64 slots
[Sun Oct 21 17:15:19 2018] [10116:5456] [error] jk_handler::mod_jk.c
(2744): Could not get endpoint for worker=worker1
[Sun Oct 21 17:15:19 2018] [10116:5456] [info] jk_handler::mod_jk.c
(2806): Service error=0 for worker=worker1

a) what does a 'free endpoint' represent?  Is it an httpd thread that it
can't get?  Is it a connection to Tomcat?  Something else?

b) it says it has 64 of whatever it's looking for.... what is a 'slot'?

I currently have no mpm enabled.  I am running on WAMP (May2018
download).   From what I can tell after much digging, Apache Windows
does not use prefork, worker, or event mpm and rather has it's own WinNT
mpm.  The default config file has only two parms for WinNT. But this mpm
config file is commented out by default.  I'll enable it.

<IfModule mpm_winnt_module>
     ThreadsPerChild        150
     MaxConnectionsPerChild   0
</IfModule>

You suggested increasing connection_pool_size.  Is there a way to see
what pool size is set to by default?  The docs talk about iis, iPlanet,
etc defaults.  But it's not clear what it's set to for my default
config.  Is there some kind of config dump option for mod_jk?

You also suggested moving to mod_proxy.  You really lost me there. I
found info on how to config proxy, etc.  But I couldn't find anything
about how proxy differs from non-proxy.  What problem does it solve? 
And how would it potentially fix my issue?

Sorry for all the questions.  But I'm very concerned about just shooting
in the dark making all kinds of changes when I really don't know if and
how it's helping or hurting.  Any education to help me understand what
the fundamental problem is with 'endpoints' in the mod_jk log will be
greatly appreciated.

Thx.

Jerry


Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

Jerry Malcolm
In reply to this post by markt
On 10/21/2018 3:17 AM, Mark Thomas wrote:
>
> We provide the Windows binaries because the build process is more involved, most users don't have the necessary toolchain  easily available and we only need to provide one binary for all Windows platforms (ok we provide several binaries with different build options but it is still a small, manageable number).
>
Mark,

Does you mean that I can use the Windows binary dll that is in the
mod_jk iis binaries download if I'm running Apache and Tomcat on a Win
Server platform instead of using mod_jk.so?

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

Rainer Jung-3
In reply to this post by Jerry Malcolm
Am 22.10.2018 um 00:43 schrieb Jerry Malcolm:

> On 10/21/2018 7:27 AM, Rainer Jung wrote:
>> Plus:
>>
>> mod_http2 uses additional threads per Apache httpd process. Look for
>> H2Workers, H2MinWorkers and H2MaxWorkers in the mod_http2 docs. By
>> default it seems you get ThreadsPerChild normal http worker threads
>> plus ThreadsPerChild HTTP/2 worker threads.
>>
>> So the old-style assumption of mod_jk, that it is enough to provide as
>> many backend connections per httpd process, as normal httpd worker
>> threads are configured (ThreadsPerChild), does no longer apply for
>> HTTP/2.
>>
>> You could try increasing the mod_jk pool size (connection_pool_size)
>> to 2*ThreadsPerChild, but you might also want to consider switching to
>> mod_proxy.
>>
>> Regards,
>>
>> Rainer
>>
> Rainer,
>
> I've done some reading up on mpm, http/2, etc.  But it's given me more
> questions than answers.  I need a bit of education help sorting this out.
>
> Back to the original problem.  I need to understand these mod_jk log
> entries (there are hundreds of them...):
>
> [Sun Oct 21 17:15:19 2018] [10116:5456] [warn]
> ajp_get_endpoint::jk_ajp_common.c (3229): Unable to get the free
> endpoint for worker worker1 from 64 slots

That means the size of the per process connection pool used to connect
to Tomcat "worker1" is 64 and all 64 connections were busy when another
one was needed. Probably this 64 was not configured by you but instead
auto-sized via ThreadsPerChild (MPM sizing). From what I understand
about the HTTP/2 implementation by mod_http2, you should try in doubling
the number of endpoints by setting
worker.worker1.connection_pool_size=128 in workers.properties without
changing ThreadsPerChild. Some info on the param can be found at
http://tomcat.apache.org/connectors-doc/reference/workers.html.

What you need to keep in mind: it is also possible, that the real
culprit is, that already 64 connections from one Apache process to the
backend were busy. That could mean, that the requests became too slow in
Tomcat or more likely in your application running in Tomcat. So I would
suggest you also take thread dumps, in this case of Tomcat worker1, when
the above message happens frequently to find out, what those 64
concurrent requests are doing (that use up all 64 possible connections).

> [Sun Oct 21 17:15:19 2018] [10116:5456] [error] jk_handler::mod_jk.c
> (2744): Could not get endpoint for worker=worker1
> [Sun Oct 21 17:15:19 2018] [10116:5456] [info] jk_handler::mod_jk.c
> (2806): Service error=0 for worker=worker1

These two are just follow-up messages of the above.

> a) what does a 'free endpoint' represent?  Is it an httpd thread that it
> can't get?  Is it a connection to Tomcat?  Something else?

Connection from a specific Apache httpd process to Tomcat. These
endpoints=connections are pooled per Apache process. The number of
connections can be configured via worker.WORKER_NAME.connection_pool_size.

> b) it says it has 64 of whatever it's looking for.... what is a 'slot'?

Just a synonym for endpoint. By slot in that message is meant an item in
the pool.

> I currently have no mpm enabled.  I am running on WAMP (May2018
> download).   From what I can tell after much digging, Apache Windows
> does not use prefork, worker, or event mpm and rather has it's own WinNT
> mpm.  The default config file has only two parms for WinNT. But this mpm
> config file is commented out by default.  I'll enable it.

Ah, Windows. So unlike Unix/Linux we do only have a single child
process, which does all the work. The number 64 is the default number of
threads in this child for Windows.

> <IfModule mpm_winnt_module>
>      ThreadsPerChild        150
>      MaxConnectionsPerChild   0
> </IfModule>

OK, after that you have 150 threads in the one and only Apache httpd
child process, that does all the work. mod_jk will automatically also
set the number of endpoints (slots) = the connection_pool_size to 150,
if not configured by you otherwise.

> You suggested increasing connection_pool_size.  Is there a way to see
> what pool size is set to by default?  The docs talk about iis, iPlanet,
> etc defaults.  But it's not clear what it's set to for my default
> config.  Is there some kind of config dump option for mod_jk?

Configure a status worker. It has plenty of interesting info. For
monitoring it is enough to configure a read-only status worker. See
http://tomcat.apache.org/connectors-doc/reference/status.html.

> You also suggested moving to mod_proxy.  You really lost me there. I
> found info on how to config proxy, etc.  But I couldn't find anything
> about how proxy differs from non-proxy.  What problem does it solve? And
> how would it potentially fix my issue?

mod_jk and mod_proxy are simply two different ways of connecting Apache
httpd to backend Tomcats. mod_jk uses a special protocol named AJP13
(and an AJP connector in Tomcat which understands that protocol).
mod_proxy simply uses HTTP or HTTPS and the normal HTTP/HTTPS connector
in Tomcat. mod_jk is provided by the Tomcat project and configured
mostly via workers.properties, mod_proxy is provided as part of Apache
httpd and configured via "normal" Proxy... httpd directives.

> Sorry for all the questions.  But I'm very concerned about just shooting
> in the dark making all kinds of changes when I really don't know if and
> how it's helping or hurting.  Any education to help me understand what
> the fundamental problem is with 'endpoints' in the mod_jk log will be
> greatly appreciated.

Now that I understand that you are on Windows, the default size of 64
woker threads in total for Apache httpd is not that big. So i suggest

- enable the explicit config with 150 as you already did

- increase connection_pool_size to 300 if you are actually using HTTP/2

- enable a read-only status worker for monitoring. The number of busy
endpoints can be seen under the busy column, the max number since start
or statistics reset under max_busy. There are also the number of
connections and max number of connections since start/statistics reset.
The difference between busy and number of connections is, that a
connection is counted in busy only, if it is currently being used for a
request/response.

- If you are till getting errors take a few thread dumps of your backend
Tomcats during the time you get errors and provide them to the mailing
list, possibly after redacting things there, that you don't want public.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: mod_jk endpoints

Rainer Jung-3
In reply to this post by Jerry Malcolm
Am 22.10.2018 um 00:53 schrieb Jerry Malcolm:

> On 10/21/2018 3:17 AM, Mark Thomas wrote:
>>
>> We provide the Windows binaries because the build process is more
>> involved, most users don't have the necessary toolchain  easily
>> available and we only need to provide one binary for all Windows
>> platforms (ok we provide several binaries with different build options
>> but it is still a small, manageable number).
>>
> Mark,
>
> Does you mean that I can use the Windows binary dll that is in the
> mod_jk iis binaries download if I'm running Apache and Tomcat on a Win
> Server platform instead of using mod_jk.so?

No, IIS binaries can not be used as module binaries for Apache httpd on
Windows. But you typically get module binaries from the same sources
that provide Apache httpd on Windows for you. The problem there is, that
the module and the web server must be build with compatibel compiler
setups. Therefore it is best to get them from the same providers (like
Apache Lounge or others).

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]