Failed allocating AJP message buffer

[Pierre Ayotte]

Hello everyone,

We are experimenting HTTD.EXE crash on Windows 2008 R2 x64 and each time we
see a bunch of error "Failed allocating AJP message buffer" and then one
message "All tomcat instances failed, no more workers left" (see below).
The HTTPD.EXE recylcle itself, users are disconnect and then the service
come back online. I did lot of search on those message but i cannot find
any solution. Some help would really be appreciated! See below for more
details

Configuration summary: Apache 2 running mod_ssl + mod_jk load balancing 5
tomcat6 instance.

The mod_jk.log
==============
(.... lots of this AJP message buffer error ....)
[Wed Jun 20 14:59:01.688 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.704 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.735 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.750 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.766 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.891 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2393): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.922 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2393): Failed allocating AJP message buffer
[Wed Jun 20 14:59:01.938 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.673 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.688 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.704 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.704 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.719 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.829 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.844 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.860 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.876 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.876 2012] [18324:12832] [error]
service::jk_lb_worker.c (1177): Failed allocating AJP message buffer of
8192 bytes.
[Wed Jun 20 14:59:08.891 2012] [18324:9768] [error]
ajp_service::jk_ajp_common.c (2407): Failed allocating AJP message buffer
[Wed Jun 20 14:59:08.891 2012] [18324:9768] [error] service::jk_lb_worker.c
(1554): All tomcat instances failed, no more workers left

We also see this error in Windows Event Log:
==============================
====
Faulting application name: httpd.exe, version: 2.2.22.0, time stamp:
0x4f242d7a
Faulting module name: libapr-1.dll, version: 1.4.5.0, time stamp: 0x4f242c33
Exception code: 0xc0000005
Fault offset: 0x00004bd8
Faulting process id: 0x6050
Faulting application start time: 0x01cd4ed4e838b9b3
Faulting application path: D:\Program Files (x86)\Apache Software
Foundation\Apache2.2\bin\httpd.exe
Faulting module path: D:\Program Files (x86)\Apache Software
Foundation\Apache2.2\bin\libapr-1.dll
Report Id: c450d6d3-bb0d-11e1-905f-005056b40061

Our Configuration
=============
Apache
     Apache/2.2.22 on Windows 2008 R2 x64
     mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in place
     mod_jk/1.2.35 with a load balancer to 5 workers

   Worker.Properties


worker.list=loadbalancer,status,manager1,manager2,manager3,manager4,manager5
   worker.V6_SMInstance1.port=8109
   worker.V6_SMInstance1.host=localhost
   worker.V6_SMInstance1.type=ajp13
   worker.V6_SMInstance1.lbfactor=1

   worker.V6_SMInstance2.port=8209
   worker.V6_SMInstance2.host=localhost
   worker.V6_SMInstance2.type=ajp13
   worker.V6_SMInstance2.lbfactor=1

   worker.V6_SMInstance3.port=8309
   worker.V6_SMInstance3.host=localhost
   worker.V6_SMInstance3.type=ajp13
   worker.V6_SMInstance3.lbfactor=1

   worker.V6_SMInstance4.port=8409
   worker.V6_SMInstance4.host=localhost
   worker.V6_SMInstance4.type=ajp13
   worker.V6_SMInstance4.lbfactor=1

   worker.V6_SMInstance5.port=8509
   worker.V6_SMInstance5.host=localhost
   worker.V6_SMInstance5.type=ajp13
   worker.V6_SMInstance5.lbfactor=1

   # Status worker for managing load balancer
   worker.status.type=status

   worker.loadbalancer.type=lb
   worker.loadbalancer.sticky_session=1

worker.loadbalancer.balance_workers=V6_SMInstance1,V6_SMInstance2,V6_SMInstance3,V6_SMInstance4,V6_SMInstance5

   worker.manager1.port=8109
   worker.manager1.host=localhost
   worker.manager1.type=ajp13

   worker.manager2.port=8209
   worker.manager2.host=localhost
   worker.manager2.type=ajp13

   worker.manager3.port=8309
   worker.manager3.host=localhost
   worker.manager3.type=ajp13

   worker.manager4.port=8409
   worker.manager4.host=localhost
   worker.manager4.type=ajp13

   worker.manager5.port=8509
   worker.manager5.host=localhost
   worker.manager5.type=ajp13

Tomcat
=============
   Tomcat 6.0.32
   5 instance of our HP Service Manager application


Regards,
Pierre Ayotte

[Rainer Jung-3]

On 21.06.2012 00:26, Pierre Ayotte wrote:
> Hello everyone,
>
> We are experimenting HTTD.EXE crash on Windows 2008 R2 x64 and each time we
> see a bunch of error "Failed allocating AJP message buffer" and then one
> message "All tomcat instances failed, no more workers left" (see below).
> The HTTPD.EXE recylcle itself, users are disconnect and then the service
> come back online. I did lot of search on those message but i cannot find
> any solution. Some help would really be appreciated!

It seems you are out of memory. Either

- there's some leak in your Apache, most probably in some non standard
module

- or your server is very tight on memory

- or your application inside Tomcat is getting slow, so there are more
and more requests getting stuck inside Apache waiting for a response
from the web app and those requests consume to much memory.

There are MPM settings for Apache on Windows that limit the number of
threads (=connections =~ concurrent requests) Apache will allow.

Monitor your server memory usage.

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Pierre Ayotte]

Hello Rainer

I already check for memory and for sure it is not a physical memory issu.
We have 12Go and the memory usage is around 6Go at the moment of the crash.

Pierre

2012/6/21 Rainer Jung <[hidden email]>

> On 21.06.2012 00:26, Pierre Ayotte wrote:
>
>> Hello everyone,
>>
>> We are experimenting HTTD.EXE crash on Windows 2008 R2 x64 and each time
>> we
>> see a bunch of error "Failed allocating AJP message buffer" and then one
>> message "All tomcat instances failed, no more workers left" (see below).
>> The HTTPD.EXE recylcle itself, users are disconnect and then the service
>> come back online. I did lot of search on those message but i cannot find
>> any solution. Some help would really be appreciated!
>>
>
> It seems you are out of memory. Either
>
> - there's some leak in your Apache, most probably in some non standard
> module
>
> - or your server is very tight on memory
>
> - or your application inside Tomcat is getting slow, so there are more and
> more requests getting stuck inside Apache waiting for a response from the
> web app and those requests consume to much memory.
>
> There are MPM settings for Apache on Windows that limit the number of
> threads (=connections =~ concurrent requests) Apache will allow.
>
> Monitor your server memory usage.
>
> Regards,
>
> Rainer
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.org<[hidden email]>
> For additional commands, e-mail: [hidden email]
>
>

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pierre,

On 6/20/12 6:26 PM, Pierre Ayotte wrote:
> Our Configuration ============= Apache Apache/2.2.22 on Windows
> 2008 R2 x64 mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in
> place mod_jk/1.2.35 with a load balancer to 5 workers

There were a bunch of fixes in mod_jk 1.2.36 and 1.2.37. Is it
possible to test with the latest version?

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/jkUwACgkQ9CaO5/Lv0PD7ewCcCMNw8g2O5Ejv9CDYhIcWmAPK
PzUAnA8RIWsETMPD9yTIVl9BQaqEo6Yu
=chWd
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Pierre Ayotte]

Hello Christopher,

Yes we saw on wednesday taht the 1.2.37 has been release sot we just put
the it in production yesterday night. All keep you inform.

We also make thoses changes to httpd.conf base on recommendation from a
collegue:

- ThreadsPerChild down to 1000 instead of 3000
- Remove ThreadLimit 9000 because apparently this does not serve on Windows

Do you think this could help?

2012/6/21 Christopher Schultz <[hidden email]>

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Pierre,
>
> On 6/20/12 6:26 PM, Pierre Ayotte wrote:
> > Our Configuration ============= Apache Apache/2.2.22 on Windows
> > 2008 R2 x64 mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in
> > place mod_jk/1.2.35 with a load balancer to 5 workers
>
> There were a bunch of fixes in mod_jk 1.2.36 and 1.2.37. Is it
> possible to test with the latest version?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk/jkUwACgkQ9CaO5/Lv0PD7ewCcCMNw8g2O5Ejv9CDYhIcWmAPK
> PzUAnA8RIWsETMPD9yTIVl9BQaqEo6Yu
> =chWd
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

[Pierre Ayotte]

Hi everyone,

Just to let you know that installing the latest release of mod_jk resolved
our issue with http crashing with errors "Failed allocating AJP message
buffer" in mod_jk.log. So we pass from 1.2.35 to 1.2.37.

Regards,


2012/6/22 Pierre Ayotte <[hidden email]>

> Hello Christopher,
>
> Yes we saw on wednesday taht the 1.2.37 has been release sot we just put
> the it in production yesterday night. All keep you inform.
>
> We also make thoses changes to httpd.conf base on recommendation from a
> collegue:
>
> - ThreadsPerChild down to 1000 instead of 3000
> - Remove ThreadLimit 9000 because apparently this does not serve on Windows
>
> Do you think this could help?
>
>
> 2012/6/21 Christopher Schultz <[hidden email]>
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Pierre,
>>
>> On 6/20/12 6:26 PM, Pierre Ayotte wrote:
>> > Our Configuration ============= Apache Apache/2.2.22 on Windows
>> > 2008 R2 x64 mod_ssl/2.2.22 OpenSSL/0.9.8t with certificates in
>> > place mod_jk/1.2.35 with a load balancer to 5 workers
>>
>> There were a bunch of fixes in mod_jk 1.2.36 and 1.2.37. Is it
>> possible to test with the latest version?
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAk/jkUwACgkQ9CaO5/Lv0PD7ewCcCMNw8g2O5Ejv9CDYhIcWmAPK
>> PzUAnA8RIWsETMPD9yTIVl9BQaqEo6Yu
>> =chWd
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pierre,

On 6/27/12 11:36 AM, Pierre Ayotte wrote:
> Just to let you know that installing the latest release of mod_jk
> resolved our issue with http crashing with errors "Failed
> allocating AJP message buffer" in mod_jk.log. So we pass from
> 1.2.35 to 1.2.37.

Good to know. There were a lot of changes to the win32 code around
1.2.35 and some of them caused all sorts of problems. I believe most
of them have been worked out and 1.2.37 should be much more stable for
you.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/rTD4ACgkQ9CaO5/Lv0PAF3gCeJKopRyticr0nah2OQ91svdrh
erwAnAoiFbupvuWaDLDJMw9kAjQAM9nU
=QBCI
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]