Quantcast

IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Konstantin Preißer-2
Hi all,

I have a system with Windows Server 2008 32 bit, IIS 7.0, Java 1.7.0_03, Tomcat 7.0.26 and the ISAPI redirector.

Since I updated the ISAPI redirector from 1.2.32 to 1.2.33, it seems that each time when IIS tries to recycle its application pool, the IIS worker process (w3wp.exe) crashes in isapi_redirect.dll. In the event log, an "Application Error" of w3wp.exe is logged (here is an English translation):

Faulty application w3wp.exe, Version 7.0.6002.18005, time stamp 0x49e023cf, faulty module isapi_redirect.dll, Version 1.2.33.0, time stamp 0x4f59be7d, exception code 0xc0000005, error offset 0x0002bb16, process ID 0x10f0, application start time 01cd0336e5b7824a.
(The process ID does not match the one of w3wp.exe before the crash, nor the one after the crash, so it seems a bit like when the new w3wp.exe is launched after recycle, it crashes and immediately another w3wp.exe is started).


In the ISAPI log, following lines appear when w3wp.exe crashes:
[Fri Mar 16 06:37:38.402 2012] [4336:6828] [error] ajp_worker_factory::jk_ajp_common.c (3006): allocating ajp worker record from shared memory
[Fri Mar 16 06:37:38.417 2012] [4336:6828] [error] wc_create_worker::jk_worker.c (150): factory for ajp13 failed for worker1
[Fri Mar 16 06:37:38.426 2012] [4336:6828] [error] build_worker_map::jk_worker.c (261): failed to create worker worker1
[Fri Mar 16 06:37:38.434 2012] [4336:6828] [error] extension_fix::jk_uri_worker_map.c (554): Could not find worker with name 'worker1' in uri map post processing.

Any idea what these lines could mean / that caused them? Note that it seems that after the crash (when a new w3wp.exe is created), pages are served fine again (though I do not know what happens with request that are made exactly in the time when IIS resets the app pool).
The crashes and these log lines didn't appear in ISAPI 1.2.32.

I think I read somewhere in the thread "Issues with the tomcat connector (On W2k8 + IIS7.5)" about some change in the ISAPI connector which has to do with shared memory, but I'm not sure.

Thanks!


Regards,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Konstantin Preißer-2
>
> Any idea what these lines could mean / that caused them? Note that it
> seems that after the crash (when a new w3wp.exe is created), pages are
> served fine again (though I do not know what happens with request that
> are made exactly in the time when IIS resets the app pool).
> The crashes and these log lines didn't appear in ISAPI 1.2.32.
>

Sorry, forgot to post my configuration:

uriworkersmap.properties:

  /*=worker1


workers.properties:

  # Define 1 real worker using ajp13
  worker.list=worker1
  # Set properties for worker1 (ajp13)
  worker.worker1.type=ajp13
  worker.worker1.host=localhost
  worker.worker1.port=8019


Thanks,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Alex Samad - Yieldbroker


> -----Original Message-----
> From: Konstantin Preißer [mailto:[hidden email]]
> Sent: Saturday, 17 March 2012 2:31 AM
> To: 'Tomcat Users List'
> Subject: RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI
> redirector 1.2.33
>
> >
> > Any idea what these lines could mean / that caused them? Note that it
> > seems that after the crash (when a new w3wp.exe is created), pages are
> > served fine again (though I do not know what happens with request that
> > are made exactly in the time when IIS resets the app pool).
> > The crashes and these log lines didn't appear in ISAPI 1.2.32.

Hi

1.2.32 was doing that for me.. 1.2.33 was meant to fix it, but I can't get 1.2.33 to load.

Do you use overlapping recycle and a web garden ?

Alex

> >
>
> Sorry, forgot to post my configuration:
>
> uriworkersmap.properties:
>
>   /*=worker1
>
>
> workers.properties:
>
>   # Define 1 real worker using ajp13
>   worker.list=worker1
>   # Set properties for worker1 (ajp13)
>   worker.worker1.type=ajp13
>   worker.worker1.host=localhost
>   worker.worker1.port=8019
>
>
> Thanks,
> Konstantin Preißer
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Konstantin Preißer-2
Hi Alex,

> -----Original Message-----
> From: Alex Samad - Yieldbroker [mailto:[hidden email]]
> Sent: Friday, March 16, 2012 5:23 PM
> To: Tomcat Users List
> Subject: RE: IIS 7.0 Worker process crashes on App Pool recycling since
> ISAPI redirector 1.2.33
>
> Hi
>
> 1.2.32 was doing that for me.. 1.2.33 was meant to fix it, but I can't
> get 1.2.33 to load.
>
> Do you use overlapping recycle and a web garden ?
>
> Alex
>

Thanks for your reply.
I don't think I'm using a web garden - I just have one application pool for all virtual hosts which use the ISAPI redirector, and that application pool consists of a maximum of 1 worker process (the default IIS values). It is set to be recycled every 1740 minutes, and "disallowOverlappingRotation" for that pool is set to "false" - I guess that means I'm using overlapping recycle (a new w3wp.exe is started which takes new requests, and after all old requests are finished, the old w3wp.exe will be stopped).

Thanks,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Alex Samad - Yieldbroker

[snip]

> >
> > Hi
> >
> > 1.2.32 was doing that for me.. 1.2.33 was meant to fix it, but I can't
> > get 1.2.33 to load.
> >
> > Do you use overlapping recycle and a web garden ?
> >
> > Alex
> >
>
> Thanks for your reply.
> I don't think I'm using a web garden - I just have one application pool for all
> virtual hosts which use the ISAPI redirector, and that application pool consists
> of a maximum of 1 worker process (the default IIS values). It is set to be
> recycled every 1740 minutes, and "disallowOverlappingRotation" for that
> pool is set to "false" - I guess that means I'm using overlapping recycle (a new
> w3wp.exe is started which takes new requests, and after all old requests are
> finished, the old w3wp.exe will be stopped).

The 1 processor thread is the key.  The problem I summarised with 1.2.32 is that the shared memory is not protected by a OS semaphore, but an in process semaphore...  1.2.33 was mean to address this by changing from an inprocess semaphore to a OS  semaphore.

I am surprised  you have 1.2.33 loading and working, every time I try to load it crashes out on me.

Only quick solution I can think is that you move back to 1.2.32 :)

You might want to trial the system under load. I found that if you had about 500 connections all try and reconnect at the same time with overlapping recycling it would corrupt the sharememory it was a bit of a silent killer for us.

>
> Thanks,
> Konstantin Preißer
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
In reply to this post by Konstantin Preißer-2
On 03/16/2012 04:27 PM, Konstantin Preißer wrote:
> Hi all,
>
> I have a system with Windows Server 2008 32 bit, IIS 7.0, Java 1.7.0_03, Tomcat 7.0.26 and the ISAPI redirector.
>

Guys,

Please try the binaries from:
http://people.apache.org/~mturk/tomcat-connectors/jk-1.2.34/

They contain two fixes:
1. Make sure we fallback to heap memory in case shared cannot be created
2. Fix shared memory create/open arguments

The point is that is that we should have correct [error] log
entries in case shared memory open fails.
If it fails you should have [warn] line, load balancer will not
function properly across multiple processes (will inside each of the process)
but it shouldn't crash.

I'd appreciate if you can check that ASAP cause we have regression in httpd
implementation so new version will be out in couple of days.


Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Konstantin Preißer-2
Hello Mladen,

> -----Original Message-----
> From: Mladen Turk [mailto:[hidden email]]
> Sent: Sunday, March 18, 2012 8:46 AM
> To: [hidden email]
> Cc: [hidden email]; [hidden email]
> Subject: Re: IIS 7.0 Worker process crashes on App Pool recycling since
> ISAPI redirector 1.2.33
>
>
> Guys,
>
> Please try the binaries from:
> http://people.apache.org/~mturk/tomcat-connectors/jk-1.2.34/
>
> They contain two fixes:
> 1. Make sure we fallback to heap memory in case shared cannot be
> created
> 2. Fix shared memory create/open arguments
>
> The point is that is that we should have correct [error] log
> entries in case shared memory open fails.
> If it fails you should have [warn] line, load balancer will not
> function properly across multiple processes (will inside each of the
> process)
> but it shouldn't crash.
>

Thank you very much. I tried the new version, and now when the Application Pool is recycled, the logs show these:

[Sun Mar 18 12:10:36.324 2012] [4144:6216] [error] jk_shm_open::jk_shm.c (220): Failed to map shared memory JK_TTWEBINFO_DYNDNS_ORG_8_JAKARTA2_ISAPI_REDIRECT with errno=87
[Sun Mar 18 12:10:36.337 2012] [4144:6216] [error] init_jk::jk_isapi_plugin.c (2801): Initializing shm:(null) errno=87. Load balancer will not work properly!

But w3wp.exe doesn't crash anymore, so I think it is fixed. (I'm wondering now why mapping the shared memory fails..)


Thanks,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
On 03/18/2012 12:16 PM, Konstantin Preißer wrote:

> Hello Mladen,
>
>
> Thank you very much. I tried the new version, and now when the Application Pool is recycled, the logs show these:
>
> [Sun Mar 18 12:10:36.324 2012] [4144:6216] [error] jk_shm_open::jk_shm.c (220): Failed to map shared memory JK_TTWEBINFO_DYNDNS_ORG_8_JAKARTA2_ISAPI_REDIRECT with errno=87
> [Sun Mar 18 12:10:36.337 2012] [4144:6216] [error] init_jk::jk_isapi_plugin.c (2801): Initializing shm:(null) errno=87. Load balancer will not work properly!
>
> But w3wp.exe doesn't crash anymore, so I think it is fixed. (I'm wondering now why mapping the shared memory fails..)
>

Thanks for looking at this promptly!

Could you try with the new builds
http://people.apache.org/~mturk/tomcat-connectors/jk-1.2.34/
(isapi_redirect_1.2.34_2-dev_winXX.zip)

I added Global\\ prefix to shared memory name so that it can
be shared from multiple processes.
Also check if the errno changed from 87 (ERROR_INVALID_PARAMETER)



Thanks
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
On 03/18/2012 02:34 PM, Mladen Turk wrote:

> On 03/18/2012 12:16 PM, Konstantin Preißer wrote:
>> Hello Mladen,
>>
>>
>> Thank you very much. I tried the new version, and now when the Application Pool is recycled, the logs show these:
>>
>> [Sun Mar 18 12:10:36.324 2012] [4144:6216] [error] jk_shm_open::jk_shm.c (220): Failed to map shared memory JK_TTWEBINFO_DYNDNS_ORG_8_JAKARTA2_ISAPI_REDIRECT with errno=87
>> [Sun Mar 18 12:10:36.337 2012] [4144:6216] [error] init_jk::jk_isapi_plugin.c (2801): Initializing shm:(null) errno=87. Load balancer will not work properly!
>>
>> But w3wp.exe doesn't crash anymore, so I think it is fixed. (I'm wondering now why mapping the shared memory fails..)
>>
>
> Thanks for looking at this promptly!
>
> Could you try with the new builds
> http://people.apache.org/~mturk/tomcat-connectors/jk-1.2.34/
> (isapi_redirect_1.2.34_2-dev_winXX.zip)
>

Forget about that one. I'll create a new one in 20 minutes


Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
On 03/18/2012 02:40 PM, Mladen Turk wrote:

> On 03/18/2012 02:34 PM, Mladen Turk wrote:
>> On 03/18/2012 12:16 PM, Konstantin Preißer wrote:
>>> Hello Mladen,
>>>
>>>
>>> Thank you very much. I tried the new version, and now when the Application Pool is recycled, the logs show these:
>>>
>>> [Sun Mar 18 12:10:36.324 2012] [4144:6216] [error] jk_shm_open::jk_shm.c (220): Failed to map shared memory JK_TTWEBINFO_DYNDNS_ORG_8_JAKARTA2_ISAPI_REDIRECT with errno=87
>>> [Sun Mar 18 12:10:36.337 2012] [4144:6216] [error] init_jk::jk_isapi_plugin.c (2801): Initializing shm:(null) errno=87. Load balancer will not work properly!
>>>
>>> But w3wp.exe doesn't crash anymore, so I think it is fixed. (I'm wondering now why mapping the shared memory fails..)
>>>
>>
>> Thanks for looking at this promptly!
>>
>> Could you try with the new builds
>> http://people.apache.org/~mturk/tomcat-connectors/jk-1.2.34/
>> (isapi_redirect_1.2.34_2-dev_winXX.zip)
>>
>

Please use isapi_redirect_1.2.34_3-dev_winXX.zip instead.

Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Konstantin Preißer-2
Hi Mladen,

> -----Original Message-----
> From: Mladen Turk [mailto:[hidden email]]
> Sent: Sunday, March 18, 2012 2:54 PM
> To: Tomcat Users List
> Subject: Re: IIS 7.0 Worker process crashes on App Pool recycling since
> ISAPI redirector 1.2.33
>
>
> Please use isapi_redirect_1.2.34_3-dev_winXX.zip instead.

Thank you.

I tried with isapi_redirect_1.2.34_3-dev_win32.zip, but unfortunately it seems that it causes w3wp.exe to crash again :(

Logs:
[Sun Mar 18 15:05:48.648 2012] [7588:5124] [error] ajp_worker_factory::jk_ajp_common.c (3006): allocating ajp worker record from shared memory
[Sun Mar 18 15:05:48.661 2012] [7588:5124] [error] wc_create_worker::jk_worker.c (150): factory for ajp13 failed for worker1
[Sun Mar 18 15:05:48.669 2012] [7588:5124] [error] build_worker_map::jk_worker.c (261): failed to create worker worker1
[Sun Mar 18 15:05:48.677 2012] [7588:5124] [error] extension_fix::jk_uri_worker_map.c (554): Could not find worker with name 'worker1' in uri map post processing.


Thanks,
Konstantin Preißer


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
On 03/18/2012 03:11 PM, Konstantin Preißer wrote:

> Hi Mladen,
>
>> -----Original Message-----
>> From: Mladen Turk [mailto:[hidden email]]
>> Sent: Sunday, March 18, 2012 2:54 PM
>> To: Tomcat Users List
>> Subject: Re: IIS 7.0 Worker process crashes on App Pool recycling since
>> ISAPI redirector 1.2.33
>>
>>
>> Please use isapi_redirect_1.2.34_3-dev_winXX.zip instead.
>
> Thank you.
>
> I tried with isapi_redirect_1.2.34_3-dev_win32.zip, but unfortunately it seems that it causes w3wp.exe to crash again :(
>

Could you please try the isapi_redirect_1.2.34_5-dev_win32.zip
If this one crashes we'll have to add some override to shared memory so we can fallback
to heap one and bypass memory corruption which seems to occur at IIS worker recycle.


Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Alex Samad - Yieldbroker
Hi

Guys (Konstantin, Mladen), goo d work, sorry caught me on a Sunday, else I would have chipped in.

Just wondering (thinking a bit left field), but  question was asked of me.  Any reason to have a shm across processes.

We already have threading inside each worker process.  What are the benefits of connection pool sharing across process, compared to just keeping them in process.

Alex

> -----Original Message-----
> From: Mladen Turk [mailto:[hidden email]]
> Sent: Monday, 19 March 2012 4:39 AM
> To: [hidden email]
> Subject: Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI
> redirector 1.2.33
>
> On 03/18/2012 03:11 PM, Konstantin Preißer wrote:
> > Hi Mladen,
> >
> >> -----Original Message-----
> >> From: Mladen Turk [mailto:[hidden email]]
> >> Sent: Sunday, March 18, 2012 2:54 PM
> >> To: Tomcat Users List
> >> Subject: Re: IIS 7.0 Worker process crashes on App Pool recycling
> >> since ISAPI redirector 1.2.33
> >>
> >>
> >> Please use isapi_redirect_1.2.34_3-dev_winXX.zip instead.
> >
> > Thank you.
> >
> > I tried with isapi_redirect_1.2.34_3-dev_win32.zip, but unfortunately
> > it seems that it causes w3wp.exe to crash again :(
> >
>
> Could you please try the isapi_redirect_1.2.34_5-dev_win32.zip
> If this one crashes we'll have to add some override to shared memory so we
> can fallback to heap one and bypass memory corruption which seems to
> occur at IIS worker recycle.
>
>
> Regards
> --
> ^TM
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
On 03/18/2012 08:30 PM, Alex Samad - Yieldbroker wrote:
> Hi
>
> Guys (Konstantin, Mladen), goo d work, sorry caught me on a Sunday, else I would have chipped in.
>
> Just wondering (thinking a bit left field), but  question was asked of me.  Any reason to have a shm across processes.
>

So that worker state is replicated across processes.
Eg, if connection to one backend is broken each process will
have to discover that by itself. Shared memory allows that if
one worker process discovers that a backend is down other won't
have to go trough the lengthy process of discovering that.

Also loadbalancer load factors are calculated across all processes
not just for the current one. So, the benefits are real.

However unlike with httpd, IIS does not offer parent/child
concept so we don't have the shared memory 'controller'.
The ultimate solution would be to start a separate controller
process, but that's probably something for the future.

Even if the current set of patches work, I'm more convinced
we should have a directive to disable shared memory usage, and
think I'm going to add that option for 1.2.34 anyhow.


Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Alex Samad - Yieldbroker
[snip]

> > Just wondering (thinking a bit left field), but  question was asked of me.
> Any reason to have a shm across processes.
> >
>
> So that worker state is replicated across processes.
> Eg, if connection to one backend is broken each process will have to discover
> that by itself. Shared memory allows that if one worker process discovers
> that a backend is down other won't have to go trough the lengthy process of
> discovering that.
>
> Also loadbalancer load factors are calculated across all processes not just for
> the current one. So, the benefits are real.

Not deny that, just thinking out aloud, also been asking myself if there is any reason  to run a 4 process web garden. The only benefit I can see is that if 1 process dies there will be other process to continue processing.

1 process + threads v's X processes + threads


>
> However unlike with httpd, IIS does not offer parent/child concept so we
> don't have the shared memory 'controller'.
> The ultimate solution would be to start a separate controller process, but
> that's probably something for the future.
>
> Even if the current set of patches work, I'm more convinced we should have
> a directive to disable shared memory usage, and think I'm going to add that
> option for 1.2.34 anyhow.

So would that effectively do what I was suggesting above ?


Thanks
>
>
> Regards
> --
> ^TM
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Konstantin Preißer-2
In reply to this post by mturk
-----Original-Nachricht-----

> Von: Mladen Turk <[hidden email]>
> An: [hidden email]
> Betreff: Re: IIS 7.0 Worker process crashes on App Pool recycling
> since ISAPI redirector 1.2.33
> Datum: Sun, 18 Mar 2012 18:39:15 +0100
>
> Could you please try the isapi_redirect_1.2.34_5-dev_win32.zip
> If this one crashes we'll have to add some override to shared memory
> so we can fallback to heap one and bypass memory corruption which
> seems to occur at IIS worker recycle.

Hi Mladen,

thank you, I tried isapi_redirect_1.2.34_5-dev_win32.zip and so far no crashes, and also no warnings/errors in the ISAPI log.
It seems the issue is fixed now.

Thanks,
Konstantin Preißer



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
On 03/18/2012 09:34 PM, [hidden email] wrote:

> -----Original-Nachricht-----
>> Von: Mladen Turk<[hidden email]>
>> An: [hidden email]
>> Betreff: Re: IIS 7.0 Worker process crashes on App Pool recycling
>> since ISAPI redirector 1.2.33
>> Datum: Sun, 18 Mar 2012 18:39:15 +0100
>>
>> Could you please try the isapi_redirect_1.2.34_5-dev_win32.zip
>> If this one crashes we'll have to add some override to shared memory
>> so we can fallback to heap one and bypass memory corruption which
>> seems to occur at IIS worker recycle.
>
> Hi Mladen,
>
> thank you, I tried isapi_redirect_1.2.34_5-dev_win32.zip and so far no crashes, and also no warnings/errors in the ISAPI log.
> It seems the issue is fixed now.
>

Cool. Thanks again for testing those.
We'll release 1.2.34 by the end of this week probably.

Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

Alex Samad - Yieldbroker
Hi

Sorry it failed for me, started up, but I got some error messages

I have 4 processes setup for non-overlapping recycle

Log 1
Log 2
[Mon Mar 19 09:43:13.970 2012] [2096:2984] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:13.986 2012] [2096:2984] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500
[Mon Mar 19 09:43:14.079 2012] [2096:2984] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:14.079 2012] [2096:2984] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500
[Mon Mar 19 09:43:53.453 2012] [2096:2288] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:53.453 2012] [2096:2288] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500

Log3
[Mon Mar 19 09:43:53.359 2012] [3388:2244] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:53.359 2012] [3388:2244] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500
[Mon Mar 19 09:43:53.391 2012] [3388:2276] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:53.391 2012] [3388:2276] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500
[Mon Mar 19 09:43:53.484 2012] [3388:2244] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:53.484 2012] [3388:2244] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500

Log4
[Mon Mar 19 09:43:53.266 2012] [2516:1892] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:53.313 2012] [2516:1892] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500
[Mon Mar 19 09:43:53.359 2012] [2516:2776] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:53.359 2012] [2516:2776] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500
[Mon Mar 19 09:43:53.500 2012] [2516:2776] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:43:53.547 2012] [2516:2776] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500


I started IIS, used my client app to connect.. that worked well, only 4 connects, then started my stress test, with 500 users.  Basically 500 users connecting to IIS.  Failed with these errors. Using performance monitor I can see that the number of ISAPI Extension requests is increasing

I turned on trace ( I can send you the files if you needed) and it seemed to work okay (I wonder if the serialisation into logging has something to do with it !
Found these

[Mon Mar 19 09:49:52.489 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (158): enter
[Mon Mar 19 09:49:52.489 2012] [5804:5836] [debug] jk_shm_open::jk_shm.c (305): Initialized shared memory Global\JK_DEV_YIELDBROKER_COM_2_JAKARTA_ISAPI_REDIRECT size=1856 free=1728 addr=0x1f0000
[Mon Mar 19 09:49:52.489 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (311): exit
[Mon Mar 19 09:49:52.504 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (158): enter
[Mon Mar 19 09:49:52.504 2012] [5804:5836] [debug] jk_shm_open::jk_shm.c (166): Shared memory is already opened
[Mon Mar 19 09:49:52.504 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (167): exit
[Mon Mar 19 09:49:52.504 2012] [5804:5836] [trace] wc_open::jk_worker.c (50): enter
[Mon Mar 19 09:49:52.504 2012] [5804:5836] [debug] jk_map_dump::jk_map.c (589): Dump of map: 'worker.maintain' -> '60'

I still have a lot of stale ISAPA extension request's

I have tried to restart IIS, whilst my client is trying to talk over 500 connections, with failed request tracking on I get 500/200 ?
Notification 128 Internal Server Error


Only 1 of the 4 logs files has this is there

[Mon Mar 19 09:58:27.878 2012] [2620:4636] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:58:27.894 2012] [2620:4636] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500
[Mon Mar 19 09:58:27.910 2012] [2620:4636] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer
[Mon Mar 19 09:58:27.910 2012] [2620:4636] [error] HttpExtensionProc::jk_isapi_plugin.c (2325): service() failed with http error 500


Thanks
Alex

-----Original Message-----
From: Mladen Turk [mailto:[hidden email]]
Sent: Monday, 19 March 2012 8:34 AM
To: [hidden email]
Subject: Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

On 03/18/2012 09:34 PM, [hidden email] wrote:

> -----Original-Nachricht-----
>> Von: Mladen Turk<[hidden email]>
>> An: [hidden email]
>> Betreff: Re: IIS 7.0 Worker process crashes on App Pool recycling
>> since ISAPI redirector 1.2.33
>> Datum: Sun, 18 Mar 2012 18:39:15 +0100
>>
>> Could you please try the isapi_redirect_1.2.34_5-dev_win32.zip
>> If this one crashes we'll have to add some override to shared memory
>> so we can fallback to heap one and bypass memory corruption which
>> seems to occur at IIS worker recycle.
>
> Hi Mladen,
>
> thank you, I tried isapi_redirect_1.2.34_5-dev_win32.zip and so far no crashes, and also no warnings/errors in the ISAPI log.
> It seems the issue is fixed now.
>

Cool. Thanks again for testing those.
We'll release 1.2.34 by the end of this week probably.

Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
On 03/19/2012 12:00 AM, Alex Samad - Yieldbroker wrote:
> Hi
>
> Sorry it failed for me, started up, but I got some error messages
>
> [Mon Mar 19 09:43:13.970 2012] [2096:2984] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer

This is caused by unable to allocate the buffer.
Now, it seems either memory is exhausted or shared memory still wrong (I recon its later)

>
> [Mon Mar 19 09:49:52.489 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (158): enter
> [Mon Mar 19 09:49:52.489 2012] [5804:5836] [debug] jk_shm_open::jk_shm.c (305): Initialized shared memory Global\JK_DEV_YIELDBROKER_COM_2_JAKARTA_ISAPI_REDIRECT size=1856 free=1728 addr=0x1f0000
> [Mon Mar 19 09:49:52.489 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (311): exit
> [Mon Mar 19 09:49:52.504 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (158): enter
> [Mon Mar 19 09:49:52.504 2012] [5804:5836] [debug] jk_shm_open::jk_shm.c (166): Shared memory is already opened
> [Mon Mar 19 09:49:52.504 2012] [5804:5836] [trace] jk_shm_open::jk_shm.c (167): exit

That's weird. jk_shm_open gets called twice from the same process/thread.
Could you send me some more log data surrounding those two calls so I can figure out from
where init_jk was called.

Please use the newest build (#6)
from http://people.apache.org/~mturk/tomcat-connectors/jk-1.2.34/



Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: IIS 7.0 Worker process crashes on App Pool recycling since ISAPI redirector 1.2.33

mturk
In reply to this post by Alex Samad - Yieldbroker
On 03/19/2012 12:00 AM, Alex Samad - Yieldbroker wrote:
> Hi
> [Mon Mar 19 09:43:53.266 2012] [2516:1892] [error] service::jk_lb_worker.c (1172): Failed allocating AJP message buffer

Please use the newest build (#8)
from http://people.apache.org/~mturk/tomcat-connectors/jk-1.2.34/

There was a bug in shared memory sync caused by attaching
new process while there was request pending
In general caused by 'generation in future'.
Now the shared memory is synced only if we are below the
shared sequence.

I have verified locally that it works now and no more those
alloc failed errors (as well as none of the earlier one :)


Regards
--
^TM

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12
Loading...