mod_jk connection problem after couple of days or a week

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

mod_jk connection problem after couple of days or a week

Jonathon Reeve
Hi All,

At our company we run JBoss for several applications, and have recently
setup a new box that is giving some rather odd errors.

The box runs:
Windows 2003 Advanced Server
Apache 2.0.54
JBoss 4.0.2
mod_jk 1.2.15 (precompiled binary)

We were previously running jk2, but since that's now sort of deprecated
we've set up this box on jk 1.2.15

Basically, the box is running fine for some time between a couple of
days to a week, and then suddenly we start getting Apache reporting an
error (503 I think, if I remember correctly from last time it happened).
Looking in the mod_jk log file we get these errors constantly:

[Mon Jan 16 09:53:46 2006] [error] jk_ajp_common.c (1026): ERROR: can't
receive the response message from tomcat, network problems or tomcat
(127.0.0.1:8990) is down -53
[Mon Jan 16 09:53:46 2006] [error] jk_ajp_common.c (1528): Tomcat is
down or network problems. Part of the response has already been sent to
the client
[Mon Jan 16 09:53:46 2006] [info]  jk_ajp_common.c (1721): Receiving
from tomcat failed, recoverable operation attempt=0
[Mon Jan 16 09:53:46 2006] [info]  jk_ajp_common.c (1749): Sending
request to tomcat failed,  recoverable operation attempt=1
[Mon Jan 16 09:53:46 2006] [error] jk_ajp_common.c (1026): ERROR: can't
receive the response message from tomcat, network problems or tomcat
(127.0.0.1:8990) is down -53

It tries a couple more times, with similar results, on the second
attempt saying "Tomcat is down or refused connection" instead of "Tomcat
is down or network problems", and on the third attempt saying "Error
connecting to tomcat. Tomcat is probably not started or is listening on
the wrong port".

I checked the source and found that the error code "-53" is derived
from the windows socket error codes - according to MSDN this one is
"Software caused connection abort. An established connection was aborted
by the software in your host computer, possibly due to a data
transmission time-out or protocol error".

JBoss starts having the following message in its logs repeatedly:

2006-01-16 09:56:39,968 WARN  [org.apache.jk.common.ChannelSocket]
processCallbacks status 2

It looks like the sockets are getting screwed up and past this point it
seems mod_jk and JBoss can't communicate properly. Unfortunately,
stopping JBoss and Apache and restarting them does no good - it seems
like the sockets are all occupied and hanging around in a bad state. We
have to actually reboot the box to get it up and running again.

Anybody had anything like this, or know what might cause it? We were
wondering if we've got pools of workers / sockets configured badly
(mis-matched).



Tomcat connectors in servers.xml:
(there are several appservers this box connects to, but they all look
like this, just with different ports)
=====================
      <Connector port="8080" address="${jboss.bind.address}"
         maxThreads="250" strategy="ms" maxHttpHeaderSize="8192"
         emptySessionPath="true"
         enableLookups="false" redirectPort="8443" acceptCount="100"
         connectionTimeout="20000" disableUploadTimeout="true"/>

      <!-- A AJP 1.3 Connector on port 8009 -->
      <Connector port="8009" address="${jboss.bind.address}"
         maxThreads="1000" minSpareThreads="30" maxSpareThreads="60"
         emptySessionPath="true" enableLookups="false"
redirectPort="8443"
         protocol="AJP/1.3"/>

==========================================================

Contents of workers.properties:
====================
worker.list=eOrig,eOrigUAT, jkstatus

# Status worker
worker.jkstatus.type=status

# eOriginations
worker.eOrig.port=8991

# eOriginations UAT
worker.eOrigUAT.host=uklxfcocr
worker.eOrigUAT.port=8991

==========================================================

Contents of httpd.conf:
(stripped out all the standard, unchanged, and basic stuff)
====================================
#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
ServerRoot "Z:/Apache Group/Apache2"

# PidFile: The file in which the server should record its process
# identification number when it starts.
PidFile logs/httpd.pid

# Timeout: The number of seconds before receives and sends time out.
Timeout 300

# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
KeepAlive On

# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited
amount.
# We recommend you leave this number high, for maximum performance.
MaxKeepAliveRequests 10000

# KeepAliveTimeout: Number of seconds to wait for the next request from
the
# same client on the same connection.
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
##

# WinNT MPM
# ThreadsPerChild: constant number of worker threads in the server
process
# MaxRequestsPerChild: maximum  number of requests a server process
serves
<IfModule mpm_winnt.c>
ThreadsPerChild 500
MaxRequestsPerChild  0
</IfModule>

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
Listen 80

#
# Dynamic Shared Object (DSO) Support
#
LoadModule access_module modules/mod_access.so
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_module modules/mod_auth.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule imap_module modules/mod_imap.so
LoadModule include_module modules/mod_include.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so

...
...


### Reverse Proxy Config - added 2005-11-28 for forwarding bug tracker
requests
# disable forward proxy
ProxyRequests Off
# pass /tracker requests to box for purpose
ProxyPass /tracker http://136.140.111.22/tracker
# rewrite headers from backend box to make it appear that request was
serviced by this box
ProxyPassReverse /tracker http://136.140.111.22/tracker

...
...

###
#
# JBoss connector
#
#LoadModule jk2_module modules/mod_jk2.0.49.so
LoadModule jk_module modules/mod_jk-apache-2.0.55.so
JkWorkersFile "Z:/Apache Group/Apache2/conf/workers.properties"
JkLogFile "Z:/apacheLogs/jk.log"
JkLogLevel info
JkMount /eorig/* eOrig
JkMount /uat/eorig/* eOrigUAT
JkMount /jkmanager/* jkstatus

<Location /jkmanager/>
    JkMount jkstatus
    Order deny,allow
    Deny from all
    Allow from 136.140.111
 </Location>


This mail has been checked at MSXI for all known virus'.
You open this at your own risk. Please make sure all replies are virus free.
Also, we do not accept or send attachments of the type exe, vbs, scr or bat due to the increased virus risk they contain.
These types of attachments will be stripped from the message.

MSXI


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: mod_jk connection problem after couple of days or a week

Robert Neumann
Hello!

 

We have exactly the same problem with the following configuration:

 

OS: Windows Server 2003 Standard Edition

Apache 2.0.54

mod_jk 1.2.14

Tomcat 5.5.9/5.5.12

JBoss 2.4.4

 

Our website basically serves JSP-Files and custom webapps, using the Tomcat
instance as Servlet/JSP container and JBoss as backend for EJBs and database
connection. In front of the Tomcat we're using the Apache for handling
requests for static content.

 

We've experienced connection losses beetween Tomcat and Apache with quite
different periods in between (half an hour up to 2-3 weeks). Both, restart
of Tomcat and/or Apache could not bring the system back to health, so the
only solution is a server reboot.

 

We could not locate the root cause, so we decided to install the load
balancing mechanism provided by the Jk-Module of the Apache httpd. We set up
2 Tomcat nodes, residing on different machines. The Apache served as initial
handler for the request and delivers requests to our primary Tomcat node
(A). We expected a switch to the second Tomcat node (B) in case of a
failover, so that there would be no downtime for the website. After a
restart of the machine where Tomcat A lives we expected the system to resume
normal operation.

 

Unfortunatly this did not solve the problem, because the balancer did not
switch to Tomcat B. So the problem seems to reside in mod_jk, which let us
finally switch back to an older version of mod_jk (1.2.6). This scenario is
running for about 6 hours now, so there is no statement possible if this
step was successful.

 

Otherwise it could be possible that it is a problem of the OS, since you
were running Windows Server 2003 too.

 

Here some log entries from both, Apache and Tomcat Node A from our load
balancer scenario. It seems that the connection is dropped by the mod_jk
module.

 

Our config is not kind of exotic, so I would be really surprised if it
should be a configuration error.

 

Apache:

 

[Thu Jan 19 08:47:52 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53

[Thu Jan 19 08:47:52 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 08:47:52 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0

[Thu Jan 19 08:47:52 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=1

[Thu Jan 19 08:53:07 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53

[Thu Jan 19 08:53:07 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 08:53:07 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0

[Thu Jan 19 08:53:07 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=1

[Thu Jan 19 08:54:10 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53

[Thu Jan 19 08:54:10 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 08:54:10 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0

[Thu Jan 19 08:54:10 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=1

[Thu Jan 19 08:54:31 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53

[Thu Jan 19 08:54:31 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 08:54:31 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0

[Thu Jan 19 08:54:31 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=1

[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53

[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 09:05:41 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0

[Thu Jan 19 09:05:41 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=1

[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53

[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 09:05:41 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=1

[Thu Jan 19 09:05:41 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=2

[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8209), err=-53

[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 09:05:41 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=2

[Thu Jan 19 09:05:41 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=3

[Thu Jan 19 09:05:41 2006] [error] jk_ajp_common.c (1758): Error connecting
to tomcat. Tomcat is probably not started or is listening on the wrong port.
worker=tomcatA_xxx failed

[Thu Jan 19 09:05:41 2006] [info]  jk_lb_worker.c (662): service failed,
worker tomcatA_xxx is in error state

[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53

[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 09:05:46 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0

[Thu Jan 19 09:05:46 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=1

[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53

[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 09:05:46 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=0

[Thu Jan 19 09:05:46 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=1

[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53

[Thu Jan 19 09:05:46 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 09:05:46 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=1

[Thu Jan 19 09:05:46 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=2

[Thu Jan 19 09:05:47 2006] [error] jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(10.194.16.111:8009), err=-53

[Thu Jan 19 09:05:47 2006] [error] jk_ajp_common.c (1528): Tomcat is down or
network problems. Part of the response has already been sent to the client

[Thu Jan 19 09:05:47 2006] [info]  jk_ajp_common.c (1721): Receiving from
tomcat failed, recoverable operation attempt=1

[Thu Jan 19 09:05:47 2006] [info]  jk_ajp_common.c (1749): Sending request
to tomcat failed,  recoverable operation attempt=2

 

Tomcat:

 

2006-01-19 08:53:02,622 [TP-Processor1] WARN  org.apache.jk.core.MsgContext
- Error sending end packet

java.net.SocketException: Connection reset by peer: socket write error

            at java.net.SocketOutputStream.socketWrite0(Native Method)

            at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

            at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)

            at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)

            at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)

            at org.apache.jk.core.MsgContext.action(MsgContext.java:293)

            at org.apache.coyote.Response.action(Response.java:182)

            at org.apache.coyote.Response.finish(Response.java:304)

            at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)

            at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)

            at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)

            at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)

            at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)

            at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)

            at java.lang.Thread.run(Thread.java:595)

2006-01-19 08:53:02,622 [TP-Processor1] WARN
org.apache.jk.common.ChannelSocket - processCallbacks status 2

2006-01-19 09:05:35,541 [TP-Processor3] WARN  org.apache.jk.core.MsgContext
- Error sending end packet

java.net.SocketException: Connection reset by peer: socket write error

            at java.net.SocketOutputStream.socketWrite0(Native Method)

            at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

            at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)

            at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)

            at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)

            at org.apache.jk.core.MsgContext.action(MsgContext.java:293)

            at org.apache.coyote.Response.action(Response.java:182)

            at org.apache.coyote.Response.finish(Response.java:304)

            at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:281)

            at
org.apache.catalina.connector.Response.finishResponse(Response.java:483)

            at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)

            at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)

            at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)

            at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)

            at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)

            at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)

            at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)

            at java.lang.Thread.run(Thread.java:595)

2006-01-19 09:05:35,541 [TP-Processor3] WARN  org.apache.jk.core.MsgContext
- Error sending end packet

java.net.SocketException: Connection reset by peer: socket write error

            at java.net.SocketOutputStream.socketWrite0(Native Method)

            at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

            at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)

            at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)

            at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)

            at org.apache.jk.core.MsgContext.action(MsgContext.java:293)

            at org.apache.coyote.Response.action(Response.java:182)

            at org.apache.coyote.Response.finish(Response.java:304)

            at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)

            at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)

            at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)

            at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)

            at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)

            at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)

            at java.lang.Thread.run(Thread.java:595)

2006-01-19 09:05:35,541 [TP-Processor3] WARN
org.apache.jk.common.ChannelSocket - processCallbacks status 2

2006-01-19 09:05:35,885 [TP-Processor3] WARN  org.apache.jk.core.MsgContext
- Error sending end packet

java.net.SocketException: Connection reset by peer: socket write error

            at java.net.SocketOutputStream.socketWrite0(Native Method)

            at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

            at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)

            at
org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:508)

            at
org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)

            at org.apache.jk.core.MsgContext.action(MsgContext.java:293)

            at org.apache.coyote.Response.action(Response.java:182)

            at org.apache.coyote.Response.finish(Response.java:304)

            at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)

            at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)

            at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)

            at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)

            at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java
:866)

            at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:684)

            at java.lang.Thread.run(Thread.java:595)

2006-01-19 09:05:35,885 [TP-Processor3] WARN
org.apache.jk.common.ChannelSocket - processCallbacks status 2 .

 

Maybe anyone out there has an idea or knowlegde according to this problem.

 

Help is greatly appreciated!

Reply | Threaded
Open this post in threaded view
|

RE: mod_jk connection problem after couple of days or a week

Jonathon Reeve
In reply to this post by Jonathon Reeve
Well I'm glad someone else is getting this, even if you have no idea
either. We wondered if we were missing some configuration from
workers.properties, since the "quick start" section
(http://tomcat.apache.org/connectors-doc/howto/quick.html) on
Tomcat's site gives an example "minimum workers.properties":

 # Define 1 real worker using ajp13
  worker.list=worker1
  # Set properties for worker1 (ajp13)
  worker.worker1.type=ajp13
  worker.worker1.host=localhost
  worker.worker1.port=8009
  worker.worker1.lbfactor=50
  worker.worker1.cachesize=10
  worker.worker1.cache_timeout=600
  worker.worker1.socket_keepalive=1
  worker.worker1.reclycle_timeout=300

... which is a lot more than we had:

worker.list=eOrig,eOrigUAT, jkstatus
# Status worker
worker.jkstatus.type=status
# eOriginations
worker.eOrig.port=8991
# eOriginations UAT
worker.eOrigUAT.host=uklxfcocr
worker.eOrigUAT.port=8991

We looked at the documentation for the settings in this file, here:
http://tomcat.apache.org/connectors-doc/howto/workers.html
and here:
http://tomcat.apache.org/connectors-doc/config/workers.html
...but it doesn't really help too much. It tells you quite precisely
what each property does, but it would be more useful to have a series of
examples along the lines of, "for a typical server doing X, you might
want something like this, and if you set this too high / low / whatever,
it will have effect Y". That said, there were a couple of things we
thought might help:

  # worker "worker2" ask operating system to send KEEP-ALIVE signal on
the connection
  worker.worker2.socket_keepalive=1
  worker.worker2.recycle_timeout=300

...since the sockets seem to be sort of dying and then hanging around.
We thought perhaps the OS should be keeping them alive and was dropping
them instead, and if they were dead that they should be recycled. With
all changes, out workers.properties file now looks like this:

worker.list=eOrig,eOrigUAT,jkstatus
# Status worker
worker.jkstatus.type=status
## eOriginations
worker.eOrig.port=8991
worker.eOrig.cachesize=50
worker.eOrig.cache_timeout=600
worker.eOrig.socket_keepalive=1
worker.eOrig.recycle_timeout=300
## eOriginations UAT
worker.eOrigUAT.host=uklxfcocr
worker.eOrigUAT.port=8991
worker.eOrigUAT.cachesize=20
worker.eOrigUAT.cache_timeout=600
worker.eOrigUAT.socket_keepalive=1
worker.eOrigUAT.recycle_timeout=300


We're also thinking about trying "worker.maintain", since it sounds
like it might help with the recycling, but we're seeing how it goes with
these changes first.

Fingers crossed...

This mail has been checked at MSXI for all known virus'.
You open this at your own risk. Please make sure all replies are virus free.
Also, we do not accept or send attachments of the type exe, vbs, scr or bat due to the increased virus risk they contain.
These types of attachments will be stripped from the message.

MSXI


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: mod_jk connection problem after couple of days or a week

Mladen Turk-2
Jonathon Reeve wrote:
> Well I'm glad someone else is getting this, even if you have no idea
> either. We wondered if we were missing some configuration from
> workers.properties, since the "quick start" section
> (http://tomcat.apache.org/connectors-doc/howto/quick.html) on
> Tomcat's site gives an example "minimum workers.properties":
>


Huh, you really tried everything :)

Couple of notes:
1. AJP connection is supposed to last forever.
2. cache_timeout, recycle_timeout, MaxRequestsPerClient
    tend to close the socket, so you have half-closed
    sockets, and that's why you are leaking resources.
3. Whenever there is a need to recycle one end of the
    channel the other end must be recycled too.
    This is done by setting connectionTimeout="milliseconds"
    in server.xml for AJP/1.3 connector.
4. If you need full failover add 'connection_timeout' and
    'prepost_timeout' to each worker.
    This implicates that CPING/CPONG packets will be send
    before each request is forwarded to Tomcat.

Regards,
Mladen.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: mod_jk connection problem after couple of days or a week

Robert Neumann
Thanks Mladen for your suggestions.

But I have to make clear that we just started with a minimum configuration
for the workers.

So we had 3 workers defined using the following config:

worker.worker_1.host=localhost
worker.worker_1.port=8009
worker.worker_1.type=ajp13

worker.worker_2.host=localhost
worker.worker_2.port=8109
worker.worker_2.type=ajp13

worker.worker_3.host=localhost
worker.worker_3.port=8209
worker.worker_3.type=ajp13

There was nothing about connection recycling - just using the standard.
So I am not sure about that what you have said about making sure that not
only one endpoint of the connection is dropped, which should not happen in
this case?!

Afterwards, we've checked the manuals again and set up a quite similar
configuration just as Jonathon did (except the keep_alive parameter, since
we were running Apache and Tomcat on the same host). That's where I could
follow your objection.

But furthermore I cannot imagine how the whole setup could be running for
about 2 weeks and then go down after a couple of minutes (after a server
reboot). Even under heavy load conditions this seems very mysterious, since
even with the "recycle configuration", the connections should not be dropped
that fast, so that there are dead sockets hanging around.

Even the problem with server reboot at all is quite strange, since it
doesn't seem to be a problem with the OS. After Tomcat was down, I could
connect to the server remotely using the MS Terminal Client (RDP -
Connection). So no general socket problem at all, in my opinion...

Is it possible that after Tomcat shutdown any Java related thread or process
is still keeping the dead sockets and no connections can be made until this
thread/process is killed during server shutdown?

Additional information:
Tomcat is running with jdk1.5.0_04, using the server JVM.

@Jonathon:
Do you use any special switches for the JVM? We're using the same switches
which worked fine for us when running the system on Win2K Server (where the
system ran fine before using mod_jk 1.2.6 - ;) ), such as settings PermSize
etc.

Regards,
Robert.

Jonathon Reeve wrote:
> Well I'm glad someone else is getting this, even if you have no idea
> either. We wondered if we were missing some configuration from
> workers.properties, since the "quick start" section
> (http://tomcat.apache.org/connectors-doc/howto/quick.html) on
> Tomcat's site gives an example "minimum workers.properties":
>


Huh, you really tried everything :)

Couple of notes:
1. AJP connection is supposed to last forever.
2. cache_timeout, recycle_timeout, MaxRequestsPerClient
    tend to close the socket, so you have half-closed
    sockets, and that's why you are leaking resources.
3. Whenever there is a need to recycle one end of the
    channel the other end must be recycled too.
    This is done by setting connectionTimeout="milliseconds"
    in server.xml for AJP/1.3 connector.
4. If you need full failover add 'connection_timeout' and
    'prepost_timeout' to each worker.
    This implicates that CPING/CPONG packets will be send
    before each request is forwarded to Tomcat.

Regards,
Mladen.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: mod_jk connection problem after couple of days or a week

Jonathon Reeve
In reply to this post by Jonathon Reeve
No we don't really change much for JVM params, we use -Xms128m -Xmx512m
for memory on this machine - that's about it.

>>> [hidden email] 20/01/2006 12:48:57 >>>


@Jonathon:
Do you use any special switches for the JVM? We're using the same
switches
which worked fine for us when running the system on Win2K Server (where
the
system ran fine before using mod_jk 1.2.6 - ;) ), such as settings
PermSize
etc.

Regards,
Robert.

Jonathon Reeve wrote:
> Well I'm glad someone else is getting this, even if you have no idea
> either. We wondered if we were missing some configuration from
> workers.properties, since the "quick start" section
> (http://tomcat.apache.org/connectors-doc/howto/quick.html) on
> Tomcat's site gives an example "minimum workers.properties":
>


Huh, you really tried everything :)

Couple of notes:
1. AJP connection is supposed to last forever.
2. cache_timeout, recycle_timeout, MaxRequestsPerClient
    tend to close the socket, so you have half-closed
    sockets, and that's why you are leaking resources.
3. Whenever there is a need to recycle one end of the
    channel the other end must be recycled too.
    This is done by setting connectionTimeout="milliseconds"
    in server.xml for AJP/1.3 connector.
4. If you need full failover add 'connection_timeout' and
    'prepost_timeout' to each worker.
    This implicates that CPING/CPONG packets will be send
    before each request is forwarded to Tomcat.

Regards,
Mladen.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



This mail has been checked at MSXI for all known virus'.
You open this at your own risk. Please make sure all replies are virus free.
Also, we do not accept or send attachments of the type exe, vbs, scr or bat due to the increased virus risk they contain.
These types of attachments will be stripped from the message.

MSXI


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: mod_jk connection problem after couple of days or a week

Guernsey, Byron (GE Consumer & Industrial)
In reply to this post by Jonathon Reeve

Just wanted to point out that on #1 below: many real world environments
have web servers in a DMZ and a firewall between their web servers an
app servers.  The default assumption should be that AJP connections can
and are frequently broken.  Firewalls will do this to you.

I'm curious why tomcat and mod_jk both won't detect properly closed
sockets out of the box?  I can understand why firewall disconnects would
go unnoticed, but if you set cache_timeout, recycle_timeout, or
MaxRequestsPerClient, I'd expect the TCP connection to notify the other
end of the close and for the app server to detect it like any well
behaved TCP-based server.

There are many ways to detect the properly closed socket- some
event-based ways as well, both in a C and Java.  It seems odd that we'd
need to set a timeout to wake up a server to cleanup properly closed
sockets- that should be inherent to its operation.

Byron
 

-----Original Message-----
From: Mladen Turk [mailto:[hidden email]]
Sent: Friday, January 20, 2006 5:19 AM
To: Tomcat Users List
Subject: Re: mod_jk connection problem after couple of days or a week

Jonathon Reeve wrote:
> Well I'm glad someone else is getting this, even if you have no idea
> either. We wondered if we were missing some configuration from
> workers.properties, since the "quick start" section
> (http://tomcat.apache.org/connectors-doc/howto/quick.html) on Tomcat's

> site gives an example "minimum workers.properties":
>


Huh, you really tried everything :)

Couple of notes:
1. AJP connection is supposed to last forever.
2. cache_timeout, recycle_timeout, MaxRequestsPerClient
    tend to close the socket, so you have half-closed
    sockets, and that's why you are leaking resources.
3. Whenever there is a need to recycle one end of the
    channel the other end must be recycled too.
    This is done by setting connectionTimeout="milliseconds"
    in server.xml for AJP/1.3 connector.
4. If you need full failover add 'connection_timeout' and
    'prepost_timeout' to each worker.
    This implicates that CPING/CPONG packets will be send
    before each request is forwarded to Tomcat.

Regards,
Mladen.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]