[Bug 63904] New: Apache http (with mod_jk to tomcat) randomly return empty response

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[Bug 63904] New: Apache http (with mod_jk to tomcat) randomly return empty response

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63904

            Bug ID: 63904
           Summary: Apache http (with mod_jk to tomcat) randomly return
                    empty response
           Product: Tomcat Connectors
           Version: 1.2.43
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: critical
          Priority: P2
         Component: mod_jk
          Assignee: [hidden email]
          Reporter: [hidden email]
  Target Milestone: ---

Created attachment 36870
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=36870&action=edit
Logs and Settings

I have a problem where our Apache/2.4.29 server randomly returning empty
response. The Apache is used as load balancer with mod_jk (mod_jk/1.2.43
OpenSSL/1.1.0g) to Apache Tomcat 8.0.35. Currently we use 8 tomcats behind the
apache http server.

The request from client, sent to apache http server, and successfully processed
by apache tomcat (the data is saved to the database) but it Apache failed to
return the response to the client. The problem occured randomly, without any
pattern. When I open the mod_jk status page, all the tomcats has Err and CE
(See the attachment).

We also see lots of errors on mod_jk logs (see attachment), something like
this:

[Tue Nov 05 06:30:01.110 2019] [4652:140053857666816] [info]
ajp_read_into_msg_buff::jk_ajp_common.c (1553): (accurateb1) receiving data
from client failed. Connection aborted or network problems
[Tue Nov 05 06:30:01.110 2019] [4652:140053857666816] [info]
ajp_service::jk_ajp_common.c (2778): (accurateb1) sending request to tomcat
failed (unrecoverable), because of client read error (attempt=1)
[Tue Nov 05 06:30:01.114 2019] [4652:140053857666816] [info]
service::jk_lb_worker.c (1590): service failed, worker accurateb1 is in local
error state
[Tue Nov 05 06:30:01.114 2019] [4652:140053857666816] [info]
service::jk_lb_worker.c (1609): unrecoverable error 400, request failed. Client
failed in the middle of request, we can't recover to another instance.
[Tue Nov 05 06:30:01.114 2019] [4652:140053857666816] [info]
jk_handler::mod_jk.c (2988): Aborting connection for worker=accurate


[Tue Nov 05 07:18:53.668 2019] [4652:140055813166848] [info]
ajp_process_callback::jk_ajp_common.c (2077): (accuratec4) Writing to client
aborted or client network problems
[Tue Nov 05 07:18:53.668 2019] [4652:140055813166848] [info]
ajp_service::jk_ajp_common.c (2778): (accuratec4) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Tue Nov 05 07:18:53.668 2019] [4652:140055813166848] [info]
service::jk_lb_worker.c (1590): service failed, worker accuratec4 is in local
error state
[Tue Nov 05 07:18:53.668 2019] [4652:140055813166848] [info]
service::jk_lb_worker.c (1609): unrecoverable error 200, request failed. Client
failed in the middle of request, we can't recover to another instance.
[Tue Nov 05 07:18:53.668 2019] [4652:140055813166848] [info]
jk_handler::mod_jk.c (2988): Aborting connection for worker=accurate

Im attaching the configuration settings:
- mod_jk.properties
- one of tomcat server.xml
- apache mod_jk.conf
- mod_jk error logs
- sample failed http request (with screenshot of failure response, and detail
request)

Is this miss configuration or bug?

THanks

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63904] Apache http (with mod_jk to tomcat) randomly return empty response

Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63904

Christopher Schultz <[hidden email]> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|NEW                         |RESOLVED

--- Comment #1 from Christopher Schultz <[hidden email]> ---
The log shows that mod_jk can't tell whether the request was processed or not:

> Client failed in the middle of request, we can't recover to another instance.

mod_jk can't prove that the request could be idempotent, so failing-over to
another server might be disastrous. There is nothing to be done but to fail the
request.

This is intended behavior in this case.

My recommendation would be for you to review all your various timeouts to make
sure mod_jk and Tomcat agree. Enable CPING/CPONG on the mod_jk end. See if that
improves anything.

If things are still behaving oddly, please post to the users list, as Bugzilla
is not intended to be a support forum. We can always re-open this bug if there
is indeed a bug (though mod_jk works in huge numbers of environments without
problems, so a bug is unlikely).

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[Bug 63904] Apache http (with mod_jk to tomcat) randomly return empty response

Bugzilla from bugzilla@apache.org
In reply to this post by Bugzilla from bugzilla@apache.org
https://bz.apache.org/bugzilla/show_bug.cgi?id=63904

--- Comment #2 from Yosep Stephen <[hidden email]> ---
I don't see any problem with the timeout settings. Here are Tomcat 8.0.35
Setting used:
<Connector port="8610"
               protocol="AJP/1.3"
               redirectPort="9610"
               maxParameterCount="1000000"
               acceptCount="1000"
               acceptorThreadCount="2"
               maxConnections="20000"
               maxPostSize="20971520"
               maxThreads="20000" />

And here are the mod_jk settings:
worker.accuratea1.host=10.10.17.6
worker.accuratea1.port=8610
worker.accuratea1.lbfactor=1

worker.accuratea1.socket_connect_timeout=5000
worker.accuratea1.ping_mode=A
worker.accuratea1.ping_timeout=10000

Where is the user group I can use to submit the discussion?

Thanks for the response, really appreciate it.

--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]