Debugging tomcat native connector

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Debugging tomcat native connector

Mark A. Claassen
Is there a way to debug the native connectors?  Specifically, we are having some problems getting the native openssl connector working on Ubuntu.
Doing an strace on the process shows a lot of FUTEX_WAIT_PRIVATE, but we don't know why.  We were hoping that there was some way to get more information.

Thanks,

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:[hidden email]
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.

Reply | Threaded
Open this post in threaded view
|

Re: Debugging tomcat native connector

Roger Brechbühl
with which connector do you have problems? nio, nio2 or apr?

I ask because we have problems with nio2-openssl when ssl session is reused
e.g. when a request is proxied with nginx.

kind regards,
Roger

Mark A. Claassen <[hidden email]> schrieb am Mi. 31. Okt. 2018 um 15:32:

> Is there a way to debug the native connectors?  Specifically, we are
> having some problems getting the native openssl connector working on Ubuntu.
> Doing an strace on the process shows a lot of FUTEX_WAIT_PRIVATE, but we
> don't know why.  We were hoping that there was some way to get more
> information.
>
> Thanks,
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:[hidden email]
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect
> those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
> assumes no legal liability or responsibility for the posting.
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Debugging tomcat native connector

Mark A. Claassen
Thanks for the reply.  I am using "org.apache.coyote.http11.Http11AprProtocol" in Tomcat 9.

The NIO connectors worked fine when we tried them, but the native ones seem to start up, but then don’t work.
 
Interestingly, we found out that these connectors do eventually start working!  We tried to connect a few hours later and they were working fine.  
We are now trying to determine exactly how long it takes for them to accept connections in hopes that this will help us figure out what is going on.

Having a way to debug this better would be helpful.


Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:[hidden email]
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.
-----Original Message-----
From: Roger Brechbühl <[hidden email]>
Sent: Wednesday, October 31, 2018 5:25 PM
To: Tomcat Users List <[hidden email]>
Subject: Re: Debugging tomcat native connector

with which connector do you have problems? nio, nio2 or apr?

I ask because we have problems with nio2-openssl when ssl session is reused e.g. when a request is proxied with nginx.

kind regards,
Roger

Mark A. Claassen <[hidden email]> schrieb am Mi. 31. Okt. 2018 um 15:32:

> Is there a way to debug the native connectors?  Specifically, we are
> having some problems getting the native openssl connector working on Ubuntu.
> Doing an strace on the process shows a lot of FUTEX_WAIT_PRIVATE, but
> we don't know why.  We were hoping that there was some way to get more
> information.
>
> Thanks,
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:[hidden email]
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those
> of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes
> no legal liability or responsibility for the posting.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Debugging tomcat native connector

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 11/1/18 11:51, Mark A. Claassen wrote:

> Thanks for the reply.  I am using
> "org.apache.coyote.http11.Http11AprProtocol" in Tomcat 9.
>
> The NIO connectors worked fine when we tried them, but the native
> ones seem to start up, but then don’t work.
>
> Interestingly, we found out that these connectors do eventually
> start working!  We tried to connect a few hours later and they were
> working fine. We are now trying to determine exactly how long it
> takes for them to accept connections in hopes that this will help
> us figure out what is going on.
>
> Having a way to debug this better would be helpful.
Are you getting log entries in catalina.out similar to these?

INFO: Starting ProtocolHandler ["ajp-nio-8215"]
Oct 10, 2018 4:23:38 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-127.0.0.1-8217"]
Oct 10, 2018 4:23:38 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6819 ms

??

If so, there should also be some timing information for initializing
certain things. For example, if the server doesn't have much entropy
in the entropy pool, then you can get a slow-startup while the crypto
engine starts. This may be worse with the APR connector than for the
NIO connectors as the source of randomness may be different by default
(e.g. /dev/urandom for Java, /dev/random for APR). That may not
actually be *true*, but it's a possibility.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvbNXYACgkQHPApP6U8
pFhXkBAAny/RjuglPB38vMd1HZ45+9ifW4zRVgGY+2VjtJLSCJKf8+bouMdr6ZWY
5MbE9ru4WYNnIE7J01gYK7vjGfrotjxNLVAsY52qiCAf9u6Q+hsZb00T5B9Fog+z
9lFm2J4alHUMTe758vBJDtuuCqiB3I3SaY2f5rp5TiCWYVExrbNDQUe50ClbqFmB
+/e0XIybWXMCPsgylfyqQIZLwJ3t/Q+1eocQUIst86+tMrJdv2PWu+b7+J2y752Y
8dy3LqUjLQ218nDaXe1DXnWajunhuwAunK0/y1fL4pqlwJYDf6GTuWbW+AVDkBZh
3Q1o74QmM230FVyBhll2HSh9wi+rOON7MnjQD8Buks8LeTzyHRdGPRnFXoZm4H99
Re5uDXHLsJ0pweucVc4hqfE+2BxW33GT36FHUjGVAijMO6V3NX4CYhiOD7XkSlaz
uvoC+3ibhZZrgmc1AkM122u0YciSLDLb5iqGoVP/cOGJ0JVEwlEPMrpwSSe0ZTXz
BB6mTRu9WzN9j4YboTNmuiYnBx6wDAEWu+f9CjFptw7qGcwrzqrFyLp+/QuYzf4l
3iEVDlXPjF0bFsuJOvTChxfVm7s5NrxUBf2hJfMxmuKjeKKPZElgnWYtyoKokCHW
GNUD+HIGvJmc5bICo+WRzyf7zu1bsVoATcNvzM+K9WF1TBsrS+I=
=7fqB
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Debugging tomcat native connector

Mark A. Claassen
Wow, thanks a lot.  That was exactly what it was.

Something else, just in the event that someone references this email thread in the future, I also realized I could put this in my logging.properties file:
org.apache.coyote.level = FINE

I didn't think to add that until after the entropy thing was figured out, but it is something that may come in handy in similar situations.


Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:[hidden email]
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.
-----Original Message-----
From: Christopher Schultz <[hidden email]>
Sent: Thursday, November 1, 2018 1:19 PM
To: [hidden email]
Subject: Re: Debugging tomcat native connector

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 11/1/18 11:51, Mark A. Claassen wrote:

> Thanks for the reply.  I am using
> "org.apache.coyote.http11.Http11AprProtocol" in Tomcat 9.
>
> The NIO connectors worked fine when we tried them, but the native ones
> seem to start up, but then don’t work.
>
> Interestingly, we found out that these connectors do eventually start
> working!  We tried to connect a few hours later and they were working
> fine. We are now trying to determine exactly how long it takes for
> them to accept connections in hopes that this will help us figure out
> what is going on.
>
> Having a way to debug this better would be helpful.
Are you getting log entries in catalina.out similar to these?

INFO: Starting ProtocolHandler ["ajp-nio-8215"] Oct 10, 2018 4:23:38 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-127.0.0.1-8217"] Oct 10, 2018 4:23:38 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6819 ms

??

If so, there should also be some timing information for initializing certain things. For example, if the server doesn't have much entropy in the entropy pool, then you can get a slow-startup while the crypto engine starts. This may be worse with the APR connector than for the NIO connectors as the source of randomness may be different by default (e.g. /dev/urandom for Java, /dev/random for APR). That may not actually be *true*, but it's a possibility.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvbNXYACgkQHPApP6U8
pFhXkBAAny/RjuglPB38vMd1HZ45+9ifW4zRVgGY+2VjtJLSCJKf8+bouMdr6ZWY
5MbE9ru4WYNnIE7J01gYK7vjGfrotjxNLVAsY52qiCAf9u6Q+hsZb00T5B9Fog+z
9lFm2J4alHUMTe758vBJDtuuCqiB3I3SaY2f5rp5TiCWYVExrbNDQUe50ClbqFmB
+/e0XIybWXMCPsgylfyqQIZLwJ3t/Q+1eocQUIst86+tMrJdv2PWu+b7+J2y752Y
8dy3LqUjLQ218nDaXe1DXnWajunhuwAunK0/y1fL4pqlwJYDf6GTuWbW+AVDkBZh
3Q1o74QmM230FVyBhll2HSh9wi+rOON7MnjQD8Buks8LeTzyHRdGPRnFXoZm4H99
Re5uDXHLsJ0pweucVc4hqfE+2BxW33GT36FHUjGVAijMO6V3NX4CYhiOD7XkSlaz
uvoC+3ibhZZrgmc1AkM122u0YciSLDLb5iqGoVP/cOGJ0JVEwlEPMrpwSSe0ZTXz
BB6mTRu9WzN9j4YboTNmuiYnBx6wDAEWu+f9CjFptw7qGcwrzqrFyLp+/QuYzf4l
3iEVDlXPjF0bFsuJOvTChxfVm7s5NrxUBf2hJfMxmuKjeKKPZElgnWYtyoKokCHW
GNUD+HIGvJmc5bICo+WRzyf7zu1bsVoATcNvzM+K9WF1TBsrS+I=
=7fqB
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]