broken pipe error keeps increasing open files

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

broken pipe error keeps increasing open files

Ayub Khan
tomcat 8.5 broken pipe increases open files on ubuntu AWS

If there is slow response from db I see this stack trace and the open files
goes high and the only way to open files go down is to remove the instance
from Amazon load balancer.

Is there a way to keep the open files low even when Broken pipe error is
thrown ?

org.apache.catalina.connector.ClientAbortException: java.io.IOException:
Broken pipe
        at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
        at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
        at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
        at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317)
        at
org.apache.catalina.connector.Response.flushBuffer(Response.java:497)
        at
org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:318)
        at
org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
        at
org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61)
        at
org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:94)
        at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at
org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:112)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:124)
        at
org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
        at
org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:172)
        at
org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:139)
        at
org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:197)
        at
org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:41)
        at
org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:320)
        at
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:116)
        at
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
        at org.apache.coyote.Response.doWrite(Response.java:501)
        at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388)
        ... 28 common frames omitted
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ayub,

On 6/19/20 16:46, Ayub Khan wrote:
> tomcat 8.5 broken pipe increases open files on ubuntu AWS

Which exact version of Tomcat 8.5? If you aren't running the latest
version (8.5.56), please upgrade and re-test.

> If there is slow response from db I see this stack trace and the
> open files goes high and the only way to open files go down is to
> remove the instance from Amazon load balancer.
>
> Is there a way to keep the open files low even when Broken pipe
> error is thrown ?

What is your evidence that file handles are being left open?

Which file handles are being left open?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7uKvMACgkQHPApP6U8
pFhs7A/+Nxx+iNaiw2tl30cpjkUyIlsS+7Zs43tTafLiMuWWSPP7AoqWe7cf9r2y
APXbvEG0Wu6ptLTUlQ/Qsgr4XqQE9dNekLxP98EtOp/a2K3XMGZ0pZlkX4FuWTmc
AUJdodwYTu9/5MKirx2D9qNblXn71dw1CLiw5BLW7k4LPgxNpx0/aNgTFw7ywIT8
AjwOyyOLBHN1Q+dO0aGuYcrEOob26YzChobEsKXWSVQWe1TOqPJDsUEh+DYWzVZM
1e3Pmd2dCag4I69WRdTJVOXo1ETxMF8sT9faimxsfp1YU6xzrxz/a+JeIzB74oVh
m0Xc+clQuVKEitYOhtaZhBN4REYx115f4Jp2R/8z7jASuji/k3vJ6jIz72ExUqtU
G29xOQV9o08JM0jTZRwmKAp6SqlkPlXXZqEX+wMsOVsY0RRh80c/2mnR411OhD6w
3OZTGnqhRZgQ3AHCqDnkmFxnHmaHdCaZR034lrSM5s5mAt5y7pcefPJyHewkQlyW
GFLt8V4Xpl8nSETkcy36o/anqqrNzy+bufi1bL7K+UXsuoWy+1hpyavesYR7Uz+p
4JdFvQU6QhU7Gz/8BJXK0Jr/cYi7NTHDFpdTR9iedjF7Wf1db2rT0WZh8+63j/L8
6IVDCDqkKBfd5SoMyaoGkL9Dein+v2evNRL06/rqPvRHSB2/gdE=
=8OK0
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
Christopher,

Sorry we are using  8.0.32 version of tomcat.

below is the configuration:

Server version: Apache Tomcat/8.0.32 (Ubuntu)
Server built:   Jan 24 2020 16:24:30 UTC
Server number:  8.0.32.0
OS Name:        Linux
OS Version:     4.4.0-1087-aws
Architecture:   amd64
JVM Version:    1.8.0_181-b13
JVM Vendor:     Oracle Corporation

I use the below command to check the file descriptors:

watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

When there an issue related to broken files, this value keeps increasing,
the only way to bring it down is to remove vm instance from AWS load
balancer.

Which version of tomcat should I install ?






On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Ayub,
>
> On 6/19/20 16:46, Ayub Khan wrote:
> > tomcat 8.5 broken pipe increases open files on ubuntu AWS
>
> Which exact version of Tomcat 8.5? If you aren't running the latest
> version (8.5.56), please upgrade and re-test.
>
> > If there is slow response from db I see this stack trace and the
> > open files goes high and the only way to open files go down is to
> > remove the instance from Amazon load balancer.
> >
> > Is there a way to keep the open files low even when Broken pipe
> > error is thrown ?
>
> What is your evidence that file handles are being left open?
>
> Which file handles are being left open?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7uKvMACgkQHPApP6U8
> pFhs7A/+Nxx+iNaiw2tl30cpjkUyIlsS+7Zs43tTafLiMuWWSPP7AoqWe7cf9r2y
> APXbvEG0Wu6ptLTUlQ/Qsgr4XqQE9dNekLxP98EtOp/a2K3XMGZ0pZlkX4FuWTmc
> AUJdodwYTu9/5MKirx2D9qNblXn71dw1CLiw5BLW7k4LPgxNpx0/aNgTFw7ywIT8
> AjwOyyOLBHN1Q+dO0aGuYcrEOob26YzChobEsKXWSVQWe1TOqPJDsUEh+DYWzVZM
> 1e3Pmd2dCag4I69WRdTJVOXo1ETxMF8sT9faimxsfp1YU6xzrxz/a+JeIzB74oVh
> m0Xc+clQuVKEitYOhtaZhBN4REYx115f4Jp2R/8z7jASuji/k3vJ6jIz72ExUqtU
> G29xOQV9o08JM0jTZRwmKAp6SqlkPlXXZqEX+wMsOVsY0RRh80c/2mnR411OhD6w
> 3OZTGnqhRZgQ3AHCqDnkmFxnHmaHdCaZR034lrSM5s5mAt5y7pcefPJyHewkQlyW
> GFLt8V4Xpl8nSETkcy36o/anqqrNzy+bufi1bL7K+UXsuoWy+1hpyavesYR7Uz+p
> 4JdFvQU6QhU7Gz/8BJXK0Jr/cYi7NTHDFpdTR9iedjF7Wf1db2rT0WZh8+63j/L8
> 6IVDCDqkKBfd5SoMyaoGkL9Dein+v2evNRL06/rqPvRHSB2/gdE=
> =8OK0
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

calder
In reply to this post by Ayub Khan
On Fri, Jun 19, 2020, 15:46 Ayub Khan <[hidden email]> wrote:

> tomcat 8.5 broken pipe increases open files on ubuntu AWS
>


If there is slow response from db


Might be a good idea to investigate the reason for the "slow response"

I see this stack trace and the open files goes high


[ snip ]


Caused by: java.io.IOException: Broken pipe
>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)



If I remember correctly, we had this issue about three+ years ago.    Is
your app using the "dbcp" (1.4) library?  We ended up moving to "dbcp2", in
addition to optimizing the DB queries.
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
Calden,

we are not using dbcp for this project. Also even if this error is thrown
why does the file descriptor keep increasing?


On Sat, Jun 20, 2020 at 8:24 PM calder <[hidden email]> wrote:

> On Fri, Jun 19, 2020, 15:46 Ayub Khan <[hidden email]> wrote:
>
> > tomcat 8.5 broken pipe increases open files on ubuntu AWS
> >
>
>
> If there is slow response from db
>
>
> Might be a good idea to investigate the reason for the "slow response"
>
> I see this stack trace and the open files goes high
>
>
> [ snip ]
>
>
> Caused by: java.io.IOException: Broken pipe
> >         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>
>
>
> If I remember correctly, we had this issue about three+ years ago.    Is
> your app using the "dbcp" (1.4) library?  We ended up moving to "dbcp2", in
> addition to optimizing the DB queries.
>


--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Christopher Schultz-2
In reply to this post by Ayub Khan
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ayub,

On 6/20/20 11:51, Ayub Khan wrote:

> Sorry we are using  8.0.32 version of tomcat.
>
> below is the configuration:
>
> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> Linux OS Version:     4.4.0-1087-aws Architecture:   amd64 JVM
> Version:    1.8.0_181-b13 JVM Vendor:     Oracle Corporation
>
> I use the below command to check the file descriptors:
>
> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

So you know there is some kind of increase in file-handle use, but you
don't know what types of file handles are increasing, right?

Can you try to find out which kinds of file handles are increasing?

I have a sneaking suspicion that it's your database connections and
not actually files open on the disk.

Are you using a database connection pool? If not, you should really
use one and limit the number of connections to something sane. If you
are using one, are you monitoring it to see how many connections are
actually being used? Are you sure you are using proper resource
management[1]? Even a single code-path that leaks connections can leak
them quickly under load.

> When there an issue related to broken files, this value keeps
> increasing, the only way to bring it down is to remove vm instance
> from AWS load balancer.> Which version of tomcat should I install
> ?

Tomcat 8.0.x hasn't been supported since its last release on 29 June
2018. That was 8.0.53. Your release is from 8 February 2016 and is
dangerously out of date (unless you are using the Ubuntu-packaged
version, in which case I hope they kept-up with security patches thee
past 4 years).

- -chris

> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> [hidden email]> wrote:
>
> Ayub,
>
> On 6/19/20 16:46, Ayub Khan wrote:
>>>> tomcat 8.5 broken pipe increases open files on ubuntu AWS
>
> Which exact version of Tomcat 8.5? If you aren't running the
> latest version (8.5.56), please upgrade and re-test.
>
>>>> If there is slow response from db I see this stack trace and
>>>> the open files goes high and the only way to open files go
>>>> down is to remove the instance from Amazon load balancer.
>>>>
>>>> Is there a way to keep the open files low even when Broken
>>>> pipe error is thrown ?
>
> What is your evidence that file handles are being left open?
>
> Which file handles are being left open?
>
> -chris
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7vlj0ACgkQHPApP6U8
pFiCqBAAqDO2+sUHpjcL8c5qwQbc0EG1XjgawI/ggjegVNSLTZ6mjdCVmxU3V10T
IAdAmckT+o5qkZKQSVAQHpNW7i5A21U0dOUF1r1EWq0WZIN3fE1dcpQVsOlvUSqD
qZ+gnGzg3UasSMJFyOj2hnU1+PBnhSFWvKcHwTda9gelqmgHp24rwpCvHFFFmH+2
I/QrHoQc1W9F0bUmGOsfBy5+eC981KMUfZrIJmUBQFOhopaD19W+yofZCl0S15E7
dxfIcqvFSF7FzZ1FraKGVd+nEUlW6PLuNqHkqpNGIstMWuQuo1RkukdXeP3FqjeA
36W5GOo6tlJTPGjbGaVPLnKw+hQuQa9he5Dv4E03C9vk1cbU6g+3hguKGhIfV3FM
2Pg4SE4LwVv3LMY1uxuuHGPgrs6CqHmMaDA6FQARhm/5iD5WJQBrXwzCYztHK/Z+
+QILfdBa1VZl9qyujBneY9oJ9yc7HeBb1DDZAAqZFrOJctRmDrdZQrUwZ+b6vUzL
kBr7fCSnzEqUbPzv5DiY6qfD2x2f1GfJFLY6LuJ36g4P0YSItRTaR2nX/VOzVM9D
oZsDFZhlSZa0U7KpdaQDpi1dw6wgW83gfc7GxrbYvQsgXZPnm25JeU/UI5p1O+cQ
q+iynFZ8poxSOIjx+2A9anwCXAiiWbyVRJ6Rmy2NwkCbXH/VhwY=
=oNTV
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Christopher Schultz-2
In reply to this post by calder
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Calder,

On 6/20/20 13:24, calder wrote:

> On Fri, Jun 19, 2020, 15:46 Ayub Khan <[hidden email]> wrote:
>
>> tomcat 8.5 broken pipe increases open files on ubuntu AWS
>>
>
>
> If there is slow response from db
>
>
> Might be a good idea to investigate the reason for the "slow
> response"
>
> I see this stack trace and the open files goes high
>
>
> [ snip ]
>
>
> Caused by: java.io.IOException: Broken pipe
>> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>
>
>
> If I remember correctly, we had this issue about three+ years ago.
> Is your app using the "dbcp" (1.4) library?  We ended up moving to
> "dbcp2", in addition to optimizing the DB queries.

DBCP 2 is preferred to DBCP 1, but not because of bugs. It's preferred
because of an improved architecture, performance, etc.

The version of DBCP that comes bundled with Tomcat is usually all
anyone really needs.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7vlowACgkQHPApP6U8
pFiOThAAq6dba6E/h49MP2e1ArNOEv5lsJf3oxaZFSlbLfYMG/WnKXve1s03/Wsy
tJJXVuzaDvVigs9yue+ucZw0bdZGAEtAS5g/Qw3GMjc2tCs6ze1MA5hQLHqXbOM3
gU5KRkyn5TBTCF/vB6EPSr25RQeuZLjFquDrhyaLn5QPsJ1np+vfCHezLjt4L99m
oa4ouequVMtSB+9SA9+3/4xym+Fw7Vba4t5Bx6uUqdNIeIjwF2PXOtmEOB+2HVTq
ijCvC7yBOvHdtGr79KLV7Eukg7uOriTNvSXweI809ufLny9SZ4+pwQp/wMKlFdof
uMBKKwBngllFSZvVtpjm4I4IrhNr0fzzI8OlLBck5pLFNP02mOjsEwM150gIJ11g
15nrdf3F8Xfv4SEsCVasTHNQSuBEfhb0qNNfO/0OKdQQuHCCdGHCJ3ILmyWKSBPp
tOb7yK8MwescUkzfAP82D1Lw796XLKyTdTuNcqt3IIHSrPmaO4beSQFafPPKZnlP
LqU+IG8Eiq+vOkQQpWbaYv2zF7F2Grpr6XGO29/nu7Pu2drQyAO4YFtxji8k4xfu
LOeaxQ53/K4WDrZVgh1wZ+7T2sQzEuLYUnpaSJIuwXXOhBnTP7s6CLTL1ule997o
93CHE9m6j2XBIioHy15Y0MkL4qlHorf6qIHANxsmdFwhSMYJ9j4=
=kkHx
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
In reply to this post by Christopher Schultz-2
Chris,

I am using HikariCP for connection pooling. If the database is leaking
connections then I should see connection not available exception.

How do I find out which file descriptors are leaking ?  these are not files
open on disk as there is no explicit disk file I/O in this application.

I just use the below command to check for open file descriptors:

watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

Thanks and Regards
Ayub

On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Ayub,
>
> On 6/20/20 11:51, Ayub Khan wrote:
> > Sorry we are using  8.0.32 version of tomcat.
> >
> > below is the configuration:
> >
> > Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> > 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> > Linux OS Version:     4.4.0-1087-aws Architecture:   amd64 JVM
> > Version:    1.8.0_181-b13 JVM Vendor:     Oracle Corporation
> >
> > I use the below command to check the file descriptors:
> >
> > watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> So you know there is some kind of increase in file-handle use, but you
> don't know what types of file handles are increasing, right?
>
> Can you try to find out which kinds of file handles are increasing?
>
> I have a sneaking suspicion that it's your database connections and
> not actually files open on the disk.
>
> Are you using a database connection pool? If not, you should really
> use one and limit the number of connections to something sane. If you
> are using one, are you monitoring it to see how many connections are
> actually being used? Are you sure you are using proper resource
> management[1]? Even a single code-path that leaks connections can leak
> them quickly under load.
>
> > When there an issue related to broken files, this value keeps
> > increasing, the only way to bring it down is to remove vm instance
> > from AWS load balancer.> Which version of tomcat should I install
> > ?
>
> Tomcat 8.0.x hasn't been supported since its last release on 29 June
> 2018. That was 8.0.53. Your release is from 8 February 2016 and is
> dangerously out of date (unless you are using the Ubuntu-packaged
> version, in which case I hope they kept-up with security patches thee
> past 4 years).
>
> - -chris
>
> > On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> > [hidden email]> wrote:
> >
> > Ayub,
> >
> > On 6/19/20 16:46, Ayub Khan wrote:
> >>>> tomcat 8.5 broken pipe increases open files on ubuntu AWS
> >
> > Which exact version of Tomcat 8.5? If you aren't running the
> > latest version (8.5.56), please upgrade and re-test.
> >
> >>>> If there is slow response from db I see this stack trace and
> >>>> the open files goes high and the only way to open files go
> >>>> down is to remove the instance from Amazon load balancer.
> >>>>
> >>>> Is there a way to keep the open files low even when Broken
> >>>> pipe error is thrown ?
> >
> > What is your evidence that file handles are being left open?
> >
> > Which file handles are being left open?
> >
> > -chris
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7vlj0ACgkQHPApP6U8
> pFiCqBAAqDO2+sUHpjcL8c5qwQbc0EG1XjgawI/ggjegVNSLTZ6mjdCVmxU3V10T
> IAdAmckT+o5qkZKQSVAQHpNW7i5A21U0dOUF1r1EWq0WZIN3fE1dcpQVsOlvUSqD
> qZ+gnGzg3UasSMJFyOj2hnU1+PBnhSFWvKcHwTda9gelqmgHp24rwpCvHFFFmH+2
> I/QrHoQc1W9F0bUmGOsfBy5+eC981KMUfZrIJmUBQFOhopaD19W+yofZCl0S15E7
> dxfIcqvFSF7FzZ1FraKGVd+nEUlW6PLuNqHkqpNGIstMWuQuo1RkukdXeP3FqjeA
> 36W5GOo6tlJTPGjbGaVPLnKw+hQuQa9he5Dv4E03C9vk1cbU6g+3hguKGhIfV3FM
> 2Pg4SE4LwVv3LMY1uxuuHGPgrs6CqHmMaDA6FQARhm/5iD5WJQBrXwzCYztHK/Z+
> +QILfdBa1VZl9qyujBneY9oJ9yc7HeBb1DDZAAqZFrOJctRmDrdZQrUwZ+b6vUzL
> kBr7fCSnzEqUbPzv5DiY6qfD2x2f1GfJFLY6LuJ36g4P0YSItRTaR2nX/VOzVM9D
> oZsDFZhlSZa0U7KpdaQDpi1dw6wgW83gfc7GxrbYvQsgXZPnm25JeU/UI5p1O+cQ
> q+iynFZ8poxSOIjx+2A9anwCXAiiWbyVRJ6Rmy2NwkCbXH/VhwY=
> =oNTV
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Felix Schumacher

Am 22.06.20 um 11:41 schrieb Ayub Khan:
> Chris,
>
> I am using HikariCP for connection pooling. If the database is leaking
> connections then I should see connection not available exception.
>
> How do I find out which file descriptors are leaking ?  these are not
files
> open on disk as there is no explicit disk file I/O in this application.
>
> I just use the below command to check for open file descriptors:
>

> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

You could have a look at the name of the files in the pids proc directory.

 $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/

Or you could use the tool lsof to find the open file descriptors.

 $ lsof -p $(cat /var/run/tomcat8.pid)

For both calls you should first change to the uid of the tomcat user or
use sudo as in your example.

Felix

>
> Thanks and Regards
> Ayub
>
> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> [hidden email]> wrote:
>
> Ayub,
>
> On 6/20/20 11:51, Ayub Khan wrote:
> >>> Sorry we are using  8.0.32 version of tomcat.
> >>>
> >>> below is the configuration:
> >>>
> >>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> >>> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> >>> Linux OS Version:     4.4.0-1087-aws Architecture:   amd64 JVM
> >>> Version:    1.8.0_181-b13 JVM Vendor:     Oracle Corporation
> >>>
> >>> I use the below command to check the file descriptors:
> >>>
> >>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> So you know there is some kind of increase in file-handle use, but you
> don't know what types of file handles are increasing, right?
>
> Can you try to find out which kinds of file handles are increasing?
>
> I have a sneaking suspicion that it's your database connections and
> not actually files open on the disk.
>
> Are you using a database connection pool? If not, you should really
> use one and limit the number of connections to something sane. If you
> are using one, are you monitoring it to see how many connections are
> actually being used? Are you sure you are using proper resource
> management[1]? Even a single code-path that leaks connections can leak
> them quickly under load.
>
> >>> When there an issue related to broken files, this value keeps
> >>> increasing, the only way to bring it down is to remove vm instance
> >>> from AWS load balancer.> Which version of tomcat should I install
> >>> ?
>
> Tomcat 8.0.x hasn't been supported since its last release on 29 June
> 2018. That was 8.0.53. Your release is from 8 February 2016 and is
> dangerously out of date (unless you are using the Ubuntu-packaged
> version, in which case I hope they kept-up with security patches thee
> past 4 years).
>
> -chris
>
> >>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> >>> [hidden email]> wrote:
> >>>
> >>> Ayub,
> >>>
> >>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>> tomcat 8.5 broken pipe increases open files on ubuntu AWS
> >>>
> >>> Which exact version of Tomcat 8.5? If you aren't running the
> >>> latest version (8.5.56), please upgrade and re-test.
> >>>
> >>>>>> If there is slow response from db I see this stack trace and
> >>>>>> the open files goes high and the only way to open files go
> >>>>>> down is to remove the instance from Amazon load balancer.
> >>>>>>
> >>>>>> Is there a way to keep the open files low even when Broken
> >>>>>> pipe error is thrown ?
> >>>
> >>> What is your evidence that file handles are being left open?
> >>>
> >>> Which file handles are being left open?
> >>>
> >>> -chris
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>>
> >>>>
> To unsubscribe, e-mail: [hidden email]
> >>>> For additional commands, e-mail: [hidden email]
> >>>>
> >>>>
> >>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
Felix,

I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and  from the output
I see majority of them are related to sockets as shown below, some of them
point to the jar file of tomcat and others to the log file which is created.

 socket:[2084570754]
 socket:[2084579487]
 socket:[2084578478]
socket:[2084570167]

On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
[hidden email]> wrote:

>
> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> > Chris,
> >
> > I am using HikariCP for connection pooling. If the database is leaking
> > connections then I should see connection not available exception.
> >
> > How do I find out which file descriptors are leaking ?  these are not
> files
> > open on disk as there is no explicit disk file I/O in this application.
> >
> > I just use the below command to check for open file descriptors:
> >
>
> > watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> You could have a look at the name of the files in the pids proc directory.
>
>  $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>
> Or you could use the tool lsof to find the open file descriptors.
>
>  $ lsof -p $(cat /var/run/tomcat8.pid)
>
> For both calls you should first change to the uid of the tomcat user or
> use sudo as in your example.
>
> Felix
>
> >
> > Thanks and Regards
> > Ayub
> >
> > On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> > [hidden email]> wrote:
> >
> > Ayub,
> >
> > On 6/20/20 11:51, Ayub Khan wrote:
> > >>> Sorry we are using  8.0.32 version of tomcat.
> > >>>
> > >>> below is the configuration:
> > >>>
> > >>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> > >>> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> > >>> Linux OS Version:     4.4.0-1087-aws Architecture:   amd64 JVM
> > >>> Version:    1.8.0_181-b13 JVM Vendor:     Oracle Corporation
> > >>>
> > >>> I use the below command to check the file descriptors:
> > >>>
> > >>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
> >
> > So you know there is some kind of increase in file-handle use, but you
> > don't know what types of file handles are increasing, right?
> >
> > Can you try to find out which kinds of file handles are increasing?
> >
> > I have a sneaking suspicion that it's your database connections and
> > not actually files open on the disk.
> >
> > Are you using a database connection pool? If not, you should really
> > use one and limit the number of connections to something sane. If you
> > are using one, are you monitoring it to see how many connections are
> > actually being used? Are you sure you are using proper resource
> > management[1]? Even a single code-path that leaks connections can leak
> > them quickly under load.
> >
> > >>> When there an issue related to broken files, this value keeps
> > >>> increasing, the only way to bring it down is to remove vm instance
> > >>> from AWS load balancer.> Which version of tomcat should I install
> > >>> ?
> >
> > Tomcat 8.0.x hasn't been supported since its last release on 29 June
> > 2018. That was 8.0.53. Your release is from 8 February 2016 and is
> > dangerously out of date (unless you are using the Ubuntu-packaged
> > version, in which case I hope they kept-up with security patches thee
> > past 4 years).
> >
> > -chris
> >
> > >>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> > >>> [hidden email]> wrote:
> > >>>
> > >>> Ayub,
> > >>>
> > >>> On 6/19/20 16:46, Ayub Khan wrote:
> > >>>>>> tomcat 8.5 broken pipe increases open files on ubuntu AWS
> > >>>
> > >>> Which exact version of Tomcat 8.5? If you aren't running the
> > >>> latest version (8.5.56), please upgrade and re-test.
> > >>>
> > >>>>>> If there is slow response from db I see this stack trace and
> > >>>>>> the open files goes high and the only way to open files go
> > >>>>>> down is to remove the instance from Amazon load balancer.
> > >>>>>>
> > >>>>>> Is there a way to keep the open files low even when Broken
> > >>>>>> pipe error is thrown ?
> > >>>
> > >>> What is your evidence that file handles are being left open?
> > >>>
> > >>> Which file handles are being left open?
> > >>>
> > >>> -chris
> > >>>>
> > >>>>
> ---------------------------------------------------------------------
> > >>>>
> > >>>>
> > To unsubscribe, e-mail: [hidden email]
> > >>>> For additional commands, e-mail: [hidden email]
> > >>>>
> > >>>>
> > >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Felix Schumacher

Am 22.06.20 um 13:22 schrieb Ayub Khan:

> Felix,
>
> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and  from the output
> I see majority of them are related to sockets as shown below, some of them
> point to the jar file of tomcat and others to the log file which is created.
>
>  socket:[2084570754]
>  socket:[2084579487]
>  socket:[2084578478]
> socket:[2084570167]

Can you try the other command (lsof -p $(cat ...tomcat.pid))? It should
give a bit more details on the used sockets that the proc directory.

Felix

>
> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> [hidden email]> wrote:
>
>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>>> Chris,
>>>
>>> I am using HikariCP for connection pooling. If the database is leaking
>>> connections then I should see connection not available exception.
>>>
>>> How do I find out which file descriptors are leaking ?  these are not
>> files
>>> open on disk as there is no explicit disk file I/O in this application.
>>>
>>> I just use the below command to check for open file descriptors:
>>>
>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>> You could have a look at the name of the files in the pids proc directory.
>>
>>  $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>
>> Or you could use the tool lsof to find the open file descriptors.
>>
>>  $ lsof -p $(cat /var/run/tomcat8.pid)
>>
>> For both calls you should first change to the uid of the tomcat user or
>> use sudo as in your example.
>>
>> Felix
>>
>>> Thanks and Regards
>>> Ayub
>>>
>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
>>> [hidden email]> wrote:
>>>
>>> Ayub,
>>>
>>> On 6/20/20 11:51, Ayub Khan wrote:
>>>>>> Sorry we are using  8.0.32 version of tomcat.
>>>>>>
>>>>>> below is the configuration:
>>>>>>
>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
>>>>>> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
>>>>>> Linux OS Version:     4.4.0-1087-aws Architecture:   amd64 JVM
>>>>>> Version:    1.8.0_181-b13 JVM Vendor:     Oracle Corporation
>>>>>>
>>>>>> I use the below command to check the file descriptors:
>>>>>>
>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>>> So you know there is some kind of increase in file-handle use, but you
>>> don't know what types of file handles are increasing, right?
>>>
>>> Can you try to find out which kinds of file handles are increasing?
>>>
>>> I have a sneaking suspicion that it's your database connections and
>>> not actually files open on the disk.
>>>
>>> Are you using a database connection pool? If not, you should really
>>> use one and limit the number of connections to something sane. If you
>>> are using one, are you monitoring it to see how many connections are
>>> actually being used? Are you sure you are using proper resource
>>> management[1]? Even a single code-path that leaks connections can leak
>>> them quickly under load.
>>>
>>>>>> When there an issue related to broken files, this value keeps
>>>>>> increasing, the only way to bring it down is to remove vm instance
>>>>>> from AWS load balancer.> Which version of tomcat should I install
>>>>>> ?
>>> Tomcat 8.0.x hasn't been supported since its last release on 29 June
>>> 2018. That was 8.0.53. Your release is from 8 February 2016 and is
>>> dangerously out of date (unless you are using the Ubuntu-packaged
>>> version, in which case I hope they kept-up with security patches thee
>>> past 4 years).
>>>
>>> -chris
>>>
>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>> Ayub,
>>>>>>
>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>>>>>>>>> tomcat 8.5 broken pipe increases open files on ubuntu AWS
>>>>>> Which exact version of Tomcat 8.5? If you aren't running the
>>>>>> latest version (8.5.56), please upgrade and re-test.
>>>>>>
>>>>>>>>> If there is slow response from db I see this stack trace and
>>>>>>>>> the open files goes high and the only way to open files go
>>>>>>>>> down is to remove the instance from Amazon load balancer.
>>>>>>>>>
>>>>>>>>> Is there a way to keep the open files low even when Broken
>>>>>>>>> pipe error is thrown ?
>>>>>> What is your evidence that file handles are being left open?
>>>>>>
>>>>>> Which file handles are being left open?
>>>>>>
>>>>>> -chris
>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>
>>> To unsubscribe, e-mail: [hidden email]
>>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>>
>>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Christopher Schultz-2
In reply to this post by Ayub Khan
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Ayub,

On 6/22/20 05:41, Ayub Khan wrote:
> I am using HikariCP for connection pooling. If the database is
> leaking connections then I should see connection not available
> exception.

That probably depends upon your connection pool configuration. It's
definitely possible to configure a connection pool to leak connections
without complaint.

> How do I find out which file descriptors are leaking ?  these are
> not files open on disk as there is no explicit disk file I/O in
> this application.
You might have to use a profiler.

> I just use the below command to check for open file descriptors:
>
> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"

Felix's suggestions for how to check the types of file handles are
good. You should be able to get host:port information from the sockets
you are seeing and, I suspect, you'll find that they all point to your
database.

- -chris

> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> [hidden email]> wrote:
>
> Ayub,
>
> On 6/20/20 11:51, Ayub Khan wrote:
>>>> Sorry we are using  8.0.32 version of tomcat.
>>>>
>>>> below is the configuration:
>>>>
>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:
>>>> Jan 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
>>>> Linux OS Version:     4.4.0-1087-aws Architecture:   amd64
>>>> JVM Version:    1.8.0_181-b13 JVM Vendor:     Oracle
>>>> Corporation
>>>>
>>>> I use the below command to check the file descriptors:
>>>>
>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>
> So you know there is some kind of increase in file-handle use, but
> you don't know what types of file handles are increasing, right?
>
> Can you try to find out which kinds of file handles are
> increasing?
>
> I have a sneaking suspicion that it's your database connections
> and not actually files open on the disk.
>
> Are you using a database connection pool? If not, you should
> really use one and limit the number of connections to something
> sane. If you are using one, are you monitoring it to see how many
> connections are actually being used? Are you sure you are using
> proper resource management[1]? Even a single code-path that leaks
> connections can leak them quickly under load.
>
>>>> When there an issue related to broken files, this value
>>>> keeps increasing, the only way to bring it down is to remove
>>>> vm instance from AWS load balancer.> Which version of tomcat
>>>> should I install ?
>
> Tomcat 8.0.x hasn't been supported since its last release on 29
> June 2018. That was 8.0.53. Your release is from 8 February 2016
> and is dangerously out of date (unless you are using the
> Ubuntu-packaged version, in which case I hope they kept-up with
> security patches thee past 4 years).
>
> -chris
>
>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
>>>> [hidden email]> wrote:
>>>>
>>>> Ayub,
>>>>
>>>> On 6/19/20 16:46, Ayub Khan wrote:
>>>>>>> tomcat 8.5 broken pipe increases open files on ubuntu
>>>>>>> AWS
>>>>
>>>> Which exact version of Tomcat 8.5? If you aren't running the
>>>> latest version (8.5.56), please upgrade and re-test.
>>>>
>>>>>>> If there is slow response from db I see this stack
>>>>>>> trace and the open files goes high and the only way to
>>>>>>> open files go down is to remove the instance from
>>>>>>> Amazon load balancer.
>>>>>>>
>>>>>>> Is there a way to keep the open files low even when
>>>>>>> Broken pipe error is thrown ?
>>>>
>>>> What is your evidence that file handles are being left open?
>>>>
>>>> Which file handles are being left open?
>>>>
>>>> -chris
>>>>>
>>>>> ------------------------------------------------------------------
- ---
>>>>>
>>>>>
>
>>>>>
To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail:
>>>>> [hidden email]
>>>>>
>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7w0U0ACgkQHPApP6U8
pFgnCw/9GxKtFIdNS71zYPAYgEY0O6bh4+SyvD7cWsNRwAEtN4jwwmvmbk+9ldGa
5u8jalMeWjBaYtVev+T7XQ0EOJN9ZuunyXHzKR8LeIqPCzka/34inQKBhlwOFyR+
lQUhod8VsDrQJ4lb2ZxC1yS/mnUFdKeV2u2Px9mIr5WdkfVAxYWCOuKTW9DrW3HE
nbPG5t3oiLS8Rx+xXssyaTii5DAaQm0X0FYkkPp1GQ8Tj0/fylsE17nV8LrxLTb5
qhv0l6avR2/IsXaD6DC8MjdUO6kz5hkWNLtO/WcKrIFI+xPcYs2+Hog7Hrajetwp
QR8f7R7ry0W0HLy3p3R10jMwDQzOWjn2RNxQqL2qfNGmOqMbTi2XskQhtkKPoM9Y
LqZJsG0Ax2kNr6N4YeYpotRVcGqCJDgLwQ3HMjkHW/Y1M4O2NicphW9iFd5L+X9Z
fh4KFz5XbAUM4py7IuMtj1SH/FrOMyfBtKl6ECJ98VbtoBPU0I6Kmc0lgbf3h9wD
yphZ+3pW6a9Oeroq7aRTd35XdFqLdi9eg4Ze19/KpAr0hw3pgPvQqZiqq5qhpOrv
HhNWfHhvzeXmmab0DeLB9On+FDp30svWv9hmAR1ZB8ViBeCDoFjaUxv1UltqS6NQ
m9UsvtNR3035EpiSw9p5Jh5+eiG6swSnJXQH4yZxkVNAsHthyBA=
=KsIE
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
In reply to this post by Felix Schumacher
Felix,

I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)  *and I saw the below
output, some in CLOSE_WAIT and others in ESTABLISHED. If there are 200 open
file descriptors 160 are in CLOSE_WAIT state. When the count for CLOSE_WAIT
increases I just have to restart tomcat.

java    65189 tomcat8  715u     IPv6          237878311       0t0       TCP
localhost:http-alt->localhost:43760 (CLOSE_WAIT)
java    65189 tomcat8  716u     IPv6          237848923       0t0       TCP
localhost:http-alt->localhost:40568 (CLOSE_WAIT)

most of the open files are in CLOSE_WAIT state I do not see anything
related to database ip.



On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
[hidden email]> wrote:

>
> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> > Felix,
> >
> > I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and  from the
> output
> > I see majority of them are related to sockets as shown below, some of
> them
> > point to the jar file of tomcat and others to the log file which is
> created.
> >
> >  socket:[2084570754]
> >  socket:[2084579487]
> >  socket:[2084578478]
> > socket:[2084570167]
>
> Can you try the other command (lsof -p $(cat ...tomcat.pid))? It should
> give a bit more details on the used sockets that the proc directory.
>
> Felix
>
> >
> > On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> > [hidden email]> wrote:
> >
> >> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>> Chris,
> >>>
> >>> I am using HikariCP for connection pooling. If the database is leaking
> >>> connections then I should see connection not available exception.
> >>>
> >>> How do I find out which file descriptors are leaking ?  these are not
> >> files
> >>> open on disk as there is no explicit disk file I/O in this application.
> >>>
> >>> I just use the below command to check for open file descriptors:
> >>>
> >>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
> >> You could have a look at the name of the files in the pids proc
> directory.
> >>
> >>  $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>
> >> Or you could use the tool lsof to find the open file descriptors.
> >>
> >>  $ lsof -p $(cat /var/run/tomcat8.pid)
> >>
> >> For both calls you should first change to the uid of the tomcat user or
> >> use sudo as in your example.
> >>
> >> Felix
> >>
> >>> Thanks and Regards
> >>> Ayub
> >>>
> >>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> >>> [hidden email]> wrote:
> >>>
> >>> Ayub,
> >>>
> >>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>> Sorry we are using  8.0.32 version of tomcat.
> >>>>>>
> >>>>>> below is the configuration:
> >>>>>>
> >>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server built:   Jan
> >>>>>> 24 2020 16:24:30 UTC Server number:  8.0.32.0 OS Name:
> >>>>>> Linux OS Version:     4.4.0-1087-aws Architecture:   amd64 JVM
> >>>>>> Version:    1.8.0_181-b13 JVM Vendor:     Oracle Corporation
> >>>>>>
> >>>>>> I use the below command to check the file descriptors:
> >>>>>>
> >>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
> >>> So you know there is some kind of increase in file-handle use, but you
> >>> don't know what types of file handles are increasing, right?
> >>>
> >>> Can you try to find out which kinds of file handles are increasing?
> >>>
> >>> I have a sneaking suspicion that it's your database connections and
> >>> not actually files open on the disk.
> >>>
> >>> Are you using a database connection pool? If not, you should really
> >>> use one and limit the number of connections to something sane. If you
> >>> are using one, are you monitoring it to see how many connections are
> >>> actually being used? Are you sure you are using proper resource
> >>> management[1]? Even a single code-path that leaks connections can leak
> >>> them quickly under load.
> >>>
> >>>>>> When there an issue related to broken files, this value keeps
> >>>>>> increasing, the only way to bring it down is to remove vm instance
> >>>>>> from AWS load balancer.> Which version of tomcat should I install
> >>>>>> ?
> >>> Tomcat 8.0.x hasn't been supported since its last release on 29 June
> >>> 2018. That was 8.0.53. Your release is from 8 February 2016 and is
> >>> dangerously out of date (unless you are using the Ubuntu-packaged
> >>> version, in which case I hope they kept-up with security patches thee
> >>> past 4 years).
> >>>
> >>> -chris
> >>>
> >>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz <
> >>>>>> [hidden email]> wrote:
> >>>>>>
> >>>>>> Ayub,
> >>>>>>
> >>>>>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>>>>> tomcat 8.5 broken pipe increases open files on ubuntu AWS
> >>>>>> Which exact version of Tomcat 8.5? If you aren't running the
> >>>>>> latest version (8.5.56), please upgrade and re-test.
> >>>>>>
> >>>>>>>>> If there is slow response from db I see this stack trace and
> >>>>>>>>> the open files goes high and the only way to open files go
> >>>>>>>>> down is to remove the instance from Amazon load balancer.
> >>>>>>>>>
> >>>>>>>>> Is there a way to keep the open files low even when Broken
> >>>>>>>>> pipe error is thrown ?
> >>>>>> What is your evidence that file handles are being left open?
> >>>>>>
> >>>>>> Which file handles are being left open?
> >>>>>>
> >>>>>> -chris
> >>>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>
> >>> To unsubscribe, e-mail: [hidden email]
> >>>>>>> For additional commands, e-mail: [hidden email]
> >>>>>>>
> >>>>>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: [hidden email]
> >>>> For additional commands, e-mail: [hidden email]
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ayub,

On 6/23/20 16:23, Ayub Khan wrote:

> I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)  *and I saw
> the below output, some in CLOSE_WAIT and others in ESTABLISHED. If
> there are 200 open file descriptors 160 are in CLOSE_WAIT state.
> When the count for CLOSE_WAIT increases I just have to restart
> tomcat.
>
> java    65189 tomcat8  715u     IPv6          237878311       0t0
> TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java    65189
> tomcat8  716u     IPv6          237848923       0t0       TCP
> localhost:http-alt->localhost:40568 (CLOSE_WAIT)

These are connections from some process into Tomcat listening on port
8080 (that's what localhost:http-alt is). So what process owns the
outgoing connection on port 40568 on the same host?

Are you using a reverse proxy?

> most of the open files are in CLOSE_WAIT state I do not see
> anything related to database ip.

Agreed. It looks like you have a reverse proxy who is losing-track of
connections, or who is (re)opening connections when it may be unnecessar
y.

Can you share your <Connector> configuration from server.xml? Remember
to remove any secrets.

- -chris

> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
> [hidden email]> wrote:
>
>>
>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
>>> Felix,
>>>
>>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and
>>> from the
>> output
>>> I see majority of them are related to sockets as shown below,
>>> some of
>> them
>>> point to the jar file of tomcat and others to the log file
>>> which is
>> created.
>>>
>>> socket:[2084570754] socket:[2084579487] socket:[2084578478]
>>> socket:[2084570167]
>>
>> Can you try the other command (lsof -p $(cat ...tomcat.pid))? It
>> should give a bit more details on the used sockets that the proc
>> directory.
>>
>> Felix
>>
>>>
>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
>>> [hidden email]> wrote:
>>>
>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>>>>> Chris,
>>>>>
>>>>> I am using HikariCP for connection pooling. If the database
>>>>> is leaking connections then I should see connection not
>>>>> available exception.
>>>>>
>>>>> How do I find out which file descriptors are leaking ?
>>>>> these are not
>>>> files
>>>>> open on disk as there is no explicit disk file I/O in this
>>>>> application.
>>>>>
>>>>> I just use the below command to check for open file
>>>>> descriptors:
>>>>>
>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc
>>>>> -l"
>>>> You could have a look at the name of the files in the pids
>>>> proc
>> directory.
>>>>
>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>>>
>>>> Or you could use the tool lsof to find the open file
>>>> descriptors.
>>>>
>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
>>>>
>>>> For both calls you should first change to the uid of the
>>>> tomcat user or use sudo as in your example.
>>>>
>>>> Felix
>>>>
>>>>> Thanks and Regards Ayub
>>>>>
>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
>>>>> [hidden email]> wrote:
>>>>>
>>>>> Ayub,
>>>>>
>>>>> On 6/20/20 11:51, Ayub Khan wrote:
>>>>>>>> Sorry we are using  8.0.32 version of tomcat.
>>>>>>>>
>>>>>>>> below is the configuration:
>>>>>>>>
>>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server
>>>>>>>> built:   Jan 24 2020 16:24:30 UTC Server number:
>>>>>>>> 8.0.32.0 OS Name: Linux OS Version:
>>>>>>>> 4.4.0-1087-aws Architecture:   amd64 JVM Version:
>>>>>>>> 1.8.0_181-b13 JVM Vendor:     Oracle Corporation
>>>>>>>>
>>>>>>>> I use the below command to check the file
>>>>>>>> descriptors:
>>>>>>>>
>>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ |
>>>>>>>> wc -l"
>>>>> So you know there is some kind of increase in file-handle
>>>>> use, but you don't know what types of file handles are
>>>>> increasing, right?
>>>>>
>>>>> Can you try to find out which kinds of file handles are
>>>>> increasing?
>>>>>
>>>>> I have a sneaking suspicion that it's your database
>>>>> connections and not actually files open on the disk.
>>>>>
>>>>> Are you using a database connection pool? If not, you
>>>>> should really use one and limit the number of connections
>>>>> to something sane. If you are using one, are you monitoring
>>>>> it to see how many connections are actually being used? Are
>>>>> you sure you are using proper resource management[1]? Even
>>>>> a single code-path that leaks connections can leak them
>>>>> quickly under load.
>>>>>
>>>>>>>> When there an issue related to broken files, this
>>>>>>>> value keeps increasing, the only way to bring it down
>>>>>>>> is to remove vm instance from AWS load balancer.>
>>>>>>>> Which version of tomcat should I install ?
>>>>> Tomcat 8.0.x hasn't been supported since its last release
>>>>> on 29 June 2018. That was 8.0.53. Your release is from 8
>>>>> February 2016 and is dangerously out of date (unless you
>>>>> are using the Ubuntu-packaged version, in which case I hope
>>>>> they kept-up with security patches thee past 4 years).
>>>>>
>>>>> -chris
>>>>>
>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz
>>>>>>>> < [hidden email]> wrote:
>>>>>>>>
>>>>>>>> Ayub,
>>>>>>>>
>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>>>>>>>>>>> tomcat 8.5 broken pipe increases open files on
>>>>>>>>>>> ubuntu AWS
>>>>>>>> Which exact version of Tomcat 8.5? If you aren't
>>>>>>>> running the latest version (8.5.56), please upgrade
>>>>>>>> and re-test.
>>>>>>>>
>>>>>>>>>>> If there is slow response from db I see this
>>>>>>>>>>> stack trace and the open files goes high and
>>>>>>>>>>> the only way to open files go down is to remove
>>>>>>>>>>> the instance from Amazon load balancer.
>>>>>>>>>>>
>>>>>>>>>>> Is there a way to keep the open files low even
>>>>>>>>>>> when Broken pipe error is thrown ?
>>>>>>>> What is your evidence that file handles are being
>>>>>>>> left open?
>>>>>>>>
>>>>>>>> Which file handles are being left open?
>>>>>>>>
>>>>>>>> -chris
>>>>>>>>>
>>>> -------------------------------------------------------------------
- --
>>>>>>>>>
>>>>>
>>>>
To unsubscribe, e-mail: [hidden email]
>>>>>>>>> For additional commands, e-mail:
>>>>>>>>> [hidden email]
>>>>>>>>>
>>>>>>>>>
>>>>>> -----------------------------------------------------------------
- ----
>>>>>>
>>>>>>
To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail:
>>>>>> [hidden email]
>>>>>>
>>>>>>
>>>>
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail:
>>>> [hidden email]
>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7yeJYACgkQHPApP6U8
pFhaHQ/9E56+1vZwl0kb3p/74AwmPkQ0wRJB08CYNpAxLHCDaQ4PJrMqGQDBfFcs
RT/ithuxS8qJXyd02YgF6hWs8OtiAkODsPqjNxhuQMrLuF7qt2x0CgPWSfMKZVLW
gLNA09kI6pQzfQrQROhfcqBQqGzaHVN8W4m+wTYOyCbMi2IjBF6lxcMtlCw7org/
AFe904+ST8D2O+S8CK97dP+74oIu1gEJniLxs6kypx9+qzpgesmhhd6H/965XCBp
ZEK6AwqedGUUPXxqxRdp6YmqflddtZdL4W1GhqbKN055+xGL+6rIlHD9W//ERJE8
JutSaX6K5Z1t8+6K48Fwkju549g63J099fKQftg5hsSZ8Xwuqo9B3qOn2HcqrSZA
+pgkdDDY5m0cHnB71GEREikABU36YnhBDeI0O4ocoQlRNnnIMAfb4hFxdwRCp5Qs
dRM/DyFwRvdbiOQ0hfoUhx9HDH6JRsWrs5m2hy4n9m+5DJcQ/fxjUcqq0LTzIP3v
upu81THiYVKBgbKpLKxadQnAzCj+TshEVF/E5D7ZWKG+enJAmnfrnIj5kKG/65HV
RozUoE69+p1xWN6mWtZq8hMVjKfX1p4z+XlgTg03K/VcdNvs5gE4aKrevx1Ueueb
hjogi6+JlQA7ysDlxHsttXrGVaY4DV9/c3ig8g7/pDQJCuoFe78=
=BMkY
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
Chris,

Yes we have nginx as reverse proxy, below is the nginx config. We notice
this issue only when there is high number of requests, during non peak
hours we do not see this issue.

location /myapp/myservice{
    #local machine
    proxy_pass          <a href="http://localhost:8080;">http://localhost:8080;
    proxy_http_version  1.1;

    proxy_set_header    Connection          $connection_upgrade;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;


        proxy_buffers 16 16k;
        proxy_buffer_size 32k;
}

location / {
#  if using AWS Load balancer, this bit checks for the presence of the
https proto flag.  if regular http is found, then issue a redirect to hit
the https endpoint instead
    if ($http_x_forwarded_proto != 'https') {
        rewrite ^ https://$host$request_uri? permanent;
    }

    proxy_pass          <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
    proxy_http_version  1.1;

    proxy_set_header    Connection          $connection_upgrade;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;


        proxy_buffers 16 16k;
        proxy_buffer_size 32k;
}

*below is the connector*

<Connector port="8080"
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               connectionTimeout="2000" maxThreads="50000"
               URIEncoding="UTF-8"
               redirectPort="8443" />


these ports are random, I am not sure who owns the process.

localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port 55866 is a
random port.



On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Ayub,
>
> On 6/23/20 16:23, Ayub Khan wrote:
> > I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)  *and I saw
> > the below output, some in CLOSE_WAIT and others in ESTABLISHED. If
> > there are 200 open file descriptors 160 are in CLOSE_WAIT state.
> > When the count for CLOSE_WAIT increases I just have to restart
> > tomcat.
> >
> > java    65189 tomcat8  715u     IPv6          237878311       0t0
> > TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java    65189
> > tomcat8  716u     IPv6          237848923       0t0       TCP
> > localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>
> These are connections from some process into Tomcat listening on port
> 8080 (that's what localhost:http-alt is). So what process owns the
> outgoing connection on port 40568 on the same host?
>
> Are you using a reverse proxy?
>
> > most of the open files are in CLOSE_WAIT state I do not see
> > anything related to database ip.
>
> Agreed. It looks like you have a reverse proxy who is losing-track of
> connections, or who is (re)opening connections when it may be unnecessar
> y.
>
> Can you share your <Connector> configuration from server.xml? Remember
> to remove any secrets.
>
> - -chris
>
> > On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
> > [hidden email]> wrote:
> >
> >>
> >> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> >>> Felix,
> >>>
> >>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/ and
> >>> from the
> >> output
> >>> I see majority of them are related to sockets as shown below,
> >>> some of
> >> them
> >>> point to the jar file of tomcat and others to the log file
> >>> which is
> >> created.
> >>>
> >>> socket:[2084570754] socket:[2084579487] socket:[2084578478]
> >>> socket:[2084570167]
> >>
> >> Can you try the other command (lsof -p $(cat ...tomcat.pid))? It
> >> should give a bit more details on the used sockets that the proc
> >> directory.
> >>
> >> Felix
> >>
> >>>
> >>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> >>> [hidden email]> wrote:
> >>>
> >>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>>>> Chris,
> >>>>>
> >>>>> I am using HikariCP for connection pooling. If the database
> >>>>> is leaking connections then I should see connection not
> >>>>> available exception.
> >>>>>
> >>>>> How do I find out which file descriptors are leaking ?
> >>>>> these are not
> >>>> files
> >>>>> open on disk as there is no explicit disk file I/O in this
> >>>>> application.
> >>>>>
> >>>>> I just use the below command to check for open file
> >>>>> descriptors:
> >>>>>
> >>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc
> >>>>> -l"
> >>>> You could have a look at the name of the files in the pids
> >>>> proc
> >> directory.
> >>>>
> >>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>
> >>>> Or you could use the tool lsof to find the open file
> >>>> descriptors.
> >>>>
> >>>> $ lsof -p $(cat /var/run/tomcat8.pid)
> >>>>
> >>>> For both calls you should first change to the uid of the
> >>>> tomcat user or use sudo as in your example.
> >>>>
> >>>> Felix
> >>>>
> >>>>> Thanks and Regards Ayub
> >>>>>
> >>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz <
> >>>>> [hidden email]> wrote:
> >>>>>
> >>>>> Ayub,
> >>>>>
> >>>>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>>>> Sorry we are using  8.0.32 version of tomcat.
> >>>>>>>>
> >>>>>>>> below is the configuration:
> >>>>>>>>
> >>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu) Server
> >>>>>>>> built:   Jan 24 2020 16:24:30 UTC Server number:
> >>>>>>>> 8.0.32.0 OS Name: Linux OS Version:
> >>>>>>>> 4.4.0-1087-aws Architecture:   amd64 JVM Version:
> >>>>>>>> 1.8.0_181-b13 JVM Vendor:     Oracle Corporation
> >>>>>>>>
> >>>>>>>> I use the below command to check the file
> >>>>>>>> descriptors:
> >>>>>>>>
> >>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ |
> >>>>>>>> wc -l"
> >>>>> So you know there is some kind of increase in file-handle
> >>>>> use, but you don't know what types of file handles are
> >>>>> increasing, right?
> >>>>>
> >>>>> Can you try to find out which kinds of file handles are
> >>>>> increasing?
> >>>>>
> >>>>> I have a sneaking suspicion that it's your database
> >>>>> connections and not actually files open on the disk.
> >>>>>
> >>>>> Are you using a database connection pool? If not, you
> >>>>> should really use one and limit the number of connections
> >>>>> to something sane. If you are using one, are you monitoring
> >>>>> it to see how many connections are actually being used? Are
> >>>>> you sure you are using proper resource management[1]? Even
> >>>>> a single code-path that leaks connections can leak them
> >>>>> quickly under load.
> >>>>>
> >>>>>>>> When there an issue related to broken files, this
> >>>>>>>> value keeps increasing, the only way to bring it down
> >>>>>>>> is to remove vm instance from AWS load balancer.>
> >>>>>>>> Which version of tomcat should I install ?
> >>>>> Tomcat 8.0.x hasn't been supported since its last release
> >>>>> on 29 June 2018. That was 8.0.53. Your release is from 8
> >>>>> February 2016 and is dangerously out of date (unless you
> >>>>> are using the Ubuntu-packaged version, in which case I hope
> >>>>> they kept-up with security patches thee past 4 years).
> >>>>>
> >>>>> -chris
> >>>>>
> >>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher Schultz
> >>>>>>>> < [hidden email]> wrote:
> >>>>>>>>
> >>>>>>>> Ayub,
> >>>>>>>>
> >>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>>>>>>> tomcat 8.5 broken pipe increases open files on
> >>>>>>>>>>> ubuntu AWS
> >>>>>>>> Which exact version of Tomcat 8.5? If you aren't
> >>>>>>>> running the latest version (8.5.56), please upgrade
> >>>>>>>> and re-test.
> >>>>>>>>
> >>>>>>>>>>> If there is slow response from db I see this
> >>>>>>>>>>> stack trace and the open files goes high and
> >>>>>>>>>>> the only way to open files go down is to remove
> >>>>>>>>>>> the instance from Amazon load balancer.
> >>>>>>>>>>>
> >>>>>>>>>>> Is there a way to keep the open files low even
> >>>>>>>>>>> when Broken pipe error is thrown ?
> >>>>>>>> What is your evidence that file handles are being
> >>>>>>>> left open?
> >>>>>>>>
> >>>>>>>> Which file handles are being left open?
> >>>>>>>>
> >>>>>>>> -chris
> >>>>>>>>>
> >>>> -------------------------------------------------------------------
> - --
> >>>>>>>>>
> >>>>>
> >>>>
> To unsubscribe, e-mail: [hidden email]
> >>>>>>>>> For additional commands, e-mail:
> >>>>>>>>> [hidden email]
> >>>>>>>>>
> >>>>>>>>>
> >>>>>> -----------------------------------------------------------------
> - ----
> >>>>>>
> >>>>>>
> To unsubscribe, e-mail: [hidden email]
> >>>>>> For additional commands, e-mail:
> >>>>>> [hidden email]
> >>>>>>
> >>>>>>
> >>>>
> >>>> -------------------------------------------------------------------
> - --
> >>>>
> >>>>
> To unsubscribe, e-mail: [hidden email]
> >>>> For additional commands, e-mail:
> >>>> [hidden email]
> >>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7yeJYACgkQHPApP6U8
> pFhaHQ/9E56+1vZwl0kb3p/74AwmPkQ0wRJB08CYNpAxLHCDaQ4PJrMqGQDBfFcs
> RT/ithuxS8qJXyd02YgF6hWs8OtiAkODsPqjNxhuQMrLuF7qt2x0CgPWSfMKZVLW
> gLNA09kI6pQzfQrQROhfcqBQqGzaHVN8W4m+wTYOyCbMi2IjBF6lxcMtlCw7org/
> AFe904+ST8D2O+S8CK97dP+74oIu1gEJniLxs6kypx9+qzpgesmhhd6H/965XCBp
> ZEK6AwqedGUUPXxqxRdp6YmqflddtZdL4W1GhqbKN055+xGL+6rIlHD9W//ERJE8
> JutSaX6K5Z1t8+6K48Fwkju549g63J099fKQftg5hsSZ8Xwuqo9B3qOn2HcqrSZA
> +pgkdDDY5m0cHnB71GEREikABU36YnhBDeI0O4ocoQlRNnnIMAfb4hFxdwRCp5Qs
> dRM/DyFwRvdbiOQ0hfoUhx9HDH6JRsWrs5m2hy4n9m+5DJcQ/fxjUcqq0LTzIP3v
> upu81THiYVKBgbKpLKxadQnAzCj+TshEVF/E5D7ZWKG+enJAmnfrnIj5kKG/65HV
> RozUoE69+p1xWN6mWtZq8hMVjKfX1p4z+XlgTg03K/VcdNvs5gE4aKrevx1Ueueb
> hjogi6+JlQA7ysDlxHsttXrGVaY4DV9/c3ig8g7/pDQJCuoFe78=
> =BMkY
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ayub,

On 6/23/20 19:17, Ayub Khan wrote:

> Yes we have nginx as reverse proxy, below is the nginx config. We
> notice this issue only when there is high number of requests,
> during non peak hours we do not see this issue.> location
> /myapp/myservice{ #local machine proxy_pass
> <a href="http://localhost:8080;">http://localhost:8080; proxy_http_version  1.1;
>
> proxy_set_header    Connection          $connection_upgrade;
> proxy_set_header    Upgrade             $http_upgrade;
> proxy_set_header    Host                $host; proxy_set_header
> X-Real-IP           $remote_addr; proxy_set_header
> X-Forwarded-For     $proxy_add_x_forwarded_for;
>
>
> proxy_buffers 16 16k; proxy_buffer_size 32k; }

You might want to read about tuning nginx to drop connections after a
certain period of time, number of requests, etc. Looks like either a
bug in nginx or a misconfiguration which allows connections to
stick-around like this. You may have to ask the nginx people. I have
no experience with nginx myself, while others here may have some
experience.

> location / { #  if using AWS Load balancer, this bit checks for the
> presence of the https proto flag.  if regular http is found, then
> issue a redirect
to hit

> the https endpoint instead if ($http_x_forwarded_proto != 'https')
> { rewrite ^ https://$host$request_uri? permanent; }
>
> proxy_pass          <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080; proxy_http_version
> 1.1;
>
> proxy_set_header    Connection          $connection_upgrade;
> proxy_set_header    Upgrade             $http_upgrade;
> proxy_set_header    Host                $host; proxy_set_header
> X-Real-IP           $remote_addr; proxy_set_header
> X-Forwarded-For     $proxy_add_x_forwarded_for;
>
>
> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>
> *below is the connector*
>
> <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="2000" maxThreads="50000" URIEncoding="UTF-8"
> redirectPort="8443" />

50k threads is a LOT of threads. Do you expect to handle 50k requests
simultaneously?

> these ports are random, I am not sure who owns the process.
>
> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port 55866
> is a random port.
I'm sure you'll find that 55866 is owned by nginx. netstat will tell you
.

I think you need to look at your nginx configuration. It would also be
a great time to upgrade to a supported version of Tomcat. I would
recommend 8.5.56 or 9.0.36.

- -chris

> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
> [hidden email]> wrote:
>
> Ayub,
>
> On 6/23/20 16:23, Ayub Khan wrote:
>>>> I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)  *and I
>>>> saw the below output, some in CLOSE_WAIT and others in
>>>> ESTABLISHED. If there are 200 open file descriptors 160 are
>>>> in CLOSE_WAIT state. When the count for CLOSE_WAIT increases
>>>> I just have to restart tomcat.
>>>>
>>>> java    65189 tomcat8  715u     IPv6          237878311
>>>> 0t0 TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java
>>>> 65189 tomcat8  716u     IPv6          237848923       0t0
>>>> TCP localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>
> These are connections from some process into Tomcat listening on
> port 8080 (that's what localhost:http-alt is). So what process owns
> the outgoing connection on port 40568 on the same host?
>
> Are you using a reverse proxy?
>
>>>> most of the open files are in CLOSE_WAIT state I do not see
>>>> anything related to database ip.
>
> Agreed. It looks like you have a reverse proxy who is losing-track
> of connections, or who is (re)opening connections when it may be
> unnecessar y.
>
> Can you share your <Connector> configuration from server.xml?
> Remember to remove any secrets.
>
> -chris
>
>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
>>>> [hidden email]> wrote:
>>>>
>>>>>
>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
>>>>>> Felix,
>>>>>>
>>>>>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>>>>> and from the
>>>>> output
>>>>>> I see majority of them are related to sockets as shown
>>>>>> below, some of
>>>>> them
>>>>>> point to the jar file of tomcat and others to the log
>>>>>> file which is
>>>>> created.
>>>>>>
>>>>>> socket:[2084570754] socket:[2084579487]
>>>>>> socket:[2084578478] socket:[2084570167]
>>>>>
>>>>> Can you try the other command (lsof -p $(cat
>>>>> ...tomcat.pid))? It should give a bit more details on the
>>>>> used sockets that the proc directory.
>>>>>
>>>>> Felix
>>>>>
>>>>>>
>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>>>>>>>> Chris,
>>>>>>>>
>>>>>>>> I am using HikariCP for connection pooling. If the
>>>>>>>> database is leaking connections then I should see
>>>>>>>> connection not available exception.
>>>>>>>>
>>>>>>>> How do I find out which file descriptors are leaking
>>>>>>>> ? these are not
>>>>>>> files
>>>>>>>> open on disk as there is no explicit disk file I/O in
>>>>>>>> this application.
>>>>>>>>
>>>>>>>> I just use the below command to check for open file
>>>>>>>> descriptors:
>>>>>>>>
>>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ |
>>>>>>>> wc -l"
>>>>>>> You could have a look at the name of the files in the
>>>>>>> pids proc
>>>>> directory.
>>>>>>>
>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>>>>>>
>>>>>>> Or you could use the tool lsof to find the open file
>>>>>>> descriptors.
>>>>>>>
>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
>>>>>>>
>>>>>>> For both calls you should first change to the uid of
>>>>>>> the tomcat user or use sudo as in your example.
>>>>>>>
>>>>>>> Felix
>>>>>>>
>>>>>>>> Thanks and Regards Ayub
>>>>>>>>
>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz
>>>>>>>> < [hidden email]> wrote:
>>>>>>>>
>>>>>>>> Ayub,
>>>>>>>>
>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
>>>>>>>>>>> Sorry we are using  8.0.32 version of tomcat.
>>>>>>>>>>>
>>>>>>>>>>> below is the configuration:
>>>>>>>>>>>
>>>>>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu)
>>>>>>>>>>> Server built:   Jan 24 2020 16:24:30 UTC Server
>>>>>>>>>>> number: 8.0.32.0 OS Name: Linux OS Version:
>>>>>>>>>>> 4.4.0-1087-aws Architecture:   amd64 JVM
>>>>>>>>>>> Version: 1.8.0_181-b13 JVM Vendor:     Oracle
>>>>>>>>>>> Corporation
>>>>>>>>>>>
>>>>>>>>>>> I use the below command to check the file
>>>>>>>>>>> descriptors:
>>>>>>>>>>>
>>>>>>>>>>> watch "sudo ls /proc/`cat
>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>>>>>>>> So you know there is some kind of increase in
>>>>>>>> file-handle use, but you don't know what types of
>>>>>>>> file handles are increasing, right?
>>>>>>>>
>>>>>>>> Can you try to find out which kinds of file handles
>>>>>>>> are increasing?
>>>>>>>>
>>>>>>>> I have a sneaking suspicion that it's your database
>>>>>>>> connections and not actually files open on the disk.
>>>>>>>>
>>>>>>>> Are you using a database connection pool? If not,
>>>>>>>> you should really use one and limit the number of
>>>>>>>> connections to something sane. If you are using one,
>>>>>>>> are you monitoring it to see how many connections are
>>>>>>>> actually being used? Are you sure you are using
>>>>>>>> proper resource management[1]? Even a single
>>>>>>>> code-path that leaks connections can leak them
>>>>>>>> quickly under load.
>>>>>>>>
>>>>>>>>>>> When there an issue related to broken files,
>>>>>>>>>>> this value keeps increasing, the only way to
>>>>>>>>>>> bring it down is to remove vm instance from AWS
>>>>>>>>>>> load balancer.> Which version of tomcat should
>>>>>>>>>>> I install ?
>>>>>>>> Tomcat 8.0.x hasn't been supported since its last
>>>>>>>> release on 29 June 2018. That was 8.0.53. Your
>>>>>>>> release is from 8 February 2016 and is dangerously
>>>>>>>> out of date (unless you are using the Ubuntu-packaged
>>>>>>>> version, in which case I hope they kept-up with
>>>>>>>> security patches thee past 4 years).
>>>>>>>>
>>>>>>>> -chris
>>>>>>>>
>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher
>>>>>>>>>>> Schultz < [hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ayub,
>>>>>>>>>>>
>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases open
>>>>>>>>>>>>>> files on ubuntu AWS
>>>>>>>>>>> Which exact version of Tomcat 8.5? If you
>>>>>>>>>>> aren't running the latest version (8.5.56),
>>>>>>>>>>> please upgrade and re-test.
>>>>>>>>>>>
>>>>>>>>>>>>>> If there is slow response from db I see
>>>>>>>>>>>>>> this stack trace and the open files goes
>>>>>>>>>>>>>> high and the only way to open files go
>>>>>>>>>>>>>> down is to remove the instance from
>>>>>>>>>>>>>> Amazon load balancer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is there a way to keep the open files low
>>>>>>>>>>>>>> even when Broken pipe error is thrown ?
>>>>>>>>>>> What is your evidence that file handles are
>>>>>>>>>>> being left open?
>>>>>>>>>>>
>>>>>>>>>>> Which file handles are being left open?
>>>>>>>>>>>
>>>>>>>>>>> -chris
>>>>>>>>>>>>
>>>>>>> ----------------------------------------------------------------
- ---
>>
>>
>>>>>>>
- ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email] For
>> additional commands, e-mail: [hidden email]
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zaP0ACgkQHPApP6U8
pFiIWw//QaDZsLlbNyYgB2XVjQzMtI0L93TnroEW+MwKnle9UNT0xYBOBk+OnFgB
nO9j1RLvbE8sROcLKv2KucZYXFxvVfx6eAp3t2OIQkaSRvjnI2Gb/ILPgI7zTWq9
/1+qjG14A1Gcxy07elRBFik2BO/XqQ6eHiRw0426Ogh7f/djFKVX6qP4TIsFWNee
shZXLt9ZA/qtmhiNyis+MdpSMenUjmZ5z9TI9iyMQirbannAxl+oVds/0rYSKZ7G
kabydTSIQpR9OBTJdTrHX79UagkulmsKSKm9f1tXaSW8WwnnEjI3tQIee5orhzHU
ZdAkMWKU9YvZC8gBLuxs6QHmHEuoxVMRGsTatnfKIdRYIPkDEqMrzpiM7AozFutS
8tFMMjQsJYBKM3CqVFr3f7WJVBYjmKMBfH6ItTwZ88RBzOMdz5Odx3JDJMQbsEH5
ylKQH9Gluqad93he0t+Uaabk6JnCdXbCpdNWyFa/fPnMp4vHFlOb7f62+UlmDT+4
DyKTl3BtDe9J6pQ0eVY4deUqFgrLCU7C4WVkOqmj9DFmFJxCMp98zXCN2h4O9lhi
OE853x4UvrPkJMPddN2pg0cJiS6Z3OQzDES1yBVbduga2LnLOJrzDpRh6NtASmNM
YC82oSScqppOgRGHKWJUrnDTyXjEeJW/eTqhnMIHHXGlwuUfVZ0=
=zQdW
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
Chris,

If some open file is owned by nginx why would it show up if I run the below
command

sudo lsof -p $(cat /var/run/tomcat8.pid)



On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Ayub,
>
> On 6/23/20 19:17, Ayub Khan wrote:
> > Yes we have nginx as reverse proxy, below is the nginx config. We
> > notice this issue only when there is high number of requests,
> > during non peak hours we do not see this issue.> location
> > /myapp/myservice{ #local machine proxy_pass
> > <a href="http://localhost:8080;">http://localhost:8080; proxy_http_version  1.1;
> >
> > proxy_set_header    Connection          $connection_upgrade;
> > proxy_set_header    Upgrade             $http_upgrade;
> > proxy_set_header    Host                $host; proxy_set_header
> > X-Real-IP           $remote_addr; proxy_set_header
> > X-Forwarded-For     $proxy_add_x_forwarded_for;
> >
> >
> > proxy_buffers 16 16k; proxy_buffer_size 32k; }
>
> You might want to read about tuning nginx to drop connections after a
> certain period of time, number of requests, etc. Looks like either a
> bug in nginx or a misconfiguration which allows connections to
> stick-around like this. You may have to ask the nginx people. I have
> no experience with nginx myself, while others here may have some
> experience.
>
> > location / { #  if using AWS Load balancer, this bit checks for the
> > presence of the https proto flag.  if regular http is found, then
> > issue a redirect
> to hit
> > the https endpoint instead if ($http_x_forwarded_proto != 'https')
> > { rewrite ^ https://$host$request_uri? permanent; }
> >
> > proxy_pass          <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080; proxy_http_version
> > 1.1;
> >
> > proxy_set_header    Connection          $connection_upgrade;
> > proxy_set_header    Upgrade             $http_upgrade;
> > proxy_set_header    Host                $host; proxy_set_header
> > X-Real-IP           $remote_addr; proxy_set_header
> > X-Forwarded-For     $proxy_add_x_forwarded_for;
> >
> >
> > proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >
> > *below is the connector*
> >
> > <Connector port="8080"
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > connectionTimeout="2000" maxThreads="50000" URIEncoding="UTF-8"
> > redirectPort="8443" />
>
> 50k threads is a LOT of threads. Do you expect to handle 50k requests
> simultaneously?
>
> > these ports are random, I am not sure who owns the process.
> >
> > localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port 55866
> > is a random port.
> I'm sure you'll find that 55866 is owned by nginx. netstat will tell you
> .
>
> I think you need to look at your nginx configuration. It would also be
> a great time to upgrade to a supported version of Tomcat. I would
> recommend 8.5.56 or 9.0.36.
>
> - -chris
>
> > On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
> > [hidden email]> wrote:
> >
> > Ayub,
> >
> > On 6/23/20 16:23, Ayub Khan wrote:
> >>>> I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)  *and I
> >>>> saw the below output, some in CLOSE_WAIT and others in
> >>>> ESTABLISHED. If there are 200 open file descriptors 160 are
> >>>> in CLOSE_WAIT state. When the count for CLOSE_WAIT increases
> >>>> I just have to restart tomcat.
> >>>>
> >>>> java    65189 tomcat8  715u     IPv6          237878311
> >>>> 0t0 TCP localhost:http-alt->localhost:43760 (CLOSE_WAIT) java
> >>>> 65189 tomcat8  716u     IPv6          237848923       0t0
> >>>> TCP localhost:http-alt->localhost:40568 (CLOSE_WAIT)
> >
> > These are connections from some process into Tomcat listening on
> > port 8080 (that's what localhost:http-alt is). So what process owns
> > the outgoing connection on port 40568 on the same host?
> >
> > Are you using a reverse proxy?
> >
> >>>> most of the open files are in CLOSE_WAIT state I do not see
> >>>> anything related to database ip.
> >
> > Agreed. It looks like you have a reverse proxy who is losing-track
> > of connections, or who is (re)opening connections when it may be
> > unnecessar y.
> >
> > Can you share your <Connector> configuration from server.xml?
> > Remember to remove any secrets.
> >
> > -chris
> >
> >>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
> >>>> [hidden email]> wrote:
> >>>>
> >>>>>
> >>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> >>>>>> Felix,
> >>>>>>
> >>>>>> I executed ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>>> and from the
> >>>>> output
> >>>>>> I see majority of them are related to sockets as shown
> >>>>>> below, some of
> >>>>> them
> >>>>>> point to the jar file of tomcat and others to the log
> >>>>>> file which is
> >>>>> created.
> >>>>>>
> >>>>>> socket:[2084570754] socket:[2084579487]
> >>>>>> socket:[2084578478] socket:[2084570167]
> >>>>>
> >>>>> Can you try the other command (lsof -p $(cat
> >>>>> ...tomcat.pid))? It should give a bit more details on the
> >>>>> used sockets that the proc directory.
> >>>>>
> >>>>> Felix
> >>>>>
> >>>>>>
> >>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> >>>>>> [hidden email]> wrote:
> >>>>>>
> >>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>>>>>>> Chris,
> >>>>>>>>
> >>>>>>>> I am using HikariCP for connection pooling. If the
> >>>>>>>> database is leaking connections then I should see
> >>>>>>>> connection not available exception.
> >>>>>>>>
> >>>>>>>> How do I find out which file descriptors are leaking
> >>>>>>>> ? these are not
> >>>>>>> files
> >>>>>>>> open on disk as there is no explicit disk file I/O in
> >>>>>>>> this application.
> >>>>>>>>
> >>>>>>>> I just use the below command to check for open file
> >>>>>>>> descriptors:
> >>>>>>>>
> >>>>>>>> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ |
> >>>>>>>> wc -l"
> >>>>>>> You could have a look at the name of the files in the
> >>>>>>> pids proc
> >>>>> directory.
> >>>>>>>
> >>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>>>>
> >>>>>>> Or you could use the tool lsof to find the open file
> >>>>>>> descriptors.
> >>>>>>>
> >>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
> >>>>>>>
> >>>>>>> For both calls you should first change to the uid of
> >>>>>>> the tomcat user or use sudo as in your example.
> >>>>>>>
> >>>>>>> Felix
> >>>>>>>
> >>>>>>>> Thanks and Regards Ayub
> >>>>>>>>
> >>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher Schultz
> >>>>>>>> < [hidden email]> wrote:
> >>>>>>>>
> >>>>>>>> Ayub,
> >>>>>>>>
> >>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>>>>>>> Sorry we are using  8.0.32 version of tomcat.
> >>>>>>>>>>>
> >>>>>>>>>>> below is the configuration:
> >>>>>>>>>>>
> >>>>>>>>>>> Server version: Apache Tomcat/8.0.32 (Ubuntu)
> >>>>>>>>>>> Server built:   Jan 24 2020 16:24:30 UTC Server
> >>>>>>>>>>> number: 8.0.32.0 OS Name: Linux OS Version:
> >>>>>>>>>>> 4.4.0-1087-aws Architecture:   amd64 JVM
> >>>>>>>>>>> Version: 1.8.0_181-b13 JVM Vendor:     Oracle
> >>>>>>>>>>> Corporation
> >>>>>>>>>>>
> >>>>>>>>>>> I use the below command to check the file
> >>>>>>>>>>> descriptors:
> >>>>>>>>>>>
> >>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
> >>>>>>>> So you know there is some kind of increase in
> >>>>>>>> file-handle use, but you don't know what types of
> >>>>>>>> file handles are increasing, right?
> >>>>>>>>
> >>>>>>>> Can you try to find out which kinds of file handles
> >>>>>>>> are increasing?
> >>>>>>>>
> >>>>>>>> I have a sneaking suspicion that it's your database
> >>>>>>>> connections and not actually files open on the disk.
> >>>>>>>>
> >>>>>>>> Are you using a database connection pool? If not,
> >>>>>>>> you should really use one and limit the number of
> >>>>>>>> connections to something sane. If you are using one,
> >>>>>>>> are you monitoring it to see how many connections are
> >>>>>>>> actually being used? Are you sure you are using
> >>>>>>>> proper resource management[1]? Even a single
> >>>>>>>> code-path that leaks connections can leak them
> >>>>>>>> quickly under load.
> >>>>>>>>
> >>>>>>>>>>> When there an issue related to broken files,
> >>>>>>>>>>> this value keeps increasing, the only way to
> >>>>>>>>>>> bring it down is to remove vm instance from AWS
> >>>>>>>>>>> load balancer.> Which version of tomcat should
> >>>>>>>>>>> I install ?
> >>>>>>>> Tomcat 8.0.x hasn't been supported since its last
> >>>>>>>> release on 29 June 2018. That was 8.0.53. Your
> >>>>>>>> release is from 8 February 2016 and is dangerously
> >>>>>>>> out of date (unless you are using the Ubuntu-packaged
> >>>>>>>> version, in which case I hope they kept-up with
> >>>>>>>> security patches thee past 4 years).
> >>>>>>>>
> >>>>>>>> -chris
> >>>>>>>>
> >>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM Christopher
> >>>>>>>>>>> Schultz < [hidden email]> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Ayub,
> >>>>>>>>>>>
> >>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>>>>>>>>>> tomcat 8.5 broken pipe increases open
> >>>>>>>>>>>>>> files on ubuntu AWS
> >>>>>>>>>>> Which exact version of Tomcat 8.5? If you
> >>>>>>>>>>> aren't running the latest version (8.5.56),
> >>>>>>>>>>> please upgrade and re-test.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> If there is slow response from db I see
> >>>>>>>>>>>>>> this stack trace and the open files goes
> >>>>>>>>>>>>>> high and the only way to open files go
> >>>>>>>>>>>>>> down is to remove the instance from
> >>>>>>>>>>>>>> Amazon load balancer.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is there a way to keep the open files low
> >>>>>>>>>>>>>> even when Broken pipe error is thrown ?
> >>>>>>>>>>> What is your evidence that file handles are
> >>>>>>>>>>> being left open?
> >>>>>>>>>>>
> >>>>>>>>>>> Which file handles are being left open?
> >>>>>>>>>>>
> >>>>>>>>>>> -chris
> >>>>>>>>>>>>
> >>>>>>> ----------------------------------------------------------------
> - ---
> >>
> >>
> >>>>>>>
> - ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email] For
> >> additional commands, e-mail: [hidden email]
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zaP0ACgkQHPApP6U8
> pFiIWw//QaDZsLlbNyYgB2XVjQzMtI0L93TnroEW+MwKnle9UNT0xYBOBk+OnFgB
> nO9j1RLvbE8sROcLKv2KucZYXFxvVfx6eAp3t2OIQkaSRvjnI2Gb/ILPgI7zTWq9
> /1+qjG14A1Gcxy07elRBFik2BO/XqQ6eHiRw0426Ogh7f/djFKVX6qP4TIsFWNee
> shZXLt9ZA/qtmhiNyis+MdpSMenUjmZ5z9TI9iyMQirbannAxl+oVds/0rYSKZ7G
> kabydTSIQpR9OBTJdTrHX79UagkulmsKSKm9f1tXaSW8WwnnEjI3tQIee5orhzHU
> ZdAkMWKU9YvZC8gBLuxs6QHmHEuoxVMRGsTatnfKIdRYIPkDEqMrzpiM7AozFutS
> 8tFMMjQsJYBKM3CqVFr3f7WJVBYjmKMBfH6ItTwZ88RBzOMdz5Odx3JDJMQbsEH5
> ylKQH9Gluqad93he0t+Uaabk6JnCdXbCpdNWyFa/fPnMp4vHFlOb7f62+UlmDT+4
> DyKTl3BtDe9J6pQ0eVY4deUqFgrLCU7C4WVkOqmj9DFmFJxCMp98zXCN2h4O9lhi
> OE853x4UvrPkJMPddN2pg0cJiS6Z3OQzDES1yBVbduga2LnLOJrzDpRh6NtASmNM
> YC82oSScqppOgRGHKWJUrnDTyXjEeJW/eTqhnMIHHXGlwuUfVZ0=
> =zQdW
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

--
--------------------------------------------------------------------
Sun Certified Enterprise Architect 1.5
Sun Certified Java Programmer 1.4
Microsoft Certified Systems Engineer 2000
http://in.linkedin.com/pub/ayub-khan/a/811/b81
mobile:+966-502674604
----------------------------------------------------------------------
It is proved that Hard Work and kowledge will get you close but attitude
will get you there. However, it's the Love
of God that will put you over the top!!
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256


Ayub,

On 6/24/20 11:05, Ayub Khan wrote:
> If some open file is owned by nginx why would it show up if I run
> the below command> sudo lsof -p $(cat /var/run/tomcat8.pid)

Because network connections have two ends.

- -chris

> On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
> [hidden email]> wrote:
>
> Ayub,
>
> On 6/23/20 19:17, Ayub Khan wrote:
>>>> Yes we have nginx as reverse proxy, below is the nginx
>>>> config. We notice this issue only when there is high number
>>>> of requests, during non peak hours we do not see this issue.>
>>>> location /myapp/myservice{ #local machine proxy_pass
>>>> <a href="http://localhost:8080;">http://localhost:8080; proxy_http_version  1.1;
>>>>
>>>> proxy_set_header    Connection          $connection_upgrade;
>>>> proxy_set_header    Upgrade             $http_upgrade;
>>>> proxy_set_header    Host                $host;
>>>> proxy_set_header X-Real-IP           $remote_addr;
>>>> proxy_set_header X-Forwarded-For
>>>> $proxy_add_x_forwarded_for;
>>>>
>>>>
>>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>
> You might want to read about tuning nginx to drop connections after
> a certain period of time, number of requests, etc. Looks like
> either a bug in nginx or a misconfiguration which allows
> connections to stick-around like this. You may have to ask the
> nginx people. I have no experience with nginx myself, while others
> here may have some experience.
>
>>>> location / { #  if using AWS Load balancer, this bit checks
>>>> for the presence of the https proto flag.  if regular http is
>>>> found, then issue a redirect
> to hit
>>>> the https endpoint instead if ($http_x_forwarded_proto !=
>>>> 'https') { rewrite ^ https://$host$request_uri? permanent; }
>>>>
>>>> proxy_pass          <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
>>>> proxy_http_version 1.1;
>>>>
>>>> proxy_set_header    Connection          $connection_upgrade;
>>>> proxy_set_header    Upgrade             $http_upgrade;
>>>> proxy_set_header    Host                $host;
>>>> proxy_set_header X-Real-IP           $remote_addr;
>>>> proxy_set_header X-Forwarded-For
>>>> $proxy_add_x_forwarded_for;
>>>>
>>>>
>>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>>>>
>>>> *below is the connector*
>>>>
>>>> <Connector port="8080"
>>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>> connectionTimeout="2000" maxThreads="50000"
>>>> URIEncoding="UTF-8" redirectPort="8443" />
>
> 50k threads is a LOT of threads. Do you expect to handle 50k
> requests simultaneously?
>
>>>> these ports are random, I am not sure who owns the process.
>>>>
>>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port
>>>> 55866 is a random port.
> I'm sure you'll find that 55866 is owned by nginx. netstat will
> tell you .
>
> I think you need to look at your nginx configuration. It would also
> be a great time to upgrade to a supported version of Tomcat. I
> would recommend 8.5.56 or 9.0.36.
>
> -chris
>
>>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
>>>> [hidden email]> wrote:
>>>>
>>>> Ayub,
>>>>
>>>> On 6/23/20 16:23, Ayub Khan wrote:
>>>>>>> I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)
>>>>>>> *and I saw the below output, some in CLOSE_WAIT and
>>>>>>> others in ESTABLISHED. If there are 200 open file
>>>>>>> descriptors 160 are in CLOSE_WAIT state. When the count
>>>>>>> for CLOSE_WAIT increases I just have to restart
>>>>>>> tomcat.
>>>>>>>
>>>>>>> java    65189 tomcat8  715u     IPv6
>>>>>>> 237878311 0t0 TCP localhost:http-alt->localhost:43760
>>>>>>> (CLOSE_WAIT) java 65189 tomcat8  716u     IPv6
>>>>>>> 237848923       0t0 TCP
>>>>>>> localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>>>>
>>>> These are connections from some process into Tomcat listening
>>>> on port 8080 (that's what localhost:http-alt is). So what
>>>> process owns the outgoing connection on port 40568 on the
>>>> same host?
>>>>
>>>> Are you using a reverse proxy?
>>>>
>>>>>>> most of the open files are in CLOSE_WAIT state I do not
>>>>>>> see anything related to database ip.
>>>>
>>>> Agreed. It looks like you have a reverse proxy who is
>>>> losing-track of connections, or who is (re)opening
>>>> connections when it may be unnecessar y.
>>>>
>>>> Can you share your <Connector> configuration from
>>>> server.xml? Remember to remove any secrets.
>>>>
>>>> -chris
>>>>
>>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
>>>>>>> [hidden email]> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
>>>>>>>>> Felix,
>>>>>>>>>
>>>>>>>>> I executed ls -l /proc/$(cat
>>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the
>>>>>>>> output
>>>>>>>>> I see majority of them are related to sockets as
>>>>>>>>> shown below, some of
>>>>>>>> them
>>>>>>>>> point to the jar file of tomcat and others to the
>>>>>>>>> log file which is
>>>>>>>> created.
>>>>>>>>>
>>>>>>>>> socket:[2084570754] socket:[2084579487]
>>>>>>>>> socket:[2084578478] socket:[2084570167]
>>>>>>>>
>>>>>>>> Can you try the other command (lsof -p $(cat
>>>>>>>> ...tomcat.pid))? It should give a bit more details on
>>>>>>>> the used sockets that the proc directory.
>>>>>>>>
>>>>>>>> Felix
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
>>>>>>>>> [hidden email]> wrote:
>>>>>>>>>
>>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>>>>>>>>>>> Chris,
>>>>>>>>>>>
>>>>>>>>>>> I am using HikariCP for connection pooling. If
>>>>>>>>>>> the database is leaking connections then I
>>>>>>>>>>> should see connection not available exception.
>>>>>>>>>>>
>>>>>>>>>>> How do I find out which file descriptors are
>>>>>>>>>>> leaking ? these are not
>>>>>>>>>> files
>>>>>>>>>>> open on disk as there is no explicit disk file
>>>>>>>>>>> I/O in this application.
>>>>>>>>>>>
>>>>>>>>>>> I just use the below command to check for open
>>>>>>>>>>> file descriptors:
>>>>>>>>>>>
>>>>>>>>>>> watch "sudo ls /proc/`cat
>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>>>>>>>>>> You could have a look at the name of the files in
>>>>>>>>>> the pids proc
>>>>>>>> directory.
>>>>>>>>>>
>>>>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>>>>>>>>>>
>>>>>>>>>> Or you could use the tool lsof to find the open
>>>>>>>>>> file descriptors.
>>>>>>>>>>
>>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
>>>>>>>>>>
>>>>>>>>>> For both calls you should first change to the uid
>>>>>>>>>> of the tomcat user or use sudo as in your
>>>>>>>>>> example.
>>>>>>>>>>
>>>>>>>>>> Felix
>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards Ayub
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher
>>>>>>>>>>> Schultz < [hidden email]> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ayub,
>>>>>>>>>>>
>>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
>>>>>>>>>>>>>> Sorry we are using  8.0.32 version of
>>>>>>>>>>>>>> tomcat.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> below is the configuration:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Server version: Apache Tomcat/8.0.32
>>>>>>>>>>>>>> (Ubuntu) Server built:   Jan 24 2020
>>>>>>>>>>>>>> 16:24:30 UTC Server number: 8.0.32.0 OS
>>>>>>>>>>>>>> Name: Linux OS Version: 4.4.0-1087-aws
>>>>>>>>>>>>>> Architecture:   amd64 JVM Version:
>>>>>>>>>>>>>> 1.8.0_181-b13 JVM Vendor:     Oracle
>>>>>>>>>>>>>> Corporation
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I use the below command to check the
>>>>>>>>>>>>>> file descriptors:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> watch "sudo ls /proc/`cat
>>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>>>>>>>>>>> So you know there is some kind of increase in
>>>>>>>>>>> file-handle use, but you don't know what types
>>>>>>>>>>> of file handles are increasing, right?
>>>>>>>>>>>
>>>>>>>>>>> Can you try to find out which kinds of file
>>>>>>>>>>> handles are increasing?
>>>>>>>>>>>
>>>>>>>>>>> I have a sneaking suspicion that it's your
>>>>>>>>>>> database connections and not actually files
>>>>>>>>>>> open on the disk.
>>>>>>>>>>>
>>>>>>>>>>> Are you using a database connection pool? If
>>>>>>>>>>> not, you should really use one and limit the
>>>>>>>>>>> number of connections to something sane. If you
>>>>>>>>>>> are using one, are you monitoring it to see how
>>>>>>>>>>> many connections are actually being used? Are
>>>>>>>>>>> you sure you are using proper resource
>>>>>>>>>>> management[1]? Even a single code-path that
>>>>>>>>>>> leaks connections can leak them quickly under
>>>>>>>>>>> load.
>>>>>>>>>>>
>>>>>>>>>>>>>> When there an issue related to broken
>>>>>>>>>>>>>> files, this value keeps increasing, the
>>>>>>>>>>>>>> only way to bring it down is to remove vm
>>>>>>>>>>>>>> instance from AWS load balancer.> Which
>>>>>>>>>>>>>> version of tomcat should I install ?
>>>>>>>>>>> Tomcat 8.0.x hasn't been supported since its
>>>>>>>>>>> last release on 29 June 2018. That was 8.0.53.
>>>>>>>>>>> Your release is from 8 February 2016 and is
>>>>>>>>>>> dangerously out of date (unless you are using
>>>>>>>>>>> the Ubuntu-packaged version, in which case I
>>>>>>>>>>> hope they kept-up with security patches thee
>>>>>>>>>>> past 4 years).
>>>>>>>>>>>
>>>>>>>>>>> -chris
>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM
>>>>>>>>>>>>>> Christopher Schultz <
>>>>>>>>>>>>>> [hidden email]> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ayub,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases
>>>>>>>>>>>>>>>>> open files on ubuntu AWS
>>>>>>>>>>>>>> Which exact version of Tomcat 8.5? If
>>>>>>>>>>>>>> you aren't running the latest version
>>>>>>>>>>>>>> (8.5.56), please upgrade and re-test.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If there is slow response from db I
>>>>>>>>>>>>>>>>> see this stack trace and the open
>>>>>>>>>>>>>>>>> files goes high and the only way to
>>>>>>>>>>>>>>>>> open files go down is to remove the
>>>>>>>>>>>>>>>>> instance from Amazon load
>>>>>>>>>>>>>>>>> balancer.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is there a way to keep the open
>>>>>>>>>>>>>>>>> files low even when Broken pipe
>>>>>>>>>>>>>>>>> error is thrown ?
>>>>>>>>>>>>>> What is your evidence that file handles
>>>>>>>>>>>>>> are being left open?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Which file handles are being left open?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -chris
>>>>>>>>>>>>>>>
>>>>>>>>>> -------------------------------------------------------------
- ---
>
>>>>>>>>>>
- ---
>>>>>
>>>>>
>>>>>>>>>>
> ---------------------------------------------------------------------
>>>>>
>
To unsubscribe, e-mail: [hidden email] For
>>>>> additional commands, e-mail: [hidden email]
>>>>>
>>>>>
>>>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zgOIACgkQHPApP6U8
pFhnWQ//SBTsCR/g8cDgrC+ju6x3BA8kpEmNbA4BP5fY8I7nrDoiegsrZDbRsoWI
rNlH3i3pLlonTbbLQN/6AeWItmgM88bupy007cEcNrNivztkOPscmFOmPZP7lNaD
JfgXQvTF1q8LFL+R120GGVR+OKM06dZ4rYbWbu+Ju46lYX6q2iVlb+n8XmmPBIPg
9dEE4kXNc+A3ZI+3JxLoWnxnDSxpG2Od2WDwt+dLV0E7XkHJCMKGqtzbPWcUtaUR
nh9rywiO4ODk57VbtU426yB6OrNxbwj2OOKpETKVfLpUsdwzvgEZgzbfeUqPpR4p
OpIXxjgBeGFhk+ssafvqOMIMI4EO/m/+uttyfQhAh05ozP99NLTZXbDcnY9gFEeY
Rk7Z7qnGey3erk83dfR3X56zovuiS6k6vHKLZcp6Js5ta1cQa9D8hOjxcVRaDWlz
AeKNEiwuuIo2tfsl2OCAxkoNpl+dooE2aXcvAWbrI7pGXMWwANzEGw4NmIUTOvpS
tjR4OqaOKiRo1TN66JZVeol0ZFlaz9InxHTEasHrdn80YJGMIFL7UmR6koZGUE2G
7k0yukOUQCDLCPc3HwCCfOw/rkygLtxKShMEcZDp2gH3AZo0z1AKBDGWyi7XyMKa
MO4DkMHhp28aRZ84Dtsyvxh/HuxBt9Vl2hj/oOkBMVbfrIT+3bg=
=q6xp
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
Chris,

Ok, I will investigate nginx side as well. Thank you for the pointers

On Wed, 24 Jun 2020, 19:45 Christopher Schultz, <
[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
>
> Ayub,
>
> On 6/24/20 11:05, Ayub Khan wrote:
> > If some open file is owned by nginx why would it show up if I run
> > the below command> sudo lsof -p $(cat /var/run/tomcat8.pid)
>
> Because network connections have two ends.
>
> - -chris
>
> > On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
> > [hidden email]> wrote:
> >
> > Ayub,
> >
> > On 6/23/20 19:17, Ayub Khan wrote:
> >>>> Yes we have nginx as reverse proxy, below is the nginx
> >>>> config. We notice this issue only when there is high number
> >>>> of requests, during non peak hours we do not see this issue.>
> >>>> location /myapp/myservice{ #local machine proxy_pass
> >>>> <a href="http://localhost:8080;">http://localhost:8080; proxy_http_version  1.1;
> >>>>
> >>>> proxy_set_header    Connection          $connection_upgrade;
> >>>> proxy_set_header    Upgrade             $http_upgrade;
> >>>> proxy_set_header    Host                $host;
> >>>> proxy_set_header X-Real-IP           $remote_addr;
> >>>> proxy_set_header X-Forwarded-For
> >>>> $proxy_add_x_forwarded_for;
> >>>>
> >>>>
> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >
> > You might want to read about tuning nginx to drop connections after
> > a certain period of time, number of requests, etc. Looks like
> > either a bug in nginx or a misconfiguration which allows
> > connections to stick-around like this. You may have to ask the
> > nginx people. I have no experience with nginx myself, while others
> > here may have some experience.
> >
> >>>> location / { #  if using AWS Load balancer, this bit checks
> >>>> for the presence of the https proto flag.  if regular http is
> >>>> found, then issue a redirect
> > to hit
> >>>> the https endpoint instead if ($http_x_forwarded_proto !=
> >>>> 'https') { rewrite ^ https://$host$request_uri? permanent; }
> >>>>
> >>>> proxy_pass          <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
> >>>> proxy_http_version 1.1;
> >>>>
> >>>> proxy_set_header    Connection          $connection_upgrade;
> >>>> proxy_set_header    Upgrade             $http_upgrade;
> >>>> proxy_set_header    Host                $host;
> >>>> proxy_set_header X-Real-IP           $remote_addr;
> >>>> proxy_set_header X-Forwarded-For
> >>>> $proxy_add_x_forwarded_for;
> >>>>
> >>>>
> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
> >>>>
> >>>> *below is the connector*
> >>>>
> >>>> <Connector port="8080"
> >>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
> >>>> connectionTimeout="2000" maxThreads="50000"
> >>>> URIEncoding="UTF-8" redirectPort="8443" />
> >
> > 50k threads is a LOT of threads. Do you expect to handle 50k
> > requests simultaneously?
> >
> >>>> these ports are random, I am not sure who owns the process.
> >>>>
> >>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port
> >>>> 55866 is a random port.
> > I'm sure you'll find that 55866 is owned by nginx. netstat will
> > tell you .
> >
> > I think you need to look at your nginx configuration. It would also
> > be a great time to upgrade to a supported version of Tomcat. I
> > would recommend 8.5.56 or 9.0.36.
> >
> > -chris
> >
> >>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
> >>>> [hidden email]> wrote:
> >>>>
> >>>> Ayub,
> >>>>
> >>>> On 6/23/20 16:23, Ayub Khan wrote:
> >>>>>>> I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)
> >>>>>>> *and I saw the below output, some in CLOSE_WAIT and
> >>>>>>> others in ESTABLISHED. If there are 200 open file
> >>>>>>> descriptors 160 are in CLOSE_WAIT state. When the count
> >>>>>>> for CLOSE_WAIT increases I just have to restart
> >>>>>>> tomcat.
> >>>>>>>
> >>>>>>> java    65189 tomcat8  715u     IPv6
> >>>>>>> 237878311 0t0 TCP localhost:http-alt->localhost:43760
> >>>>>>> (CLOSE_WAIT) java 65189 tomcat8  716u     IPv6
> >>>>>>> 237848923       0t0 TCP
> >>>>>>> localhost:http-alt->localhost:40568 (CLOSE_WAIT)
> >>>>
> >>>> These are connections from some process into Tomcat listening
> >>>> on port 8080 (that's what localhost:http-alt is). So what
> >>>> process owns the outgoing connection on port 40568 on the
> >>>> same host?
> >>>>
> >>>> Are you using a reverse proxy?
> >>>>
> >>>>>>> most of the open files are in CLOSE_WAIT state I do not
> >>>>>>> see anything related to database ip.
> >>>>
> >>>> Agreed. It looks like you have a reverse proxy who is
> >>>> losing-track of connections, or who is (re)opening
> >>>> connections when it may be unnecessar y.
> >>>>
> >>>> Can you share your <Connector> configuration from
> >>>> server.xml? Remember to remove any secrets.
> >>>>
> >>>> -chris
> >>>>
> >>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
> >>>>>>> [hidden email]> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
> >>>>>>>>> Felix,
> >>>>>>>>>
> >>>>>>>>> I executed ls -l /proc/$(cat
> >>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the
> >>>>>>>> output
> >>>>>>>>> I see majority of them are related to sockets as
> >>>>>>>>> shown below, some of
> >>>>>>>> them
> >>>>>>>>> point to the jar file of tomcat and others to the
> >>>>>>>>> log file which is
> >>>>>>>> created.
> >>>>>>>>>
> >>>>>>>>> socket:[2084570754] socket:[2084579487]
> >>>>>>>>> socket:[2084578478] socket:[2084570167]
> >>>>>>>>
> >>>>>>>> Can you try the other command (lsof -p $(cat
> >>>>>>>> ...tomcat.pid))? It should give a bit more details on
> >>>>>>>> the used sockets that the proc directory.
> >>>>>>>>
> >>>>>>>> Felix
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
> >>>>>>>>> [hidden email]> wrote:
> >>>>>>>>>
> >>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
> >>>>>>>>>>> Chris,
> >>>>>>>>>>>
> >>>>>>>>>>> I am using HikariCP for connection pooling. If
> >>>>>>>>>>> the database is leaking connections then I
> >>>>>>>>>>> should see connection not available exception.
> >>>>>>>>>>>
> >>>>>>>>>>> How do I find out which file descriptors are
> >>>>>>>>>>> leaking ? these are not
> >>>>>>>>>> files
> >>>>>>>>>>> open on disk as there is no explicit disk file
> >>>>>>>>>>> I/O in this application.
> >>>>>>>>>>>
> >>>>>>>>>>> I just use the below command to check for open
> >>>>>>>>>>> file descriptors:
> >>>>>>>>>>>
> >>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
> >>>>>>>>>> You could have a look at the name of the files in
> >>>>>>>>>> the pids proc
> >>>>>>>> directory.
> >>>>>>>>>>
> >>>>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
> >>>>>>>>>>
> >>>>>>>>>> Or you could use the tool lsof to find the open
> >>>>>>>>>> file descriptors.
> >>>>>>>>>>
> >>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
> >>>>>>>>>>
> >>>>>>>>>> For both calls you should first change to the uid
> >>>>>>>>>> of the tomcat user or use sudo as in your
> >>>>>>>>>> example.
> >>>>>>>>>>
> >>>>>>>>>> Felix
> >>>>>>>>>>
> >>>>>>>>>>> Thanks and Regards Ayub
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher
> >>>>>>>>>>> Schultz < [hidden email]> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Ayub,
> >>>>>>>>>>>
> >>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
> >>>>>>>>>>>>>> Sorry we are using  8.0.32 version of
> >>>>>>>>>>>>>> tomcat.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> below is the configuration:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Server version: Apache Tomcat/8.0.32
> >>>>>>>>>>>>>> (Ubuntu) Server built:   Jan 24 2020
> >>>>>>>>>>>>>> 16:24:30 UTC Server number: 8.0.32.0 OS
> >>>>>>>>>>>>>> Name: Linux OS Version: 4.4.0-1087-aws
> >>>>>>>>>>>>>> Architecture:   amd64 JVM Version:
> >>>>>>>>>>>>>> 1.8.0_181-b13 JVM Vendor:     Oracle
> >>>>>>>>>>>>>> Corporation
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I use the below command to check the
> >>>>>>>>>>>>>> file descriptors:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> watch "sudo ls /proc/`cat
> >>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
> >>>>>>>>>>> So you know there is some kind of increase in
> >>>>>>>>>>> file-handle use, but you don't know what types
> >>>>>>>>>>> of file handles are increasing, right?
> >>>>>>>>>>>
> >>>>>>>>>>> Can you try to find out which kinds of file
> >>>>>>>>>>> handles are increasing?
> >>>>>>>>>>>
> >>>>>>>>>>> I have a sneaking suspicion that it's your
> >>>>>>>>>>> database connections and not actually files
> >>>>>>>>>>> open on the disk.
> >>>>>>>>>>>
> >>>>>>>>>>> Are you using a database connection pool? If
> >>>>>>>>>>> not, you should really use one and limit the
> >>>>>>>>>>> number of connections to something sane. If you
> >>>>>>>>>>> are using one, are you monitoring it to see how
> >>>>>>>>>>> many connections are actually being used? Are
> >>>>>>>>>>> you sure you are using proper resource
> >>>>>>>>>>> management[1]? Even a single code-path that
> >>>>>>>>>>> leaks connections can leak them quickly under
> >>>>>>>>>>> load.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> When there an issue related to broken
> >>>>>>>>>>>>>> files, this value keeps increasing, the
> >>>>>>>>>>>>>> only way to bring it down is to remove vm
> >>>>>>>>>>>>>> instance from AWS load balancer.> Which
> >>>>>>>>>>>>>> version of tomcat should I install ?
> >>>>>>>>>>> Tomcat 8.0.x hasn't been supported since its
> >>>>>>>>>>> last release on 29 June 2018. That was 8.0.53.
> >>>>>>>>>>> Your release is from 8 February 2016 and is
> >>>>>>>>>>> dangerously out of date (unless you are using
> >>>>>>>>>>> the Ubuntu-packaged version, in which case I
> >>>>>>>>>>> hope they kept-up with security patches thee
> >>>>>>>>>>> past 4 years).
> >>>>>>>>>>>
> >>>>>>>>>>> -chris
> >>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM
> >>>>>>>>>>>>>> Christopher Schultz <
> >>>>>>>>>>>>>> [hidden email]> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ayub,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
> >>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases
> >>>>>>>>>>>>>>>>> open files on ubuntu AWS
> >>>>>>>>>>>>>> Which exact version of Tomcat 8.5? If
> >>>>>>>>>>>>>> you aren't running the latest version
> >>>>>>>>>>>>>> (8.5.56), please upgrade and re-test.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If there is slow response from db I
> >>>>>>>>>>>>>>>>> see this stack trace and the open
> >>>>>>>>>>>>>>>>> files goes high and the only way to
> >>>>>>>>>>>>>>>>> open files go down is to remove the
> >>>>>>>>>>>>>>>>> instance from Amazon load
> >>>>>>>>>>>>>>>>> balancer.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is there a way to keep the open
> >>>>>>>>>>>>>>>>> files low even when Broken pipe
> >>>>>>>>>>>>>>>>> error is thrown ?
> >>>>>>>>>>>>>> What is your evidence that file handles
> >>>>>>>>>>>>>> are being left open?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Which file handles are being left open?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -chris
> >>>>>>>>>>>>>>>
> >>>>>>>>>> -------------------------------------------------------------
> - ---
> >
> >>>>>>>>>>
> - ---
> >>>>>
> >>>>>
> >>>>>>>>>>
> > ---------------------------------------------------------------------
> >>>>>
> >
> To unsubscribe, e-mail: [hidden email] For
> >>>>> additional commands, e-mail: [hidden email]
> >>>>>
> >>>>>
> >>>>
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zgOIACgkQHPApP6U8
> pFhnWQ//SBTsCR/g8cDgrC+ju6x3BA8kpEmNbA4BP5fY8I7nrDoiegsrZDbRsoWI
> rNlH3i3pLlonTbbLQN/6AeWItmgM88bupy007cEcNrNivztkOPscmFOmPZP7lNaD
> JfgXQvTF1q8LFL+R120GGVR+OKM06dZ4rYbWbu+Ju46lYX6q2iVlb+n8XmmPBIPg
> 9dEE4kXNc+A3ZI+3JxLoWnxnDSxpG2Od2WDwt+dLV0E7XkHJCMKGqtzbPWcUtaUR
> nh9rywiO4ODk57VbtU426yB6OrNxbwj2OOKpETKVfLpUsdwzvgEZgzbfeUqPpR4p
> OpIXxjgBeGFhk+ssafvqOMIMI4EO/m/+uttyfQhAh05ozP99NLTZXbDcnY9gFEeY
> Rk7Z7qnGey3erk83dfR3X56zovuiS6k6vHKLZcp6Js5ta1cQa9D8hOjxcVRaDWlz
> AeKNEiwuuIo2tfsl2OCAxkoNpl+dooE2aXcvAWbrI7pGXMWwANzEGw4NmIUTOvpS
> tjR4OqaOKiRo1TN66JZVeol0ZFlaz9InxHTEasHrdn80YJGMIFL7UmR6koZGUE2G
> 7k0yukOUQCDLCPc3HwCCfOw/rkygLtxKShMEcZDp2gH3AZo0z1AKBDGWyi7XyMKa
> MO4DkMHhp28aRZ84Dtsyvxh/HuxBt9Vl2hj/oOkBMVbfrIT+3bg=
> =q6xp
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: broken pipe error keeps increasing open files

Ayub Khan
Chris,

Was just thinking if the file descriptors belonged to nginx why do they
disappear as soon as I restart tomcat ? I tried restarting nginx and the
open file descriptors don't disappear.

When I execute lsof -p <nginx pid> I do not see file descriptors in close
wait state

On Wed, 24 Jun 2020, 20:32 Ayub Khan, <[hidden email]> wrote:

> Chris,
>
> Ok, I will investigate nginx side as well. Thank you for the pointers
>
> On Wed, 24 Jun 2020, 19:45 Christopher Schultz, <
> [hidden email]> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>>
>> Ayub,
>>
>> On 6/24/20 11:05, Ayub Khan wrote:
>> > If some open file is owned by nginx why would it show up if I run
>> > the below command> sudo lsof -p $(cat /var/run/tomcat8.pid)
>>
>> Because network connections have two ends.
>>
>> - -chris
>>
>> > On Wed, Jun 24, 2020 at 5:53 PM Christopher Schultz <
>> > [hidden email]> wrote:
>> >
>> > Ayub,
>> >
>> > On 6/23/20 19:17, Ayub Khan wrote:
>> >>>> Yes we have nginx as reverse proxy, below is the nginx
>> >>>> config. We notice this issue only when there is high number
>> >>>> of requests, during non peak hours we do not see this issue.>
>> >>>> location /myapp/myservice{ #local machine proxy_pass
>> >>>> <a href="http://localhost:8080;">http://localhost:8080; proxy_http_version  1.1;
>> >>>>
>> >>>> proxy_set_header    Connection          $connection_upgrade;
>> >>>> proxy_set_header    Upgrade             $http_upgrade;
>> >>>> proxy_set_header    Host                $host;
>> >>>> proxy_set_header X-Real-IP           $remote_addr;
>> >>>> proxy_set_header X-Forwarded-For
>> >>>> $proxy_add_x_forwarded_for;
>> >>>>
>> >>>>
>> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>> >
>> > You might want to read about tuning nginx to drop connections after
>> > a certain period of time, number of requests, etc. Looks like
>> > either a bug in nginx or a misconfiguration which allows
>> > connections to stick-around like this. You may have to ask the
>> > nginx people. I have no experience with nginx myself, while others
>> > here may have some experience.
>> >
>> >>>> location / { #  if using AWS Load balancer, this bit checks
>> >>>> for the presence of the https proto flag.  if regular http is
>> >>>> found, then issue a redirect
>> > to hit
>> >>>> the https endpoint instead if ($http_x_forwarded_proto !=
>> >>>> 'https') { rewrite ^ https://$host$request_uri? permanent; }
>> >>>>
>> >>>> proxy_pass          <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
>> >>>> proxy_http_version 1.1;
>> >>>>
>> >>>> proxy_set_header    Connection          $connection_upgrade;
>> >>>> proxy_set_header    Upgrade             $http_upgrade;
>> >>>> proxy_set_header    Host                $host;
>> >>>> proxy_set_header X-Real-IP           $remote_addr;
>> >>>> proxy_set_header X-Forwarded-For
>> >>>> $proxy_add_x_forwarded_for;
>> >>>>
>> >>>>
>> >>>> proxy_buffers 16 16k; proxy_buffer_size 32k; }
>> >>>>
>> >>>> *below is the connector*
>> >>>>
>> >>>> <Connector port="8080"
>> >>>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> >>>> connectionTimeout="2000" maxThreads="50000"
>> >>>> URIEncoding="UTF-8" redirectPort="8443" />
>> >
>> > 50k threads is a LOT of threads. Do you expect to handle 50k
>> > requests simultaneously?
>> >
>> >>>> these ports are random, I am not sure who owns the process.
>> >>>>
>> >>>> localhost:http-alt->localhost:55866 (CLOSE_WAIT) , here port
>> >>>> 55866 is a random port.
>> > I'm sure you'll find that 55866 is owned by nginx. netstat will
>> > tell you .
>> >
>> > I think you need to look at your nginx configuration. It would also
>> > be a great time to upgrade to a supported version of Tomcat. I
>> > would recommend 8.5.56 or 9.0.36.
>> >
>> > -chris
>> >
>> >>>> On Wed, Jun 24, 2020 at 12:48 AM Christopher Schultz <
>> >>>> [hidden email]> wrote:
>> >>>>
>> >>>> Ayub,
>> >>>>
>> >>>> On 6/23/20 16:23, Ayub Khan wrote:
>> >>>>>>> I executed  *sudo lsof -p $(cat /var/run/tomcat8.pid)
>> >>>>>>> *and I saw the below output, some in CLOSE_WAIT and
>> >>>>>>> others in ESTABLISHED. If there are 200 open file
>> >>>>>>> descriptors 160 are in CLOSE_WAIT state. When the count
>> >>>>>>> for CLOSE_WAIT increases I just have to restart
>> >>>>>>> tomcat.
>> >>>>>>>
>> >>>>>>> java    65189 tomcat8  715u     IPv6
>> >>>>>>> 237878311 0t0 TCP localhost:http-alt->localhost:43760
>> >>>>>>> (CLOSE_WAIT) java 65189 tomcat8  716u     IPv6
>> >>>>>>> 237848923       0t0 TCP
>> >>>>>>> localhost:http-alt->localhost:40568 (CLOSE_WAIT)
>> >>>>
>> >>>> These are connections from some process into Tomcat listening
>> >>>> on port 8080 (that's what localhost:http-alt is). So what
>> >>>> process owns the outgoing connection on port 40568 on the
>> >>>> same host?
>> >>>>
>> >>>> Are you using a reverse proxy?
>> >>>>
>> >>>>>>> most of the open files are in CLOSE_WAIT state I do not
>> >>>>>>> see anything related to database ip.
>> >>>>
>> >>>> Agreed. It looks like you have a reverse proxy who is
>> >>>> losing-track of connections, or who is (re)opening
>> >>>> connections when it may be unnecessar y.
>> >>>>
>> >>>> Can you share your <Connector> configuration from
>> >>>> server.xml? Remember to remove any secrets.
>> >>>>
>> >>>> -chris
>> >>>>
>> >>>>>>> On Mon, Jun 22, 2020 at 4:27 PM Felix Schumacher <
>> >>>>>>> [hidden email]> wrote:
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> Am 22.06.20 um 13:22 schrieb Ayub Khan:
>> >>>>>>>>> Felix,
>> >>>>>>>>>
>> >>>>>>>>> I executed ls -l /proc/$(cat
>> >>>>>>>>> /var/run/tomcat8.pid)/fd/ and from the
>> >>>>>>>> output
>> >>>>>>>>> I see majority of them are related to sockets as
>> >>>>>>>>> shown below, some of
>> >>>>>>>> them
>> >>>>>>>>> point to the jar file of tomcat and others to the
>> >>>>>>>>> log file which is
>> >>>>>>>> created.
>> >>>>>>>>>
>> >>>>>>>>> socket:[2084570754] socket:[2084579487]
>> >>>>>>>>> socket:[2084578478] socket:[2084570167]
>> >>>>>>>>
>> >>>>>>>> Can you try the other command (lsof -p $(cat
>> >>>>>>>> ...tomcat.pid))? It should give a bit more details on
>> >>>>>>>> the used sockets that the proc directory.
>> >>>>>>>>
>> >>>>>>>> Felix
>> >>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Mon, Jun 22, 2020 at 1:28 PM Felix Schumacher <
>> >>>>>>>>> [hidden email]> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Am 22.06.20 um 11:41 schrieb Ayub Khan:
>> >>>>>>>>>>> Chris,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I am using HikariCP for connection pooling. If
>> >>>>>>>>>>> the database is leaking connections then I
>> >>>>>>>>>>> should see connection not available exception.
>> >>>>>>>>>>>
>> >>>>>>>>>>> How do I find out which file descriptors are
>> >>>>>>>>>>> leaking ? these are not
>> >>>>>>>>>> files
>> >>>>>>>>>>> open on disk as there is no explicit disk file
>> >>>>>>>>>>> I/O in this application.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I just use the below command to check for open
>> >>>>>>>>>>> file descriptors:
>> >>>>>>>>>>>
>> >>>>>>>>>>> watch "sudo ls /proc/`cat
>> >>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>> >>>>>>>>>> You could have a look at the name of the files in
>> >>>>>>>>>> the pids proc
>> >>>>>>>> directory.
>> >>>>>>>>>>
>> >>>>>>>>>> $ ls -l /proc/$(cat /var/run/tomcat8.pid)/fd/
>> >>>>>>>>>>
>> >>>>>>>>>> Or you could use the tool lsof to find the open
>> >>>>>>>>>> file descriptors.
>> >>>>>>>>>>
>> >>>>>>>>>> $ lsof -p $(cat /var/run/tomcat8.pid)
>> >>>>>>>>>>
>> >>>>>>>>>> For both calls you should first change to the uid
>> >>>>>>>>>> of the tomcat user or use sudo as in your
>> >>>>>>>>>> example.
>> >>>>>>>>>>
>> >>>>>>>>>> Felix
>> >>>>>>>>>>
>> >>>>>>>>>>> Thanks and Regards Ayub
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Sun, Jun 21, 2020 at 8:18 PM Christopher
>> >>>>>>>>>>> Schultz < [hidden email]> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Ayub,
>> >>>>>>>>>>>
>> >>>>>>>>>>> On 6/20/20 11:51, Ayub Khan wrote:
>> >>>>>>>>>>>>>> Sorry we are using  8.0.32 version of
>> >>>>>>>>>>>>>> tomcat.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> below is the configuration:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Server version: Apache Tomcat/8.0.32
>> >>>>>>>>>>>>>> (Ubuntu) Server built:   Jan 24 2020
>> >>>>>>>>>>>>>> 16:24:30 UTC Server number: 8.0.32.0 OS
>> >>>>>>>>>>>>>> Name: Linux OS Version: 4.4.0-1087-aws
>> >>>>>>>>>>>>>> Architecture:   amd64 JVM Version:
>> >>>>>>>>>>>>>> 1.8.0_181-b13 JVM Vendor:     Oracle
>> >>>>>>>>>>>>>> Corporation
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I use the below command to check the
>> >>>>>>>>>>>>>> file descriptors:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> watch "sudo ls /proc/`cat
>> >>>>>>>>>>>>>> /var/run/tomcat8.pid`/fd/ | wc -l"
>> >>>>>>>>>>> So you know there is some kind of increase in
>> >>>>>>>>>>> file-handle use, but you don't know what types
>> >>>>>>>>>>> of file handles are increasing, right?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Can you try to find out which kinds of file
>> >>>>>>>>>>> handles are increasing?
>> >>>>>>>>>>>
>> >>>>>>>>>>> I have a sneaking suspicion that it's your
>> >>>>>>>>>>> database connections and not actually files
>> >>>>>>>>>>> open on the disk.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Are you using a database connection pool? If
>> >>>>>>>>>>> not, you should really use one and limit the
>> >>>>>>>>>>> number of connections to something sane. If you
>> >>>>>>>>>>> are using one, are you monitoring it to see how
>> >>>>>>>>>>> many connections are actually being used? Are
>> >>>>>>>>>>> you sure you are using proper resource
>> >>>>>>>>>>> management[1]? Even a single code-path that
>> >>>>>>>>>>> leaks connections can leak them quickly under
>> >>>>>>>>>>> load.
>> >>>>>>>>>>>
>> >>>>>>>>>>>>>> When there an issue related to broken
>> >>>>>>>>>>>>>> files, this value keeps increasing, the
>> >>>>>>>>>>>>>> only way to bring it down is to remove vm
>> >>>>>>>>>>>>>> instance from AWS load balancer.> Which
>> >>>>>>>>>>>>>> version of tomcat should I install ?
>> >>>>>>>>>>> Tomcat 8.0.x hasn't been supported since its
>> >>>>>>>>>>> last release on 29 June 2018. That was 8.0.53.
>> >>>>>>>>>>> Your release is from 8 February 2016 and is
>> >>>>>>>>>>> dangerously out of date (unless you are using
>> >>>>>>>>>>> the Ubuntu-packaged version, in which case I
>> >>>>>>>>>>> hope they kept-up with security patches thee
>> >>>>>>>>>>> past 4 years).
>> >>>>>>>>>>>
>> >>>>>>>>>>> -chris
>> >>>>>>>>>>>
>> >>>>>>>>>>>>>> On Sat, Jun 20, 2020 at 6:28 PM
>> >>>>>>>>>>>>>> Christopher Schultz <
>> >>>>>>>>>>>>>> [hidden email]> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Ayub,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On 6/19/20 16:46, Ayub Khan wrote:
>> >>>>>>>>>>>>>>>>> tomcat 8.5 broken pipe increases
>> >>>>>>>>>>>>>>>>> open files on ubuntu AWS
>> >>>>>>>>>>>>>> Which exact version of Tomcat 8.5? If
>> >>>>>>>>>>>>>> you aren't running the latest version
>> >>>>>>>>>>>>>> (8.5.56), please upgrade and re-test.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> If there is slow response from db I
>> >>>>>>>>>>>>>>>>> see this stack trace and the open
>> >>>>>>>>>>>>>>>>> files goes high and the only way to
>> >>>>>>>>>>>>>>>>> open files go down is to remove the
>> >>>>>>>>>>>>>>>>> instance from Amazon load
>> >>>>>>>>>>>>>>>>> balancer.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Is there a way to keep the open
>> >>>>>>>>>>>>>>>>> files low even when Broken pipe
>> >>>>>>>>>>>>>>>>> error is thrown ?
>> >>>>>>>>>>>>>> What is your evidence that file handles
>> >>>>>>>>>>>>>> are being left open?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Which file handles are being left open?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> -chris
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>> -------------------------------------------------------------
>> - ---
>> >
>> >>>>>>>>>>
>> - ---
>> >>>>>
>> >>>>>
>> >>>>>>>>>>
>> > ---------------------------------------------------------------------
>> >>>>>
>> >
>> To unsubscribe, e-mail: [hidden email] For
>> >>>>> additional commands, e-mail: [hidden email]
>> >>>>>
>> >>>>>
>> >>>>
>> >>
>> >> ---------------------------------------------------------------------
>> >>
>> >>
>> To unsubscribe, e-mail: [hidden email]
>> >> For additional commands, e-mail: [hidden email]
>> >>
>> >>
>> >
>> -----BEGIN PGP SIGNATURE-----
>> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>>
>> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl7zgOIACgkQHPApP6U8
>> pFhnWQ//SBTsCR/g8cDgrC+ju6x3BA8kpEmNbA4BP5fY8I7nrDoiegsrZDbRsoWI
>> rNlH3i3pLlonTbbLQN/6AeWItmgM88bupy007cEcNrNivztkOPscmFOmPZP7lNaD
>> JfgXQvTF1q8LFL+R120GGVR+OKM06dZ4rYbWbu+Ju46lYX6q2iVlb+n8XmmPBIPg
>> 9dEE4kXNc+A3ZI+3JxLoWnxnDSxpG2Od2WDwt+dLV0E7XkHJCMKGqtzbPWcUtaUR
>> nh9rywiO4ODk57VbtU426yB6OrNxbwj2OOKpETKVfLpUsdwzvgEZgzbfeUqPpR4p
>> OpIXxjgBeGFhk+ssafvqOMIMI4EO/m/+uttyfQhAh05ozP99NLTZXbDcnY9gFEeY
>> Rk7Z7qnGey3erk83dfR3X56zovuiS6k6vHKLZcp6Js5ta1cQa9D8hOjxcVRaDWlz
>> AeKNEiwuuIo2tfsl2OCAxkoNpl+dooE2aXcvAWbrI7pGXMWwANzEGw4NmIUTOvpS
>> tjR4OqaOKiRo1TN66JZVeol0ZFlaz9InxHTEasHrdn80YJGMIFL7UmR6koZGUE2G
>> 7k0yukOUQCDLCPc3HwCCfOw/rkygLtxKShMEcZDp2gH3AZo0z1AKBDGWyi7XyMKa
>> MO4DkMHhp28aRZ84Dtsyvxh/HuxBt9Vl2hj/oOkBMVbfrIT+3bg=
>> =q6xp
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
12