tomcat8/java8 "broken pipe" error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

tomcat8/java8 "broken pipe" error

Lund, Holly (CONTR)

Continuously receiving the below error after upgrade to Tomcat  8.0.43 /java 1.8.0_162/Apache 2.4.25 from tomcat6/java6/apache2.2 on solaris 10 OS

This only happens under load


29-May-2018 11:30:22.677 WARNING [commons-pool-EvictionTimer] org.apache.tomcat.dbcp.dbcp2.SwallowedExceptionLogger.onSwallowE
xception An internal object pool swallowed an Exception.
 java.sql.SQLRecoverableException: IO Error: Broken pipe (Write failed)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:682)
        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:711)
        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:30)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:558)
        at org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
        at org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:255)
        at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:888)
        at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureIdle(GenericObjectPool.java:952)
        at org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureMinIdle(GenericObjectPool.java:931)
        at org.apache.tomcat.dbcp.pool2.impl.BaseGenericObjectPool$Evictor.run(BaseGenericObjectPool.java:1047)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
        at oracle.net.ns.DataPacket.send(DataPacket.java:209)
        at oracle.net.ns.NetOutputStream.write(NetOutputStream.java:180)
        at oracle.net.ns.NetOutputStream.write(NetOutputStream.java:136)
        at oracle.net.ano.AnoComm.a(Unknown Source)
        at oracle.net.ano.Ano.negotiation(Unknown Source)
        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:292)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
        ... 12 more


It appears that tomcat is disconnecting from the database after 60 seconds  (from Oracle logs)

Context.xml


        <Resource name="jdbc/ataaps" auth="Container"
              type="javax.sql.DataSource" driverClassName="oracle.jdbc.OracleDriver"
              url="jdbc:oracle:thin:@//averna.doe.gov:11900/ataaps.averna.doe.gov"
              username="xxxxxxx" password="xxxxxxxxxx" maxTotal="-1"   maxIdle="100" minIdle="5"
              maxWaitMillis="300000" removeAbandonedOnMaintenance="true" removeAbandonedTimeout="300"
              logAbandoned="false" testOnBorrow="true" testOnReturn="false"
              timeBetweenEvictionRunsMillis="300000" minEvictableIdleTimeMillis="120000"
              defaultAutoCommit="true" initialSize="5" testWhileIdle="false" numTestsPerEvictionRun="5"
              validationQuery="SELECT 1 FROM dual"/>






Holly Lund
Senior Unix Systems Administrator
Criterion Systems Inc  - Contractor to the OCIO IM-61
U. S. Department of Energy
1000 Independence Ave., SW
Washington, DC 20585
Phone: 202-586-4431 (Work) 301-905-6576 (Cell)
Email: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: tomcat8/java8 "broken pipe" error

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Holly,

On 6/5/18 7:23 AM, Lund, Holly (CONTR) wrote:

>
> Continuously receiving the below error after upgrade to Tomcat
> 8.0.43 /java 1.8.0_162/Apache 2.4.25 from tomcat6/java6/apache2.2
> on solaris 10 OS
>
> This only happens under load
>
>
> 29-May-2018 11:30:22.677 WARNING [commons-pool-EvictionTimer]
> org.apache.tomcat.dbcp.dbcp2.SwallowedExceptionLogger.onSwallowE
> xception An internal object pool swallowed an Exception.
> java.sql.SQLRecoverableException: IO Error: Broken pipe (Write
> failed) at
> oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:682) at
> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:7
11)
>
>
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
> at
> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension
.java:30)
>
>
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:558)
> at
> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(
DriverConnectionFactory.java:38)
>
>
at
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Poolab
leConnectionFactory.java:255)
> at
> org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObje
ctPool.java:888)
>
>
at
org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureIdle(GenericOb
jectPool.java:952)
> at
> org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureMinIdle(Gene
ricObjectPool.java:931)
>
>
at
org.apache.tomcat.dbcp.pool2.impl.BaseGenericObjectPool$Evictor.run(Base
GenericObjectPool.java:1047)
> at java.util.TimerThread.mainLoop(Timer.java:555) at
> java.util.TimerThread.run(Timer.java:505) Caused by:
> java.net.SocketException: Broken pipe (Write failed) at
> java.net.SocketOutputStream.socketWrite0(Native Method) at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
>
>
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

> at oracle.net.ns.DataPacket.send(DataPacket.java:209) at
> oracle.net.ns.NetOutputStream.write(NetOutputStream.java:180) at
> oracle.net.ns.NetOutputStream.write(NetOutputStream.java:136) at
> oracle.net.ano.AnoComm.a(Unknown Source) at
> oracle.net.ano.Ano.negotiation(Unknown Source) at
> oracle.net.ns.NSProtocol.connect(NSProtocol.java:292) at
> oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
> at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
> ... 12 more
>
>
> It appears that tomcat is disconnecting from the database after 60
> seconds  (from Oracle logs)
>
> Context.xml
>
>
> <Resource name="jdbc/ataaps" auth="Container"
> type="javax.sql.DataSource"
> driverClassName="oracle.jdbc.OracleDriver"
> url="jdbc:oracle:thin:@//averna.doe.gov:11900/ataaps.averna.doe.gov"
>
>
username="xxxxxxx" password="xxxxxxxxxx" maxTotal="-1"   maxIdle="100"
minIdle="5"
> maxWaitMillis="300000" removeAbandonedOnMaintenance="true"
> removeAbandonedTimeout="300" logAbandoned="false"
> testOnBorrow="true" testOnReturn="false"
> timeBetweenEvictionRunsMillis="300000"
> minEvictableIdleTimeMillis="120000" defaultAutoCommit="true"
> initialSize="5" testWhileIdle="false" numTestsPerEvictionRun="5"
> validationQuery="SELECT 1 FROM dual"/>

I would check the settings on any firewall you have between the
application and the database. Perhaps there is a 60-second
connection-time-limit being imposed there?

Also perhaps check to see if the database has a connection-limit for
the user you are using. maxTotal="-1" seems ... like a bad idea.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlsWplwACgkQHPApP6U8
pFjrhA/9FnQlYQ6yIyZhI+VJtnr+tO9JZJCX+kibFM9v9kLMKMhCDvoFvnuiJmDc
nDlC2Al3uGqF8Py51CN8mkboZjqNORsr8yQzGpliLZTAyKkGLNq56tbejEwghLNF
6+g8HZDasz8V+8Yk0QL9QipNTW0+OinZORX38Bipvl6s7PtgYsSa1YlbEKpoO3gp
ePpE+Iy8CePy3/uyx7UhL642ANE5fjh5pjP0q9DKbYC6dwH/vrYeRWoEqMnlm5oe
MN3HvU6S2eIDil4XJ0nHPvGeIvfIS1fABu4JllOsWCobuzTVTFXlXXkHM5zfTf69
Q3hnlO5HGR0jQPtJzpg9eSQZMFEh77xIyZ7lugyHweWGgyMBDar1fdo/wCuTCTdr
X2Ec+NpY5MzZCfAWyJk/QVKN+a8ST8YQ+5FLfGcBqUVpxR6Od6Z05ZtHtw1iPcHE
YRKMJLoV+Zf9dAruiS5HYuqC0fqi+k+mqtRKXmpFzLTyzAf29V+eXPL61DjSR7nh
SvzLNPgSQssW0SYkTPItZBMC2Yq6hRnHLEXMBrAiQfNqeGOXEcUVOj/HwNc9qtRe
DpcQSJ9qv3P2kbsDmQptz/7nn44OEQ1OtybG7OUIlsYdshnReL0WT9hhN2S+yiLO
1IkKJxSfyWaHa/W60c4lQ0kkFydfIFuzzpcpIJ1p18bMkSo6ffc=
=OWPV
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: tomcat8/java8 "broken pipe" error

Suvendu Sekhar Mondal
On Tue, Jun 5, 2018, 8:34 PM Christopher Schultz <
[hidden email]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Holly,
>
> On 6/5/18 7:23 AM, Lund, Holly (CONTR) wrote:
> >
> > Continuously receiving the below error after upgrade to Tomcat
> > 8.0.43 /java 1.8.0_162/Apache 2.4.25 from tomcat6/java6/apache2.2
> > on solaris 10 OS
> >
> > This only happens under load
> >
> >
> > 29-May-2018 11:30:22.677 WARNING [commons-pool-EvictionTimer]
> > org.apache.tomcat.dbcp.dbcp2.SwallowedExceptionLogger.onSwallowE
> > xception An internal object pool swallowed an Exception.
> > java.sql.SQLRecoverableException: IO Error: Broken pipe (Write
> > failed) at
> > oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:682) at
> > oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:7
> 11)
> >
> >
> at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
> > at
> > oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension
> .java:30)
> >
> >
> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:558)
> > at
> > org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(
> DriverConnectionFactory.java:38)
> >
> >
> at
> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Poolab
> leConnectionFactory.java:255)
> > at
> > org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.create(GenericObje
> ctPool.java:888)
> >
> >
> at
> org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureIdle(GenericOb
> jectPool.java:952)
> > at
> > org.apache.tomcat.dbcp.pool2.impl.GenericObjectPool.ensureMinIdle(Gene
> ricObjectPool.java:931)
> >
> >
> at
> org.apache.tomcat.dbcp.pool2.impl.BaseGenericObjectPool$Evictor.run(Base
> GenericObjectPool.java:1047)
> > at java.util.TimerThread.mainLoop(Timer.java:555) at
> > java.util.TimerThread.run(Timer.java:505) Caused by:
> > java.net.SocketException: Broken pipe (Write failed) at
> > java.net.SocketOutputStream.socketWrite0(Native Method) at
> > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
> >
> >
> at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
> > at oracle.net.ns.DataPacket.send(DataPacket.java:209) at
> > oracle.net.ns.NetOutputStream.write(NetOutputStream.java:180) at
> > oracle.net.ns.NetOutputStream.write(NetOutputStream.java:136) at
> > oracle.net.ano.AnoComm.a(Unknown Source) at
> > oracle.net.ano.Ano.negotiation(Unknown Source) at
> > oracle.net.ns.NSProtocol.connect(NSProtocol.java:292) at
> > oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
> > at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
> > ... 12 more
> >
> >
> > It appears that tomcat is disconnecting from the database after 60
> > seconds  (from Oracle logs)
> >
> > Context.xml
> >
> >
> > <Resource name="jdbc/ataaps" auth="Container"
> > type="javax.sql.DataSource"
> > driverClassName="oracle.jdbc.OracleDriver"
> > url="jdbc:oracle:thin:@//averna.doe.gov:11900/ataaps.averna.doe.gov"
> >
> >
> username="xxxxxxx" password="xxxxxxxxxx" maxTotal="-1"   maxIdle="100"
> minIdle="5"
> > maxWaitMillis="300000" removeAbandonedOnMaintenance="true"
> > removeAbandonedTimeout="300" logAbandoned="false"
> > testOnBorrow="true" testOnReturn="false"
> > timeBetweenEvictionRunsMillis="300000"
> > minEvictableIdleTimeMillis="120000" defaultAutoCommit="true"
> > initialSize="5" testWhileIdle="false" numTestsPerEvictionRun="5"
> > validationQuery="SELECT 1 FROM dual"/>
>
> I would check the settings on any firewall you have between the
> application and the database. Perhaps there is a 60-second
> connection-time-limit being imposed there?
>
> Also perhaps check to see if the database has a connection-limit for
> the user you are using. maxTotal="-1" seems ... like a bad idea.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlsWplwACgkQHPApP6U8
> pFjrhA/9FnQlYQ6yIyZhI+VJtnr+tO9JZJCX+kibFM9v9kLMKMhCDvoFvnuiJmDc
> nDlC2Al3uGqF8Py51CN8mkboZjqNORsr8yQzGpliLZTAyKkGLNq56tbejEwghLNF
> 6+g8HZDasz8V+8Yk0QL9QipNTW0+OinZORX38Bipvl6s7PtgYsSa1YlbEKpoO3gp
> ePpE+Iy8CePy3/uyx7UhL642ANE5fjh5pjP0q9DKbYC6dwH/vrYeRWoEqMnlm5oe
> MN3HvU6S2eIDil4XJ0nHPvGeIvfIS1fABu4JllOsWCobuzTVTFXlXXkHM5zfTf69
> Q3hnlO5HGR0jQPtJzpg9eSQZMFEh77xIyZ7lugyHweWGgyMBDar1fdo/wCuTCTdr
> X2Ec+NpY5MzZCfAWyJk/QVKN+a8ST8YQ+5FLfGcBqUVpxR6Od6Z05ZtHtw1iPcHE
> YRKMJLoV+Zf9dAruiS5HYuqC0fqi+k+mqtRKXmpFzLTyzAf29V+eXPL61DjSR7nh
> SvzLNPgSQssW0SYkTPItZBMC2Yq6hRnHLEXMBrAiQfNqeGOXEcUVOj/HwNc9qtRe
> DpcQSJ9qv3P2kbsDmQptz/7nn44OEQ1OtybG7OUIlsYdshnReL0WT9hhN2S+yiLO
> 1IkKJxSfyWaHa/W60c4lQ0kkFydfIFuzzpcpIJ1p18bMkSo6ffc=
> =OWPV
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


Holly,

That error generally indicates a connection failure. App was trying to
write to a connection which was already closed.

As Chris indicated, definitely there is some timeout set between Tomcat and
your database. Please look for that.

One interesting thing is that you are getting it under load testing. Couple
of things:
1.Possibly your app server was very busy under load and it took 60+sec to
process the request. After processing while it tried to write back to an
open connection issue occurred. To identify that, you need to monitor JVMs
health. Watch for high memory usage and long/frequent GC pauses.

2.With you current setting, app is allowed to open unlimited DB connection,
which is not a good idea. It can overload your DB server. Situation will
get worse if there is some connection leakage. Please monitor jdbc
connection pool usage during problem.