Reloading JNDI <Resource>

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Reloading JNDI <Resource>

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

All,

I have a JNDI <Resource> which is a JDBC DataSource. It is set to
singleton="true" via defaults (not explicitly set).

The JDBC Connections in this DataSource pool (using dbcp2 as provided
by Tomcat) have TLS configuration including client certificates,
trusted server certificates, etc.

I'd like to be able to "bounce" the DataSource so that each Connection
is re-established in order to pick-up any new TLS configuration --
specifically, reloading the key store and trust store for the connection
.

What's the best way to do that?

It looks like Tomcat will automatically register a JMX bean for the
DataSource under
/Catalina:DataSource/[host]/[webapp]/javax.sql.DataSource/[dataSourceNam
e].
That JMX Bean has a "close" method that can be called. Calling "close"
seems to kill the DataSource, but a webapp reload can resurrect it.

There is also an "evict" method which takes zero arguments; that
appears to call the DBCP2 eviction process which removes "old" pool
members, not all.

I wasn't able to find anything else via the JMX method.

The class of the DataSource coming back from the JNDI lookup is
org.apache.tomcat.dbcp.dbcp2.BasicDataSource. It looks like I might be
able to downcast the object and call some other methods to stop /
restart it, but that seems fragile.

BasicDataSource.getConnectionPool is protected, but could be
forced-open via reflection without a SecurityManager (yuck). Then
GenericObjectPool.clear() could be called, but that only affects idle
connections in the pool and you can't be sue you've evicted all of them.

Any suggestions, other than reloading the context? I'm looking for a
zero-downtime solution.

Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8bHoUACgkQHPApP6U8
pFgfqhAAol3JNOn/PIBZ0jD53N/jfIYjycF9NhHZ3EK7wMrB/clAl6rOVvR72C0E
ljt7Q2zx69+tTezd6yOqx5KejCU6oF9bVBR1NDdwAf1QPG50m+B35k6S2Aj6o3Ge
+5RNMS2OVDYHKi5nIxS5N1y8LtVcWBbo1LQN+QTjV6e5nEztg+fLct+HtDhwNYIv
KwRDiDeztR/ulxKrDEHpuJzUHmIa+c+hD6teYLLPtEcOD11fXXO/VTtO3ZL37qCn
JhEjb5ysoqHuIUH82CjzBghsEliZPczue5u7eaGhccnXR9y9WMM59yIf2jd3AH2L
2RVtw1/txdo3YmUGLJUi5IFQkQ8C8d13dSKT7I/KqO9NFywkiIPp7kimHmLdxvK5
vGJYFvZ6GV22oHWT0OaytaqbL1v4lf58k3frtt3bLRV7+2l78XUmsGsVksBr+3E+
7S0nFw7Cc7vvDJP0euvHlZEEoK+EsnTtDLd7EoFLU6Mct2U58NT1nmp5Aepnia8/
oKSAd1SaHiPCZeqHadQl8N4dyTuRKtwbH1Hlv9+o7hnyISLcwf5Nx9PHjgVn9BEZ
aOqpC5C0JxpGLXKDt4wTwjiIXdGo8c2T7pSV/pP82GnMhWAyHvGV1DPNHTvdoEKW
Y4Z/azWfNdKPbKxHcOgu3Lfx2oDFpDn40Yx5SK823ERn/hNUTG4=
=KgAg
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Reloading JNDI <Resource>

Mark Thomas-2
On July 24, 2020 5:46:45 PM UTC, Christopher Schultz <[hidden email]> wrote:

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA256
>
>All,
>
>I have a JNDI <Resource> which is a JDBC DataSource. It is set to
>singleton="true" via defaults (not explicitly set).
>
>The JDBC Connections in this DataSource pool (using dbcp2 as provided
>by Tomcat) have TLS configuration including client certificates,
>trusted server certificates, etc.
>
>I'd like to be able to "bounce" the DataSource so that each Connection
>is re-established in order to pick-up any new TLS configuration --
>specifically, reloading the key store and trust store for the
>connection
>.
>
>What's the best way to do that?
>
>It looks like Tomcat will automatically register a JMX bean for the
>DataSource under
>/Catalina:DataSource/[host]/[webapp]/javax.sql.DataSource/[dataSourceNam
>e].
>That JMX Bean has a "close" method that can be called. Calling "close"
>seems to kill the DataSource, but a webapp reload can resurrect it.
>
>There is also an "evict" method which takes zero arguments; that
>appears to call the DBCP2 eviction process which removes "old" pool
>members, not all.
>
>I wasn't able to find anything else via the JMX method.
>
>The class of the DataSource coming back from the JNDI lookup is
>org.apache.tomcat.dbcp.dbcp2.BasicDataSource. It looks like I might be
>able to downcast the object and call some other methods to stop /
>restart it, but that seems fragile.
>
>BasicDataSource.getConnectionPool is protected, but could be
>forced-open via reflection without a SecurityManager (yuck). Then
>GenericObjectPool.clear() could be called, but that only affects idle
>connections in the pool and you can't be sue you've evicted all of
>them.
>
>Any suggestions, other than reloading the context? I'm looking for a
>zero-downtime solution.
>
>Thanks,
>- -chris
>-----BEGIN PGP SIGNATURE-----
>Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
>iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8bHoUACgkQHPApP6U8
>pFgfqhAAol3JNOn/PIBZ0jD53N/jfIYjycF9NhHZ3EK7wMrB/clAl6rOVvR72C0E
>ljt7Q2zx69+tTezd6yOqx5KejCU6oF9bVBR1NDdwAf1QPG50m+B35k6S2Aj6o3Ge
>+5RNMS2OVDYHKi5nIxS5N1y8LtVcWBbo1LQN+QTjV6e5nEztg+fLct+HtDhwNYIv
>KwRDiDeztR/ulxKrDEHpuJzUHmIa+c+hD6teYLLPtEcOD11fXXO/VTtO3ZL37qCn
>JhEjb5ysoqHuIUH82CjzBghsEliZPczue5u7eaGhccnXR9y9WMM59yIf2jd3AH2L
>2RVtw1/txdo3YmUGLJUi5IFQkQ8C8d13dSKT7I/KqO9NFywkiIPp7kimHmLdxvK5
>vGJYFvZ6GV22oHWT0OaytaqbL1v4lf58k3frtt3bLRV7+2l78XUmsGsVksBr+3E+
>7S0nFw7Cc7vvDJP0euvHlZEEoK+EsnTtDLd7EoFLU6Mct2U58NT1nmp5Aepnia8/
>oKSAd1SaHiPCZeqHadQl8N4dyTuRKtwbH1Hlv9+o7hnyISLcwf5Nx9PHjgVn9BEZ
>aOqpC5C0JxpGLXKDt4wTwjiIXdGo8c2T7pSV/pP82GnMhWAyHvGV1DPNHTvdoEKW
>Y4Z/azWfNdKPbKxHcOgu3Lfx2oDFpDn40Yx5SK823ERn/hNUTG4=
>=KgAg
>-----END PGP SIGNATURE-----
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: [hidden email]
>For additional commands, e-mail: [hidden email]

Short of code changes, I think you have identified all the solutions I can think of.

If you want to be able to force a refresh of all the objects in the pool then you'd have to start with code changes to Commons Pool.

Could you do something with the validation query? Something that queried some database provided env var that told you if the current cert was being used? That would require a round trip though which would be slow.

How about simply dropping all the connections at the database server side? The next validation query fails and picks up a new connection. Assuming you updated the files on the client side I think you would pick up the new config.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Reloading JNDI <Resource>

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 8/2/20 11:55, Mark Thomas wrote:
> On July 24, 2020 5:46:45 PM UTC, Christopher Schultz
<[hidden email]> wrote:

> All,
>
> I have a JNDI <Resource> which is a JDBC DataSource. It is set to
> singleton="true" via defaults (not explicitly set).
>
> The JDBC Connections in this DataSource pool (using dbcp2 as
> provided by Tomcat) have TLS configuration including client
> certificates, trusted server certificates, etc.
>
> I'd like to be able to "bounce" the DataSource so that each
> Connection is re-established in order to pick-up any new TLS
> configuration -- specifically, reloading the key store and trust
> store for the connection .
>
> What's the best way to do that?
>
> It looks like Tomcat will automatically register a JMX bean for
> the DataSource under
> /Catalina:DataSource/[host]/[webapp]/javax.sql.DataSource/[dataSourceN
am
>
>
e].

> That JMX Bean has a "close" method that can be called. Calling
> "close" seems to kill the DataSource, but a webapp reload can
> resurrect it.
>
> There is also an "evict" method which takes zero arguments; that
> appears to call the DBCP2 eviction process which removes "old"
> pool members, not all.
>
> I wasn't able to find anything else via the JMX method.
>
> The class of the DataSource coming back from the JNDI lookup is
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource. It looks like I might
> be able to downcast the object and call some other methods to stop
> / restart it, but that seems fragile.
>
> BasicDataSource.getConnectionPool is protected, but could be
> forced-open via reflection without a SecurityManager (yuck). Then
> GenericObjectPool.clear() could be called, but that only affects
> idle connections in the pool and you can't be sue you've evicted
> all of them.
>
> Any suggestions, other than reloading the context? I'm looking for
> a zero-downtime solution.
>
> Thanks, -chris
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>
> Short of code changes, I think you have identified all the
> solutions I can think of.> If you want to be able to force a
> refresh of all the objects in the pool then you'd have to start
> with code changes to Commons Pool.
Okay. If I can come up with some suggestions, it that something you
could look at getting into upstream? We can modify our own copy of
course (have we diverged significantly at this point? Or was that
pre-dbcp-2.x?) but upstream is better IMO if possible.

> Could you do something with the validation query? Something that
> queried some database provided env var that told you if the
> current cert was being used? That would require a round trip though
> which would be slow.
Yeah, that might be slow. I'm also not sure what my options are for
querying the current client certificate being used.

> How about simply dropping all the connections at the database
> server side? The next validation query fails and picks up a new
> connection. Assuming you updated the files on the client side I
> think you would pick up the new config.

That sounds wonderfully hacky, but what I really need to do is have
the pool reconfigure the driver with a "new" connection URL. The URL
of course won't have changed, but presumably re-initing the pool will
cause everything to be re-read, including (my goal) the keystore.

Now, maybe I'm putting too much faith in the DriverManager and/or JDBC
driver. It's possible that I will have to literally change the URL in
some trivial way so that the DM/driver doesn't just say "oh, I have
one of those already!" and return the existing objects.

Certainly further testing of those components is warranted before
proceeding any further with the connection pool. It might all be
wasted effort.

Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8oPB0ACgkQHPApP6U8
pFibNw/+MwUGt+aohxG6jQOup3TKCTC5B6aXEO6VByJq/9DOETlngiFXJGE+lctA
H3Ma+Kr+VpOvEzXxtxEDcvZlTwLy3dGWsiXeR0EDDZEoSwxDDg6LmRwWAfdT1OBo
dRc904x9N2njA302LpPLiS7auNxO0njzkCcS/J3q2GqnwMRBFPJlPWQryxwkw5HX
Ro3FtJf6Y8YbOLH7JuetQtAa7Czqs6MY8YlvqJP03hkboyX3hUQ3AwcY+XOJ7KQR
OAUdtQFqc6b3QoNtahXmv3LsQYH6me/Nc1yzCHh0hDJ22SKCDzEUN9hEyOLu3LrF
+W6Wgop7Bx75WroDWbf4aJ4fEWGzOOJn8VMN3xDJeANO0O1e/W1u4ftFOP6lEuZJ
UNvDswdj1JmcbhxKWzgkkz0IAUnnY0h49tSpYD+zkwi1wgf3yGhQY05gadhqXfH1
lN9mL7spHJrNnmmDUphoszDCkn0ZxmXU8xTl14vXHsl04FNbmJegzq99nlr1x7Z7
H3ZOVJzb3wBXK3ctKJGsJ8eJ6eIQBn5rFtd+fMqoI+2mamkvXjy7GRtnm8wAwECP
noTwHb69IW0KZtriXFSsfi3l2pBT3WtNYHZjJapeNy1uW11YnKlLOzb5ZUrtYi7N
UHSfpLP4Lk7olcoq/7yF91tIfY27FwTew2CYTCef6uL2SeZPyAk=
=DMvU
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Reloading JNDI <Resource>

Phil Steitz
In reply to this post by Christopher Schultz-2


On 7/24/20 10:46 AM, Christopher Schultz wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> All,
>
> I have a JNDI <Resource> which is a JDBC DataSource. It is set to
> singleton="true" via defaults (not explicitly set).
>
> The JDBC Connections in this DataSource pool (using dbcp2 as provided
> by Tomcat) have TLS configuration including client certificates,
> trusted server certificates, etc.
>
> I'd like to be able to "bounce" the DataSource so that each Connection
> is re-established in order to pick-up any new TLS configuration --
> specifically, reloading the key store and trust store for the connection
> .
>
> What's the best way to do that?

Do you want the connections that are checked out to clients to be
force-closed, so they will get exceptions when they try to use them, or
are you OK waiting until they get returned to the pool?

If you are OK being kind to clients, there is an enhancement request
pending for DBCP (https://issues.apache.org/jira/browse/DBCP-559)
<https://issues.apache.org/jira/browse/DBCP-559> that might do what you
want.  I  have not tested this, but I suspect that BDS closed followed
by "open" (new method proposed, to be JMX-exposed) would cause idle
connections to be closed immediately, checked out to be closed when they
come back and new ones to be created in a new pool created by the open
request.

If you really need to kill them all immediately, that would (as Mark
said) require an invalidateAll or somesuch method added to commons pool,
then exposed by DBCP as maybe closeAll.

Phil
<https://issues.apache.org/jira/browse/DBCP-559>

>
> It looks like Tomcat will automatically register a JMX bean for the
> DataSource under
> /Catalina:DataSource/[host]/[webapp]/javax.sql.DataSource/[dataSourceNam
> e].
> That JMX Bean has a "close" method that can be called. Calling "close"
> seems to kill the DataSource, but a webapp reload can resurrect it.
>
> There is also an "evict" method which takes zero arguments; that
> appears to call the DBCP2 eviction process which removes "old" pool
> members, not all.
>
> I wasn't able to find anything else via the JMX method.
>
> The class of the DataSource coming back from the JNDI lookup is
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource. It looks like I might be
> able to downcast the object and call some other methods to stop /
> restart it, but that seems fragile.
>
> BasicDataSource.getConnectionPool is protected, but could be
> forced-open via reflection without a SecurityManager (yuck). Then
> GenericObjectPool.clear() could be called, but that only affects idle
> connections in the pool and you can't be sue you've evicted all of them.
>
> Any suggestions, other than reloading the context? I'm looking for a
> zero-downtime solution.
>
> Thanks,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8bHoUACgkQHPApP6U8
> pFgfqhAAol3JNOn/PIBZ0jD53N/jfIYjycF9NhHZ3EK7wMrB/clAl6rOVvR72C0E
> ljt7Q2zx69+tTezd6yOqx5KejCU6oF9bVBR1NDdwAf1QPG50m+B35k6S2Aj6o3Ge
> +5RNMS2OVDYHKi5nIxS5N1y8LtVcWBbo1LQN+QTjV6e5nEztg+fLct+HtDhwNYIv
> KwRDiDeztR/ulxKrDEHpuJzUHmIa+c+hD6teYLLPtEcOD11fXXO/VTtO3ZL37qCn
> JhEjb5ysoqHuIUH82CjzBghsEliZPczue5u7eaGhccnXR9y9WMM59yIf2jd3AH2L
> 2RVtw1/txdo3YmUGLJUi5IFQkQ8C8d13dSKT7I/KqO9NFywkiIPp7kimHmLdxvK5
> vGJYFvZ6GV22oHWT0OaytaqbL1v4lf58k3frtt3bLRV7+2l78XUmsGsVksBr+3E+
> 7S0nFw7Cc7vvDJP0euvHlZEEoK+EsnTtDLd7EoFLU6Mct2U58NT1nmp5Aepnia8/
> oKSAd1SaHiPCZeqHadQl8N4dyTuRKtwbH1Hlv9+o7hnyISLcwf5Nx9PHjgVn9BEZ
> aOqpC5C0JxpGLXKDt4wTwjiIXdGo8c2T7pSV/pP82GnMhWAyHvGV1DPNHTvdoEKW
> Y4Z/azWfNdKPbKxHcOgu3Lfx2oDFpDn40Yx5SK823ERn/hNUTG4=
> =KgAg
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

Reply | Threaded
Open this post in threaded view
|

Re: Reloading JNDI <Resource>

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Phil,

On 8/3/20 21:43, Phil Steitz wrote:

>
>
> On 7/24/20 10:46 AM, Christopher Schultz wrote: All,
>
> I have a JNDI <Resource> which is a JDBC DataSource. It is set to
> singleton="true" via defaults (not explicitly set).
>
> The JDBC Connections in this DataSource pool (using dbcp2 as
> provided by Tomcat) have TLS configuration including client
> certificates, trusted server certificates, etc.
>
> I'd like to be able to "bounce" the DataSource so that each
> Connection is re-established in order to pick-up any new TLS
> configuration -- specifically, reloading the key store and trust
> store for the connection .
>
> What's the best way to do that?
>
>> Do you want the connections that are checked out to clients to
>> be force-closed, so they will get exceptions when they try to use
>> them, or are you OK waiting until they get returned to the pool?

Waiting is totally fine for me. The idea is to rotate a TLS
certificate in advance of its expiration. I would expect that quite a
long grace period would be tolerable (like 24 hours), but my
application only uses db connections for a few ms at a time before
returning them.

Others may have other expectations or requirements.

>> If you are OK being kind to clients, there is an enhancement
>> request pending for DBCP
>> (https://issues.apache.org/jira/browse/DBCP-559)
>> <https://issues.apache.org/jira/browse/DBCP-559> that might do
>> what you want.  I  have not tested this, but I suspect that BDS
>> closed followed by "open" (new method proposed, to be
>> JMX-exposed) would cause idle connections to be closed
>> immediately, checked out to be closed when they come back and new
>> ones to be created in a new pool created by the open request.

This sounds like exactly what I'm looking for. Subject to the JDBC
driver and DriverManager also playing along, of course.

>> If you really need to kill them all immediately, that would (as
>> Mark said) require an invalidateAll or somesuch method added to
>> commons pool, then exposed by DBCP as maybe closeAll.

Agreed, but not important for me as of yet.

Thanks!

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8pxCYACgkQHPApP6U8
pFiPbRAAvfSBbgjdFX+/n2iuhLWC+VPH+BBBeaLC+PlDXK/6QXZiUohUdmO5WO6X
ODh7N370DaLpI2feve/4by6WtNXEbqAe+SH8udrxKZFziYpKvah62sSqZvtwNAuw
vEJ8ZjT29HAGBDGI6ZD7BaQR71lX8F6/QPKNsm1UBMMuxAVvoNZVF4dvG+tE86l+
a26Pv4IEglc30RiNHxyiz4SqQR9geuA0bhd18hRrNpJ1z3gis/zUzzHrWKWlCijT
zCFgbeUp37mcFPvnnqF6Nc0nbnw7GMdom6a7ulkQUbVyJQlvnfPpX6IWSW+ZTiNh
V4h4PmB5wwzrHse9Wdt1MH7mm0j3X1FO0UJwJLbj7lcgd6oy06NBO1YNaId9EhVt
rQVKwmHQpHSWwbuFoB+rk2xztFiq3vtON8NeFW37ktlG3jrdSFO0E/p5qcHJ0yxx
2VF9LH1/kzbCqBWyIzYT2DxnqxjYLU+TjxyZ6MBufZirbMubCWo6Q9HH1Azss8cc
u869HHg0Id/xK0up+26OIlGoDnIfEIhdq7M3v7N6MgIyLVaxnkAp27Qioa8Bjmu9
L/Idm7HM/36JFsBtPwuGwf0Rsi4e4l6rNw67EN9IDjmTZaBkQ4+u+A+yBxlzfE7J
NQbykcgs0LJkalCZVtbaKyypTE2K1T5Vi07DTNGu2IkEGgRuKTE=
=210/
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]