Memory leak with Jersey 2.33

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory leak with Jersey 2.33

Mark Eggers
Folks,

This is probably not a Tomcat issue, but any thoughts on how to resolve
this would be greatly appreciated.

I am running into an apparent ClassLoader leak with the following
configuration:

Windows 10 Professional (64 bit, latest updates)
OpenJDK 11.0.10
Apache Tomcat 7.0.107
Jersey Rest 2.33

I have a simple application:

ApplicationConfig:

@ApplicationPath("/service")
public class ApplicationConfig extends Application {

     public ApplicationConfig() {
     }

     @Override
     public Set<Class<?>> getClasses(){
         Set<Class<?>> resources = new java.util.HashSet<>();

 
resources.add(org.mdeggers.cplanapi.resource.CostInfoResource.class);
 
resources.add(org.mdeggers.cplanapi.resource.InstituteInfoResource.class);
 
resources.add(org.mdeggers.cplanapi.resource.InstituteTypeInfoResource.class);

 
resources.add(org.mdeggers.cplanapi.mapper.NotFoundExceptionMapper.class);

         return resources;
     }
}

One of three resources:

@Path("/v1/inst")
public class InstituteInfoResource {

     public InstituteInfoResource() {
     }

     @GET
     @Path("/info")
     @Produces({MediaType.APPLICATION_JSON})
     public InstituteContainer getNationalList() {
         InstituteInfo instituteinfo = new InstituteInfo();
         return instituteinfo.getNational();
     }

     @GET
     @Path("/info/{ state : [A-Z]{2} }")
     @Produces({MediaType.APPLICATION_JSON})
     public InstituteContainer getStateList(@PathParam("state") String
state) {
         InstituteInfo instituteinfo = new InstituteInfo();
         return instituteinfo.getState(state);
     }
}

The InstituteInfo class is a DAO that makes requests into a MySQL
database and returns a container with the information.

This all works, but on undeploying the Tomcat Manager complains that
there is a leak after unloading the web application.

Doing a heap dump and looking at it with the following OQL in Eclipse
MAT shows that the application still resides in memory, with started=false.

SELECT wcl.contextName.toString() AS contextName,
    wcl.started AS started,
    wcl.@retainedHeapSize AS retainedSize
FROM org.apache.catalina.loader.WebappClassLoader wcl

contextName   started      retainedSize
/cplanapi     false        586,720
/docs         true          46,664
/host-manager true          47,288
/manager      true          47,392
/examples     true          85,432
               true          46,912

GC roots for cplanapi are:

class com.sun.naming.internal.ResourceManager
'- propertiesCache java.util.WeakHashMap
    '- table java.util.WeakHashMap$Entry[16]
       '- java.util.WeakHashMap$Entry
          '- referent org.apache.catalina.loader.WebappClassLoader

This occurs on both Windows and Linux, and with the above JRE as well as
Java 1.8.0_202.

. . . just my two cents
/mde/


OpenPGP_signature (505 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak with Jersey 2.33

Mark Eggers
On 1/31/2021 9:39 PM, Mark Eggers wrote:

> Folks,
>
> This is probably not a Tomcat issue, but any thoughts on how to resolve
> this would be greatly appreciated.
>
> I am running into an apparent ClassLoader leak with the following
> configuration:
>
> Windows 10 Professional (64 bit, latest updates)
> OpenJDK 11.0.10
> Apache Tomcat 7.0.107
> Jersey Rest 2.33
>
> I have a simple application:
>
> ApplicationConfig:
>
> @ApplicationPath("/service")
> public class ApplicationConfig extends Application {
>
>      public ApplicationConfig() {
>      }
>
>      @Override
>      public Set<Class<?>> getClasses(){
>          Set<Class<?>> resources = new java.util.HashSet<>();
>
>
> resources.add(org.mdeggers.cplanapi.resource.CostInfoResource.class);
>
> resources.add(org.mdeggers.cplanapi.resource.InstituteInfoResource.class);
>
> resources.add(org.mdeggers.cplanapi.resource.InstituteTypeInfoResource.class);
>
>
>
> resources.add(org.mdeggers.cplanapi.mapper.NotFoundExceptionMapper.class);
>
>          return resources;
>      }
> }
>
> One of three resources:
>
> @Path("/v1/inst")
> public class InstituteInfoResource {
>
>      public InstituteInfoResource() {
>      }
>
>      @GET
>      @Path("/info")
>      @Produces({MediaType.APPLICATION_JSON})
>      public InstituteContainer getNationalList() {
>          InstituteInfo instituteinfo = new InstituteInfo();
>          return instituteinfo.getNational();
>      }
>
>      @GET
>      @Path("/info/{ state : [A-Z]{2} }")
>      @Produces({MediaType.APPLICATION_JSON})
>      public InstituteContainer getStateList(@PathParam("state") String
> state) {
>          InstituteInfo instituteinfo = new InstituteInfo();
>          return instituteinfo.getState(state);
>      }
> }
>
> The InstituteInfo class is a DAO that makes requests into a MySQL
> database and returns a container with the information.
>
> This all works, but on undeploying the Tomcat Manager complains that
> there is a leak after unloading the web application.
>
> Doing a heap dump and looking at it with the following OQL in Eclipse
> MAT shows that the application still resides in memory, with started=false.
>
> SELECT wcl.contextName.toString() AS contextName,
>     wcl.started AS started,
>     wcl.@retainedHeapSize AS retainedSize
> FROM org.apache.catalina.loader.WebappClassLoader wcl
>
> contextName   started      retainedSize
> /cplanapi     false        586,720
> /docs         true          46,664
> /host-manager true          47,288
> /manager      true          47,392
> /examples     true          85,432
>                true          46,912
>
> GC roots for cplanapi are:
>
> class com.sun.naming.internal.ResourceManager
> '- propertiesCache java.util.WeakHashMap
>     '- table java.util.WeakHashMap$Entry[16]
>        '- java.util.WeakHashMap$Entry
>           '- referent org.apache.catalina.loader.WebappClassLoader
>
> This occurs on both Windows and Linux, and with the above JRE as well as
> Java 1.8.0_202.
>
> . . . just my two cents
> /mde/
>
Please ignore this. Doing two rounds of GC through the manager removed
the application from memory (as confirmed by a heap dump).

Now I need to figure out how to force a GC, since we had an OOM
Metaspace issue with repeated deployments of the application.

. . . just my two cents
/mde/


OpenPGP_signature (505 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak with Jersey 2.33

Mark Thomas-2
On 01/02/2021 06:37, Mark Eggers wrote:
> On 1/31/2021 9:39 PM, Mark Eggers wrote:

<snip/>

>> GC roots for cplanapi are:
>>
>> class com.sun.naming.internal.ResourceManager
>> '- propertiesCache java.util.WeakHashMap
>>     '- table java.util.WeakHashMap$Entry[16]
>>        '- java.util.WeakHashMap$Entry
>>           '- referent org.apache.catalina.loader.WebappClassLoader
>>
>> This occurs on both Windows and Linux, and with the above JRE as well
>> as Java 1.8.0_202.
>>
>> . . . just my two cents
>> /mde/
>>
>
> Please ignore this. Doing two rounds of GC through the manager removed
> the application from memory (as confirmed by a heap dump).
>
> Now I need to figure out how to force a GC, since we had an OOM
> Metaspace issue with repeated deployments of the application.

Is the GC root above the only one? I've seen similar behaviour in the
past where weak references appear to be the cause of a leak but closer
inspection uncovers a strong reference.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory leak with Jersey 2.33

Mark Eggers
Mark,

On 2/1/2021 1:33 AM, Mark Thomas wrote:
> Is the GC root above the only one? I've seen similar behaviour in the
> past where weak references appear to be the cause of a leak but closer
> inspection uncovers a strong reference.
>
> Mark

Thanks for putting me on the correct track. I'm using log4j2 2.14.0 and
Jersey REST 2.33 which includes Jackson 2.11.3.

Analyzing the heap dump using Eclipse MAT and excluding weak references
led me down a fun rabbit hole.

Leak 1:
https://github.com/FasterXML/jackson-core/issues/400

Fix:
in setenv.(bat/sh) add
-Dcom.fasterxml.jackson.core.util.BufferRecyclers.trackReusableBuffers=true

In a servlet context listener add
int released = releaseBuffers(); // with the proper import, of course

Leak 2:
https://issues.apache.org/jira/browse/LOG4J2-578

Although it's marked as fixed, it apparently is not.

Fix (for now):
in setenv(bat/sh) add
-Dlog4j2.disable.jmx=true

With all of that done, undeploying the web application and doing a heap
dump shows no traces of the offending app (all WebappClassLoaders have
started=true).

Again, thanks for the tip.

. . . just my two cents.


OpenPGP_signature (505 bytes) Download Attachment