Class loader takes long time after server inactivity - Tomcat Version 9.0.10

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Class loader takes long time after server inactivity - Tomcat Version 9.0.10

Niall Fitzpatrick
Hi Folks,



I have a web application that, after a period of inactivity (1-2hours), will stall for 3 – 35 seconds upon the first new session. All subsequent sessions will not experience this delay unless another period of inactivity occurs.



I’ve found that the delay occurs when loading jars that live in the WEB-INF/lib folder.



The issue doesn’t occur when the server is first started. Only after a period of 1-2 hours approx. will the next session experience the delay when loading a jar. This leads me to believe that when the server is started ‘knowledge’ of the jars might be cached, however, after a period of inactivity this cache may expire. The result of this is that the next session after this period causes the lib folder to be searched again which seems to take the 5-35 seconds.



My question: Is this theory above correct, and if so, is there a way to configure Tomcat such that this cache doesn’t expire so that it doesn’t need to be re-populated when a new session begins?



Tomcat Version 9.0.10



Any help appreciated,

Thanks,

Niall

Reply | Threaded
Open this post in threaded view
|

Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

markt
On 14/01/2020 13:42, Niall Fitzpatrick wrote:

> Hi Folks,
>
>
>
> I have a web application that, after a period of inactivity (1-2hours), will stall for 3 – 35 seconds upon the first new session. All subsequent sessions will not experience this delay unless another period of inactivity occurs.
>
>
>
> I’ve found that the delay occurs when loading jars that live in the WEB-INF/lib folder.
>
>
>
> The issue doesn’t occur when the server is first started. Only after a period of 1-2 hours approx. will the next session experience the delay when loading a jar. This leads me to believe that when the server is started ‘knowledge’ of the jars might be cached, however, after a period of inactivity this cache may expire. The result of this is that the next session after this period causes the lib folder to be searched again which seems to take the 5-35 seconds.
>
>
>
> My question: Is this theory above correct, and if so, is there a way to configure Tomcat such that this cache doesn’t expire so that it doesn’t need to be re-populated when a new session begins?

It depends. Which resources are slow to load?

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

M. Manna
In reply to this post by Niall Fitzpatrick
Hey Niall,

On Tue, 14 Jan 2020 at 13:42, Niall Fitzpatrick <[hidden email]>
wrote:

> Hi Folks,
>
>
>
> I have a web application that, after a period of inactivity (1-2hours),
> will stall for 3 – 35 seconds upon the first new session. All subsequent
> sessions will not experience this delay unless another period of inactivity
> occurs.
>
>
>
> I’ve found that the delay occurs when loading jars that live in the
> WEB-INF/lib folder.
>
>
>
> The issue doesn’t occur when the server is first started. Only after a
> period of 1-2 hours approx. will the next session experience the delay when
> loading a jar. This leads me to believe that when the server is started
> ‘knowledge’ of the jars might be cached, however, after a period of
> inactivity this cache may expire. The result of this is that the next
> session after this period causes the lib folder to be searched again which
> seems to take the 5-35 seconds.
>
>
>
> My question: Is this theory above correct, and if so, is there a way to
> configure Tomcat such that this cache doesn’t expire so that it doesn’t
> need to be re-populated when a new session begins?
>
>
>
> Tomcat Version 9.0.10
>
>
>
> Any help appreciated,
>
> Thanks,
>
> Niall
>

 Perhaps others can also correct me if i have missed - Tomcat does have a
resource caching and memory settings aka Resources:

https://tomcat.apache.org/tomcat-9.0-doc/config/resources.html

That said, there is very small possibility (also partly coincidence) that
this is impacted by garbage collection cycles. We have got a production
scenario where garbage collection interfered with busy hour processing due
to having lower memory. But again, that's my own issue.
I think the link above may help you configure some things?

Regards,
Reply | Threaded
Open this post in threaded view
|

Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

Niall Fitzpatrick
In reply to this post by markt




> Hi Folks,
>
>
>
> I have a web application that, after a period of inactivity (1-2hours), will stall for 3 – 35 seconds upon the first new session. All subsequent sessions will not experience this delay unless another period of inactivity occurs.
>
>
>
> I’ve found that the delay occurs when loading jars that live in the WEB-INF/lib folder.
>
>
>
> The issue doesn’t occur when the server is first started. Only after a period of 1-2 hours approx. will the next session experience the delay when loading a jar. This leads me to believe that when the server is started ‘knowledge’ of the jars might be cached, however, after a period of inactivity this cache may expire. The result of this is that the next session after this period causes the lib folder to be searched again which seems to take the 5-35 seconds.
>
>
>
> My question: Is this theory above correct, and if so, is there a way to configure Tomcat such that this cache doesn’t expire so that it doesn’t need to be re-populated when a new session begins?

It depends. Which resources are slow to load?

Mark

Hi Mark,

Doesn't seem to matter which Jar. At first it happened when loading a class from my own custom Jar. I removed this code to determine if it was the cause. However, then I found the same problem further downstream. When calling DocumentBuilderFactory.newInstance() which lives in xml-apis-1.3.04 The delay of 5-35 seconds occurred here instead.

Niall




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

markt
On 14/01/2020 14:40, Niall Fitzpatrick wrote:

>
>
>
>
>> Hi Folks,
>>
>>
>>
>> I have a web application that, after a period of inactivity (1-2hours), will stall for 3 – 35 seconds upon the first new session. All subsequent sessions will not experience this delay unless another period of inactivity occurs.
>>
>>
>>
>> I’ve found that the delay occurs when loading jars that live in the WEB-INF/lib folder.
>>
>>
>>
>> The issue doesn’t occur when the server is first started. Only after a period of 1-2 hours approx. will the next session experience the delay when loading a jar. This leads me to believe that when the server is started ‘knowledge’ of the jars might be cached, however, after a period of inactivity this cache may expire. The result of this is that the next session after this period causes the lib folder to be searched again which seems to take the 5-35 seconds.
>>
>>
>>
>> My question: Is this theory above correct, and if so, is there a way to configure Tomcat such that this cache doesn’t expire so that it doesn’t need to be re-populated when a new session begins?
>
> It depends. Which resources are slow to load?
>
> Mark
>
> Hi Mark,

Thanks for the additional info. That helps a lot.

Apologies for the following information dump but there are lots of
factors to make you aware of.

> Doesn't seem to matter which Jar. At first it happened when loading a class from my own custom Jar.

That should be a one-time cost. Once loaded, classes remain in memory
until the web application is stopped.

Finding the right class can be expensive. The more JARs the application
has, the more expensive the process will be. Tomcat does cache the list
of entries in a JAR for a short period of time to speed up this search.
If the application has been completely idle for a while then you will
see a pause on the first attempt to load a class while those caches are
refreshed.

Tomcat can't hold the cache for too long as it uses memory and locks files.

One possible solution is to use a ServletContextListener and load the
classes you know will be / are likely to be required when the
application starts.

> I removed this code to determine if it was the cause. However, then I found the same problem further downstream. When calling DocumentBuilderFactory.newInstance() which lives in xml-apis-1.3.04 The delay of 5-35 seconds occurred here instead.

Ah. That will probably be the biggest culprit. DocumentBuilderFactory
uses an SPI mechanism to find the correct instance to create. That SPI
mechanism involves searching every JAR for a configuration file. The
expectation, although it may not be documented as clearly as it should,
is that DocumentBuilderFactory.newInstance() is called only once during
the lifetime of a web application and the result cached.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

Rhuberg,Anthony
In reply to this post by Niall Fitzpatrick
Hi,

We have experienced similar behavior. I have posted previously about this issue so forgive me if I repeat myself here.

Some background first based on our recent experiences. We noticed an increase in CPU usage and disk I/O after we migrated from Tomcat 7.0.55 to Tomcat 9.0.19. What changed? Tomcat has a (Context, Engine, Host, Loader) maintenance thread that performs various tasks (backgroundProcess()) and runs every 10 seconds by default (or configurable by Engine.backgroundProcessorDelay). This maintenance thread is not new and runs in Tomcat 7/8/9; it is started for each web application after it is successfully initialized. Some of those tasks include deleting expired sessions and closing any open files handles (such as the web application jars). The Tomcat 7 WebappClassLoader.java implementation conditionally closes jars. It will only close the set of jars associated with a web application if they were not accessed within the last 90 second period called the jarOpenInterval. The purpose of the jarOpenInterval appears to be to avoid processing delays when loading classes as the WebappClassLoader is singled threaded. In Tomcat 7.0.84 or later, that period was made configurable by the jarOpenInterval property (the WebappClassLoader was refactored into WebappClassLoaderBase). The jarOpenInterval feature was removed in Tomcat 8.5 (probably because of the addition of the ParallelWebappClassLoader); therefore, in Tomcat 8.5 and later, backgroundProcess() unconditionally unloads the jars (StandardRoot.gc()). When the application requires a class that is not currently in the class loader cache, the class path is searched and then it is cached until its needed. After a period of time, we expect all the relevant classes to be loaded into the cache and the jars to close without the need to open again. However, there are cases where the jars are being repeatedly searched for the same class or file (why search again, if it was found, it should be cached?). This behavior happens regardless of what Tomcat version our web applications are deployed; however, when deployed in Tomcat 7, it is not apparent since the jars are read (searched) often enough in an active environment (within 90 seconds) and not closed.

Ok - what about DocumentBuilderFactory.newInstance()? The jar reload issue appears to be related to various class and resource search algorithms, such as ServiceLoader (https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html).
The following usages contribute to this behavior (there are others)
XML parsing
      DocumentBuilderFactory.newInstance()
 XML transformations
     TransformerFactory.newInstance()
     XPathFactory.newInstance()

How to resolve? We are taking multiple approaches. Here are a few:
1. We implement wrapper classes around any applicable factory to get the requested implementation once and then subsequently reuse; reuse could mean using a singleton instance or just caching the factory implementation class that was found and creating new instances based on that class (again avoid the search).
2. Avoid the ServiceLoader search by setting the applicable factory system property on the JVM when starting Tomcat instances : like: -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl

Here is my previous post: http://mail-archives.apache.org/mod_mbox/tomcat-users/201910.mbox/%3CDM5PR0102MB336678F059DBD91B83ED8BD580950%40DM5PR0102MB3366.prod.exchangelabs.com%3E


-----Original Message-----
From: Niall Fitzpatrick <[hidden email]>
Sent: Tuesday, January 14, 2020 9:41 AM
To: Tomcat Users List <[hidden email]>
Subject: Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10





> Hi Folks,
>
>
>
> I have a web application that, after a period of inactivity (1-2hours), will stall for 3 - 35 seconds upon the first new session. All subsequent sessions will not experience this delay unless another period of inactivity occurs.
>
>
>
> I've found that the delay occurs when loading jars that live in the WEB-INF/lib folder.
>
>
>
> The issue doesn't occur when the server is first started. Only after a period of 1-2 hours approx. will the next session experience the delay when loading a jar. This leads me to believe that when the server is started 'knowledge' of the jars might be cached, however, after a period of inactivity this cache may expire. The result of this is that the next session after this period causes the lib folder to be searched again which seems to take the 5-35 seconds.
>
>
>
> My question: Is this theory above correct, and if so, is there a way to configure Tomcat such that this cache doesn't expire so that it doesn't need to be re-populated when a new session begins?

It depends. Which resources are slow to load?

Mark

Hi Mark,

Doesn't seem to matter which Jar. At first it happened when loading a class from my own custom Jar. I removed this code to determine if it was the cause. However, then I found the same problem further downstream. When calling DocumentBuilderFactory.newInstance() which lives in xml-apis-1.3.04 The delay of 5-35 seconds occurred here instead.

Niall




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

Christopher Schultz-2
In reply to this post by markt
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 1/14/20 9:58 AM, Mark Thomas wrote:

> On 14/01/2020 14:40, Niall Fitzpatrick wrote:
>>
>>
>>
>>
>>> Hi Folks,
>>>
>>>
>>>
>>> I have a web application that, after a period of inactivity
>>> (1-2hours), will stall for 3 – 35 seconds upon the first new
>>> session. All subsequent sessions will not experience this delay
>>> unless another period of inactivity occurs.
>>>
>>>
>>>
>>> I’ve found that the delay occurs when loading jars that live in
>>> the WEB-INF/lib folder.
>>>
>>>
>>>
>>> The issue doesn’t occur when the server is first started. Only
>>> after a period of 1-2 hours approx. will the next session
>>> experience the delay when loading a jar. This leads me to
>>> believe that when the server is started ‘knowledge’ of the jars
>>> might be cached, however, after a period of inactivity this
>>> cache may expire. The result of this is that the next session
>>> after this period causes the lib folder to be searched again
>>> which seems to take the 5-35 seconds.
>>>
>>>
>>>
>>> My question: Is this theory above correct, and if so, is there
>>> a way to configure Tomcat such that this cache doesn’t expire
>>> so that it doesn’t need to be re-populated when a new session
>>> begins?
>>
>> It depends. Which resources are slow to load?
>>
>> Mark
>>
>> Hi Mark,
>
> Thanks for the additional info. That helps a lot.
>
> Apologies for the following information dump but there are lots of
> factors to make you aware of.
>
>> Doesn't seem to matter which Jar. At first it happened when
>> loading a class from my own custom Jar.
>
> That should be a one-time cost. Once loaded, classes remain in
> memory until the web application is stopped.
>
> Finding the right class can be expensive. The more JARs the
> application has, the more expensive the process will be. Tomcat
> does cache the list of entries in a JAR for a short period of time
> to speed up this search. If the application has been completely
> idle for a while then you will see a pause on the first attempt to
> load a class while those caches are refreshed.
>
> Tomcat can't hold the cache for too long as it uses memory and
> locks files.
>
> One possible solution is to use a ServletContextListener and load
> the classes you know will be / are likely to be required when the
> application starts.
>
>> I removed this code to determine if it was the cause. However,
>> then I found the same problem further downstream. When calling
>> DocumentBuilderFactory.newInstance() which lives in
>> xml-apis-1.3.04 The delay of 5-35 seconds occurred here instead.
>
> Ah. That will probably be the biggest culprit.
> DocumentBuilderFactory uses an SPI mechanism to find the correct
> instance to create. That SPI mechanism involves searching every JAR
> for a configuration file. The expectation, although it may not be
> documented as clearly as it should, is that
> DocumentBuilderFactory.newInstance() is called only once during the
> lifetime of a web application and the result cached.

+1

But 5-30 seconds is a looong time.

Niall, any chance you have a particularly slow disk, loaded server, or
maybe a network share where these files are located?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl4d7L0ACgkQHPApP6U8
pFi3RA//UMh2x8qb5l2/dPeeq6flT9SfneHpMkrwARaZv2KJbKT6esn1xQOm6bBh
Ta2luoS8rIExBbjGujLDU+Ko8yXP7gxBSj+PR9b2wKHiygWDTk/bHcGNbmDkztFA
ZyrtyOyO6kq4zcHykZxPxAk5RUAFe1Zk9ni6wVHxwqhKA1QE4luv1T13Q6ju/PXV
89sjxw4JXEnNdcV9BvbllrJ/uDwh7xXiHCrvHFxnT9HYvKNzlCQmEQOnrkLGbc9c
6rL7vQ6bo0ubIx8tL036ZPFvZkWERnedHnxR3eyO3/8Y0PDta+Br3wm/o0PDVcX6
esrUZv2UxXT/JVMrWBZWf3dO8IKSj8Q9WjU+CsRoCmrHrlRo+cuCrPj32n78M8NM
CWMqpt/oG1mbjBtLRPWXlwvk8R9/qtQKe+1fpCef6cmVjsUXpFfxNgUHxnjw28Oq
HkGdQY2Npf2+S3nepBEVQv1WAJMnwcws/Ue7U4qyQJJrKAceeqUBUi8YEWoFEOum
RGQxShWb3Qql47ozfp5q6bcfyZPRpEEa0rYBXpTkF0mlAgi3msxNw0hn6nCEcbsI
t+BH1GTba0vwR2bpy/5xXaBuGeMtX8/8AG6ol/QyfngRN2xo2ir1Skc6xtClqMJA
1/8FUUv5g3eTzrz11jNMTBWzbTuCLqYwgkzymT0gfeechI8K/dM=
=wQwj
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

Niall Fitzpatrick


________________________________
From: Christopher Schultz <[hidden email]>
Sent: 14 January 2020 16:30
To: [hidden email] <[hidden email]>
Subject: Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 1/14/20 9:58 AM, Mark Thomas wrote:

> On 14/01/2020 14:40, Niall Fitzpatrick wrote:
>>
>>
>>
>>
>>> Hi Folks,
>>>
>>>
>>>
>>> I have a web application that, after a period of inactivity
>>> (1-2hours), will stall for 3 – 35 seconds upon the first new
>>> session. All subsequent sessions will not experience this delay
>>> unless another period of inactivity occurs.
>>>
>>>
>>>
>>> I’ve found that the delay occurs when loading jars that live in
>>> the WEB-INF/lib folder.
>>>
>>>
>>>
>>> The issue doesn’t occur when the server is first started. Only
>>> after a period of 1-2 hours approx. will the next session
>>> experience the delay when loading a jar. This leads me to
>>> believe that when the server is started ‘knowledge’ of the jars
>>> might be cached, however, after a period of inactivity this
>>> cache may expire. The result of this is that the next session
>>> after this period causes the lib folder to be searched again
>>> which seems to take the 5-35 seconds.
>>>
>>>
>>>
>>> My question: Is this theory above correct, and if so, is there
>>> a way to configure Tomcat such that this cache doesn’t expire
>>> so that it doesn’t need to be re-populated when a new session
>>> begins?
>>
>> It depends. Which resources are slow to load?
>>
>> Mark
>>
>> Hi Mark,
>
> Thanks for the additional info. That helps a lot.
>
> Apologies for the following information dump but there are lots of
> factors to make you aware of.
>
>> Doesn't seem to matter which Jar. At first it happened when
>> loading a class from my own custom Jar.
>
> That should be a one-time cost. Once loaded, classes remain in
> memory until the web application is stopped.
>
> Finding the right class can be expensive. The more JARs the
> application has, the more expensive the process will be. Tomcat
> does cache the list of entries in a JAR for a short period of time
> to speed up this search. If the application has been completely
> idle for a while then you will see a pause on the first attempt to
> load a class while those caches are refreshed.
>
> Tomcat can't hold the cache for too long as it uses memory and
> locks files.
>
> One possible solution is to use a ServletContextListener and load
> the classes you know will be / are likely to be required when the
> application starts.
>
>> I removed this code to determine if it was the cause. However,
>> then I found the same problem further downstream. When calling
>> DocumentBuilderFactory.newInstance() which lives in
>> xml-apis-1.3.04 The delay of 5-35 seconds occurred here instead.
>
> Ah. That will probably be the biggest culprit.
> DocumentBuilderFactory uses an SPI mechanism to find the correct
> instance to create. That SPI mechanism involves searching every JAR
> for a configuration file. The expectation, although it may not be
> documented as clearly as it should, is that
> DocumentBuilderFactory.newInstance() is called only once during the
> lifetime of a web application and the result cached.

+1

But 5-30 seconds is a looong time.

Niall, any chance you have a particularly slow disk, loaded server, or
maybe a network share where these files are located?

- -chris


Hi Chris,
I don't believe so. A customer reported the issue, and I was able to recreate it on our own test environment. I will say this - the web application has 263 jars in it's lib folder. That's got to be a lot of searching.

Thanks for everyone's suggestions. I'm going to try the wrapper class solution where I can cache the result. I'll send an update with what I learned.

Niall

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

Rhuberg,Anthony
If it helps for reference.... Here is our singleton...

import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.TransformerFactory;
import javax.xml.xpath.XPathFactory;
import javax.xml.xpath.XPathFactoryConfigurationException;

/**
 * This class provides a set of methods to obtain a shared instance of various pre-configured XML parser factories. This
 * is intended to avoid searching the classpath when looking for an implementation of a factory. Using an enumeration to
 * create a shared instance guarantees a thread-safe singleton since the enumeration itself ensures internally that only
 * a single instance is available. Each singleton instance is constructed when the enumeration is referenced for the
 * first time. Configuration of a these factories is not expected to be thread safe; an application programmer should
 * not allow an instance of this factory to have its setter methods accessed from more than one thread.
 */
public final class XMLFactory {

    private enum DocumentBuilderFactorySingleton {
        SINGLETON;

        public final DocumentBuilderFactory instance;

        private DocumentBuilderFactorySingleton() {
            instance = DocumentBuilderFactory.newInstance();
        }
    }

    private enum TransformerFactorySingleton {
        SINGLETON;

        public final TransformerFactory instance;

        private TransformerFactorySingleton() {
            instance = TransformerFactory.newInstance();
        }
    }

    private enum XPathFactorySingleton {
        SINGLETON;

        public final XPathFactory instance;

        private XPathFactorySingleton() {
            instance = XPathFactory.newInstance();
        }
    }

    private XMLFactory() {
        return;
    }

    /**
     * This method returns a shared instance of a default document builder factory. Do not call any setter methods on
     * this shared instance. If an alternative configuration is necessary, a new factory should be created.
     * @return shared instance of document builder factory
     */
    public static final DocumentBuilderFactory getDocumentBuilderFactoryInstance() {
        return DocumentBuilderFactorySingleton.SINGLETON.instance;
    }

    /**
     * This method returns a new instance of a document builder factory.
     * @return new instance of document builder factory
     */
    public static final DocumentBuilderFactory newDocumentBuilderFactoryInstance() {
        return DocumentBuilderFactory.newInstance(getDocumentBuilderFactoryInstance().getClass().getName(), null);
    }

    /**
     * This method returns a shared instance of a default transformer factory. Do not call any setter methods on this
     * shared instance. If an alternative configuration is necessary, a new factory should be created.
     * @return shared instance of transformer factory
     */
    public static final TransformerFactory getTransformerFactoryInstance() {
        return TransformerFactorySingleton.SINGLETON.instance;
    }

    /**
     * This method returns a new instance of a transformer factory.
     * @return new instance of transformer factory
     */
    public static final TransformerFactory newTransformerFactoryInstance() {
        return TransformerFactory.newInstance(getTransformerFactoryInstance().getClass().getName(), null);
    }

    /**
     * This method returns a shared instance of a default xpath factory. Do not call any setter methods on this shared
     * instance. If an alternative configuration is necessary, a new factory should be created.
     * @return shared instance of xpath factory
     */
    public static final XPathFactory getXPathFactoryInstance() {
        return XPathFactorySingleton.SINGLETON.instance;
    }

    /**
     * This method returns a new instance of a xpath factory.
     * @return new instance of xpath factory
     * @throws XPathFactoryConfigurationException
     */
    public static final XPathFactory newXPathFactoryInstance() throws XPathFactoryConfigurationException {
        return XPathFactory.newInstance(XPathFactory.DEFAULT_OBJECT_MODEL_URI,
                getXPathFactoryInstance().getClass().getName(), null);
    }

}

-----Original Message-----
From: Niall Fitzpatrick <[hidden email]>
Sent: Tuesday, January 14, 2020 11:48 AM
To: Tomcat Users List <[hidden email]>
Subject: Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10



________________________________
From: Christopher Schultz <[hidden email]>
Sent: 14 January 2020 16:30
To: [hidden email] <[hidden email]>
Subject: Re: Class loader takes long time after server inactivity - Tomcat Version 9.0.10

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

On 1/14/20 9:58 AM, Mark Thomas wrote:

> On 14/01/2020 14:40, Niall Fitzpatrick wrote:
>>
>>
>>
>>
>>> Hi Folks,
>>>
>>>
>>>
>>> I have a web application that, after a period of inactivity
>>> (1-2hours), will stall for 3 - 35 seconds upon the first new
>>> session. All subsequent sessions will not experience this delay
>>> unless another period of inactivity occurs.
>>>
>>>
>>>
>>> I've found that the delay occurs when loading jars that live in the
>>> WEB-INF/lib folder.
>>>
>>>
>>>
>>> The issue doesn't occur when the server is first started. Only after
>>> a period of 1-2 hours approx. will the next session experience the
>>> delay when loading a jar. This leads me to believe that when the
>>> server is started 'knowledge' of the jars might be cached, however,
>>> after a period of inactivity this cache may expire. The result of
>>> this is that the next session after this period causes the lib
>>> folder to be searched again which seems to take the 5-35 seconds.
>>>
>>>
>>>
>>> My question: Is this theory above correct, and if so, is there a way
>>> to configure Tomcat such that this cache doesn't expire so that it
>>> doesn't need to be re-populated when a new session begins?
>>
>> It depends. Which resources are slow to load?
>>
>> Mark
>>
>> Hi Mark,
>
> Thanks for the additional info. That helps a lot.
>
> Apologies for the following information dump but there are lots of
> factors to make you aware of.
>
>> Doesn't seem to matter which Jar. At first it happened when loading a
>> class from my own custom Jar.
>
> That should be a one-time cost. Once loaded, classes remain in memory
> until the web application is stopped.
>
> Finding the right class can be expensive. The more JARs the
> application has, the more expensive the process will be. Tomcat does
> cache the list of entries in a JAR for a short period of time to speed
> up this search. If the application has been completely idle for a
> while then you will see a pause on the first attempt to load a class
> while those caches are refreshed.
>
> Tomcat can't hold the cache for too long as it uses memory and locks
> files.
>
> One possible solution is to use a ServletContextListener and load the
> classes you know will be / are likely to be required when the
> application starts.
>
>> I removed this code to determine if it was the cause. However, then I
>> found the same problem further downstream. When calling
>> DocumentBuilderFactory.newInstance() which lives in
>> xml-apis-1.3.04 The delay of 5-35 seconds occurred here instead.
>
> Ah. That will probably be the biggest culprit.
> DocumentBuilderFactory uses an SPI mechanism to find the correct
> instance to create. That SPI mechanism involves searching every JAR
> for a configuration file. The expectation, although it may not be
> documented as clearly as it should, is that
> DocumentBuilderFactory.newInstance() is called only once during the
> lifetime of a web application and the result cached.

+1

But 5-30 seconds is a looong time.

Niall, any chance you have a particularly slow disk, loaded server, or maybe a network share where these files are located?

- -chris


Hi Chris,
I don't believe so. A customer reported the issue, and I was able to recreate it on our own test environment. I will say this - the web application has 263 jars in it's lib folder. That's got to be a lot of searching.

Thanks for everyone's suggestions. I'm going to try the wrapper class solution where I can cache the result. I'll send an update with what I learned.

Niall

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]



CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]