Massive Startup Time after Server Reboot

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Massive Startup Time after Server Reboot

Jerry Malcolm
This is a weird one.  It started a few months ago.   I have TC 9 running
on Windows Server 16.  After I reboot the entire server, Tomcat takes
forever on startup.  It normally starts in about 30 seconds.  But after
a server reboot it takes up to 15 minutes... chugging along at a snail's
pace starting up all of the apps on all of the virtual hosts.  It always
finally gets there with everything successfully running.  Other servers
on the same box (Apache, JAMES, ISC BIND, MySQL) don't have any problem
starting up.  CPU, Disk, Memory, etc. usages are barely showing on the
performance graphs.  There's nothing in the Catalina log or system.err
other than showing a couple of minute gap in the time stamp between each
app as it starts up.  If I need to reboot again later, it boots again in
about 30 sec as expected.

First question... any ideas off the top of your head that might be
causing this?

Second question... is there any other logging I can turn on that might
explain what TC is blocking on?

Thanks.

Jerry


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Rainer Jung-3
Most likely it hangs waiting for enough entropy for random number
generator seeding.

Try whether the problem goes away if you add

   -Djava.security.egd=file:/dev/urandom

to you process flags. If you are using older Java than Java 8 (not
possible for Tomcat 9 but just in case you also have older software
stacks running), then it would be

   -Djava.security.egd=file:/dev/./urandom

The cryptic /./ is not a typo, you would need it for Java before Java 8.

If this doesn't help, then you should try capturing a few stack dumps
(thread dumps) during the long startup time. People here can help
interpret them.

Regards,

Rainer

Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:

> This is a weird one.  It started a few months ago.   I have TC 9 running
> on Windows Server 16.  After I reboot the entire server, Tomcat takes
> forever on startup.  It normally starts in about 30 seconds.  But after
> a server reboot it takes up to 15 minutes... chugging along at a snail's
> pace starting up all of the apps on all of the virtual hosts.  It always
> finally gets there with everything successfully running.  Other servers
> on the same box (Apache, JAMES, ISC BIND, MySQL) don't have any problem
> starting up.  CPU, Disk, Memory, etc. usages are barely showing on the
> performance graphs.  There's nothing in the Catalina log or system.err
> other than showing a couple of minute gap in the time stamp between each
> app as it starts up.  If I need to reboot again later, it boots again in
> about 30 sec as expected.
>
> First question... any ideas off the top of your head that might be
> causing this?
>
> Second question... is there any other logging I can turn on that might
> explain what TC is blocking on?
>
> Thanks.
>
> Jerry

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

info@flyingfischer.ch
Try

apt-get install haveged
update-rc.d haveged defaults

This increases the system entropy for random generation and reduces boot
time for Tomcat considerably.

Markus


Am 18.05.19 um 22:18 schrieb Rainer Jung:

> Most likely it hangs waiting for enough entropy for random number
> generator seeding.
>
> Try whether the problem goes away if you add
>
>   -Djava.security.egd=file:/dev/urandom
>
> to you process flags. If you are using older Java than Java 8 (not
> possible for Tomcat 9 but just in case you also have older software
> stacks running), then it would be
>
>   -Djava.security.egd=file:/dev/./urandom
>
> The cryptic /./ is not a typo, you would need it for Java before Java 8.
>
> If this doesn't help, then you should try capturing a few stack dumps
> (thread dumps) during the long startup time. People here can help
> interpret them.
>
> Regards,
>
> Rainer
>
> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>> This is a weird one.  It started a few months ago.   I have TC 9
>> running on Windows Server 16.  After I reboot the entire server,
>> Tomcat takes forever on startup.  It normally starts in about 30
>> seconds.  But after a server reboot it takes up to 15 minutes...
>> chugging along at a snail's pace starting up all of the apps on all
>> of the virtual hosts.  It always finally gets there with everything
>> successfully running.  Other servers on the same box (Apache, JAMES,
>> ISC BIND, MySQL) don't have any problem starting up.  CPU, Disk,
>> Memory, etc. usages are barely showing on the performance graphs. 
>> There's nothing in the Catalina log or system.err other than showing
>> a couple of minute gap in the time stamp between each app as it
>> starts up.  If I need to reboot again later, it boots again in about
>> 30 sec as expected.
>>
>> First question... any ideas off the top of your head that might be
>> causing this?
>>
>> Second question... is there any other logging I can turn on that
>> might explain what TC is blocking on?
>>
>> Thanks.
>>
>> Jerry
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

info@flyingfischer.ch
Sorry, you seem to be lost on a Windows Server...

...haveged won't help you in this situation.

Markus


Am 18.05.19 um 23:39 schrieb [hidden email]:

> Try
>
> apt-get install haveged
> update-rc.d haveged defaults
>
> This increases the system entropy for random generation and reduces boot
> time for Tomcat considerably.
>
> Markus
>
>
> Am 18.05.19 um 22:18 schrieb Rainer Jung:
>> Most likely it hangs waiting for enough entropy for random number
>> generator seeding.
>>
>> Try whether the problem goes away if you add
>>
>>   -Djava.security.egd=file:/dev/urandom
>>
>> to you process flags. If you are using older Java than Java 8 (not
>> possible for Tomcat 9 but just in case you also have older software
>> stacks running), then it would be
>>
>>   -Djava.security.egd=file:/dev/./urandom
>>
>> The cryptic /./ is not a typo, you would need it for Java before Java 8.
>>
>> If this doesn't help, then you should try capturing a few stack dumps
>> (thread dumps) during the long startup time. People here can help
>> interpret them.
>>
>> Regards,
>>
>> Rainer
>>
>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>> This is a weird one.  It started a few months ago.   I have TC 9
>>> running on Windows Server 16.  After I reboot the entire server,
>>> Tomcat takes forever on startup.  It normally starts in about 30
>>> seconds.  But after a server reboot it takes up to 15 minutes...
>>> chugging along at a snail's pace starting up all of the apps on all
>>> of the virtual hosts.  It always finally gets there with everything
>>> successfully running.  Other servers on the same box (Apache, JAMES,
>>> ISC BIND, MySQL) don't have any problem starting up.  CPU, Disk,
>>> Memory, etc. usages are barely showing on the performance graphs. 
>>> There's nothing in the Catalina log or system.err other than showing
>>> a couple of minute gap in the time stamp between each app as it
>>> starts up.  If I need to reboot again later, it boots again in about
>>> 30 sec as expected.
>>>
>>> First question... any ideas off the top of your head that might be
>>> causing this?
>>>
>>> Second question... is there any other logging I can turn on that
>>> might explain what TC is blocking on?
>>>
>>> Thanks.
>>>
>>> Jerry
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Jerry Malcolm
In reply to this post by Rainer Jung-3
Rainer,

No change with the urandom parm.  I am attaching a portion of the
Catalina log.  The first half shows between 8 and 15 seconds to deploy
each app for a single virtual host (there are no war files, the app is
already exploded in the appbase dir).  I have quite a few virtual hosts
with several apps each.  That initial server start took 21+ minutes.   I
then restarted the TC service and got the ~.5 sec start per webapp shown
in the 2nd half of the log below.

Can you refresh me on how to capture the stack dumps you suggested? 
It's been a while....

Thx.

Jerry

  First start of Tomcat after server reboot
---------------------------------------------------------------------
  [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
[8,579] ms
tory [C:\domains\<myDomain>.com\webapps\cis]
irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in
[11,486] ms
tory [C:\domains\<myDomain>.com\webapps\gallery]
irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
[9,204] ms
tory [C:\domains\<myDomain>.com\webapps\gl]
irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [8,469] ms
tory [C:\domains\<myDomain>.com\webapps\idmanager]
irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
[8,689] ms
tory [C:\domains\<myDomain>.com\webapps\itemtrack]
irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
[6,907] ms
tory [C:\domains\<myDomain>.com\webapps\malcolment]
irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in
[8,469] ms
tory [C:\domains\<myDomain>.com\webapps\notebook]
irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
[10,189] ms
tory [C:\domains\<myDomain>.com\webapps\order]
irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
[8,501] ms
tory [C:\domains\<myDomain>.com\webapps\payment]
irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
[14,209] ms
tory [C:\domains\<myDomain>.com\webapps\projectmanager]
irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has finished
in [9,018] ms
  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has finished
in [62] ms

  Restart of Tomcat
---------------------------------------------------------------------
tory [C:\domains\<myDomain>.com\webapps\cis]
irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [594] ms
tory [C:\domains\<myDomain>.com\webapps\gallery]
irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
[547] ms
tory [C:\domains\<myDomain>.com\webapps\gl]
irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [562] ms
tory [C:\domains\<myDomain>.com\webapps\idmanager]
irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
[578] ms
tory [C:\domains\<myDomain>.com\webapps\itemtrack]
irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
[547] ms
tory [C:\domains\<myDomain>.com\webapps\malcolment]
irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in
[579] ms
tory [C:\domains\<myDomain>.com\webapps\notebook]
irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
[547] ms
tory [C:\domains\<myDomain>.com\webapps\order]
irectory [C:\domains\<myDomain>.com\webapps\order] has finished in [531] ms
tory [C:\domains\<myDomain>.com\webapps\payment]
irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
[562] ms
tory [C:\domains\<myDomain>.com\webapps\projectmanager]
irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has finished
in [531] ms

On 5/18/2019 3:18 PM, Rainer Jung wrote:

> Most likely it hangs waiting for enough entropy for random number
> generator seeding.
>
> Try whether the problem goes away if you add
>
>   -Djava.security.egd=file:/dev/urandom
>
> to you process flags. If you are using older Java than Java 8 (not
> possible for Tomcat 9 but just in case you also have older software
> stacks running), then it would be
>
>   -Djava.security.egd=file:/dev/./urandom
>
> The cryptic /./ is not a typo, you would need it for Java before Java 8.
>
> If this doesn't help, then you should try capturing a few stack dumps
> (thread dumps) during the long startup time. People here can help
> interpret them.
>
> Regards,
>
> Rainer
>
> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>> This is a weird one.  It started a few months ago.   I have TC 9
>> running on Windows Server 16.  After I reboot the entire server,
>> Tomcat takes forever on startup.  It normally starts in about 30
>> seconds.  But after a server reboot it takes up to 15 minutes...
>> chugging along at a snail's pace starting up all of the apps on all
>> of the virtual hosts.  It always finally gets there with everything
>> successfully running. Other servers on the same box (Apache, JAMES,
>> ISC BIND, MySQL) don't have any problem starting up.  CPU, Disk,
>> Memory, etc. usages are barely showing on the performance graphs. 
>> There's nothing in the Catalina log or system.err other than showing
>> a couple of minute gap in the time stamp between each app as it
>> starts up.  If I need to reboot again later, it boots again in about
>> 30 sec as expected.
>>
>> First question... any ideas off the top of your head that might be
>> causing this?
>>
>> Second question... is there any other logging I can turn on that
>> might explain what TC is blocking on?
>>
>> Thanks.
>>
>> Jerry
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Jerry Malcolm
Just an FYI.... this server has been in production a little over a
year.  This TC load problem only started 2-3 months ago.  So 'something'
changed to cause this problem to manifest itself. It's been too long to
try to correlate a specific server change to the start of this problem. 
But I just wanted to point out that this problem has NOT been around
since day 1 on this server.

Jerry

On 5/19/2019 5:03 PM, Jerry Malcolm wrote:

> Rainer,
>
> No change with the urandom parm.  I am attaching a portion of the
> Catalina log.  The first half shows between 8 and 15 seconds to deploy
> each app for a single virtual host (there are no war files, the app is
> already exploded in the appbase dir).  I have quite a few virtual
> hosts with several apps each.  That initial server start took 21+
> minutes.   I then restarted the TC service and got the ~.5 sec start
> per webapp shown in the 2nd half of the log below.
>
> Can you refresh me on how to capture the stack dumps you suggested? 
> It's been a while....
>
> Thx.
>
> Jerry
>
>  First start of Tomcat after server reboot
> ---------------------------------------------------------------------
>  [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
> [8,579] ms
> tory [C:\domains\<myDomain>.com\webapps\cis]
> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in
> [11,486] ms
> tory [C:\domains\<myDomain>.com\webapps\gallery]
> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
> [9,204] ms
> tory [C:\domains\<myDomain>.com\webapps\gl]
> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in
> [8,469] ms
> tory [C:\domains\<myDomain>.com\webapps\idmanager]
> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
> [8,689] ms
> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> [6,907] ms
> tory [C:\domains\<myDomain>.com\webapps\malcolment]
> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished
> in [8,469] ms
> tory [C:\domains\<myDomain>.com\webapps\notebook]
> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> [10,189] ms
> tory [C:\domains\<myDomain>.com\webapps\order]
> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
> [8,501] ms
> tory [C:\domains\<myDomain>.com\webapps\payment]
> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
> [14,209] ms
> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
> finished in [9,018] ms
>  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has
> finished in [62] ms
>
>  Restart of Tomcat
> ---------------------------------------------------------------------
> tory [C:\domains\<myDomain>.com\webapps\cis]
> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [594] ms
> tory [C:\domains\<myDomain>.com\webapps\gallery]
> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
> [547] ms
> tory [C:\domains\<myDomain>.com\webapps\gl]
> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [562] ms
> tory [C:\domains\<myDomain>.com\webapps\idmanager]
> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
> [578] ms
> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> [547] ms
> tory [C:\domains\<myDomain>.com\webapps\malcolment]
> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished
> in [579] ms
> tory [C:\domains\<myDomain>.com\webapps\notebook]
> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> [547] ms
> tory [C:\domains\<myDomain>.com\webapps\order]
> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
> [531] ms
> tory [C:\domains\<myDomain>.com\webapps\payment]
> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
> [562] ms
> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
> finished in [531] ms
>
> On 5/18/2019 3:18 PM, Rainer Jung wrote:
>> Most likely it hangs waiting for enough entropy for random number
>> generator seeding.
>>
>> Try whether the problem goes away if you add
>>
>>   -Djava.security.egd=file:/dev/urandom
>>
>> to you process flags. If you are using older Java than Java 8 (not
>> possible for Tomcat 9 but just in case you also have older software
>> stacks running), then it would be
>>
>>   -Djava.security.egd=file:/dev/./urandom
>>
>> The cryptic /./ is not a typo, you would need it for Java before Java 8.
>>
>> If this doesn't help, then you should try capturing a few stack dumps
>> (thread dumps) during the long startup time. People here can help
>> interpret them.
>>
>> Regards,
>>
>> Rainer
>>
>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>> This is a weird one.  It started a few months ago.   I have TC 9
>>> running on Windows Server 16.  After I reboot the entire server,
>>> Tomcat takes forever on startup. It normally starts in about 30
>>> seconds.  But after a server reboot it takes up to 15 minutes...
>>> chugging along at a snail's pace starting up all of the apps on all
>>> of the virtual hosts.  It always finally gets there with everything
>>> successfully running. Other servers on the same box (Apache, JAMES,
>>> ISC BIND, MySQL) don't have any problem starting up. CPU, Disk,
>>> Memory, etc. usages are barely showing on the performance graphs. 
>>> There's nothing in the Catalina log or system.err other than showing
>>> a couple of minute gap in the time stamp between each app as it
>>> starts up.  If I need to reboot again later, it boots again in about
>>> 30 sec as expected.
>>>
>>> First question... any ideas off the top of your head that might be
>>> causing this?
>>>
>>> Second question... is there any other logging I can turn on that
>>> might explain what TC is blocking on?
>>>
>>> Thanks.
>>>
>>> Jerry
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

André Warnier (tomcat)
On 20.05.2019 00:09, Jerry Malcolm wrote:
> Just an FYI.... this server has been in production a little over a year.  This TC load
> problem only started 2-3 months ago.  So 'something' changed to cause this problem to
> manifest itself. It's been too long to try to correlate a specific server change to the
> start of this problem. But I just wanted to point out that this problem has NOT been
> around since day 1 on this server.

Just in case, since this is a Windows server ..
2-3 months ago, an update of a virus scanner ?
(of the intrusive kind which pre-checks each file that wants to be opened)
Checking may be easy or not, depending on your access : disable the virus scanner just the
time to start tomcat and check.

>
> Jerry
>
> On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
>> Rainer,
>>
>> No change with the urandom parm.  I am attaching a portion of the Catalina log.  The
>> first half shows between 8 and 15 seconds to deploy each app for a single virtual host
>> (there are no war files, the app is already exploded in the appbase dir).  I have quite
>> a few virtual hosts with several apps each.  That initial server start took 21+ minutes.
>>   I then restarted the TC service and got the ~.5 sec start per webapp shown in the 2nd
>> half of the log below.
>>
>> Can you refresh me on how to capture the stack dumps you suggested? It's been a while....
>>
>> Thx.
>>
>> Jerry
>>
>>  First start of Tomcat after server reboot
>> ---------------------------------------------------------------------
>>  [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in [8,579] ms
>> tory [C:\domains\<myDomain>.com\webapps\cis]
>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [11,486] ms
>> tory [C:\domains\<myDomain>.com\webapps\gallery]
>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in [9,204] ms
>> tory [C:\domains\<myDomain>.com\webapps\gl]
>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [8,469] ms
>> tory [C:\domains\<myDomain>.com\webapps\idmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in [8,689] ms
>> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
>> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in [6,907] ms
>> tory [C:\domains\<myDomain>.com\webapps\malcolment]
>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in [8,469] ms
>> tory [C:\domains\<myDomain>.com\webapps\notebook]
>> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in [10,189] ms
>> tory [C:\domains\<myDomain>.com\webapps\order]
>> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in [8,501] ms
>> tory [C:\domains\<myDomain>.com\webapps\payment]
>> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in [14,209] ms
>> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has finished in [9,018] ms
>>  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
>> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in [62] ms
>>
>>  Restart of Tomcat ---------------------------------------------------------------------
>> tory [C:\domains\<myDomain>.com\webapps\cis]
>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [594] ms
>> tory [C:\domains\<myDomain>.com\webapps\gallery]
>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in [547] ms
>> tory [C:\domains\<myDomain>.com\webapps\gl]
>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [562] ms
>> tory [C:\domains\<myDomain>.com\webapps\idmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in [578] ms
>> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
>> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in [547] ms
>> tory [C:\domains\<myDomain>.com\webapps\malcolment]
>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in [579] ms
>> tory [C:\domains\<myDomain>.com\webapps\notebook]
>> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in [547] ms
>> tory [C:\domains\<myDomain>.com\webapps\order]
>> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in [531] ms
>> tory [C:\domains\<myDomain>.com\webapps\payment]
>> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in [562] ms
>> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has finished in [531] ms
>>
>> On 5/18/2019 3:18 PM, Rainer Jung wrote:
>>> Most likely it hangs waiting for enough entropy for random number generator seeding.
>>>
>>> Try whether the problem goes away if you add
>>>
>>>   -Djava.security.egd=file:/dev/urandom
>>>
>>> to you process flags. If you are using older Java than Java 8 (not possible for Tomcat
>>> 9 but just in case you also have older software stacks running), then it would be
>>>
>>>   -Djava.security.egd=file:/dev/./urandom
>>>
>>> The cryptic /./ is not a typo, you would need it for Java before Java 8.
>>>
>>> If this doesn't help, then you should try capturing a few stack dumps (thread dumps)
>>> during the long startup time. People here can help interpret them.
>>>
>>> Regards,
>>>
>>> Rainer
>>>
>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>>> This is a weird one.  It started a few months ago.   I have TC 9 running on Windows
>>>> Server 16.  After I reboot the entire server, Tomcat takes forever on startup. It
>>>> normally starts in about 30 seconds.  But after a server reboot it takes up to 15
>>>> minutes... chugging along at a snail's pace starting up all of the apps on all of the
>>>> virtual hosts.  It always finally gets there with everything successfully running.
>>>> Other servers on the same box (Apache, JAMES, ISC BIND, MySQL) don't have any problem
>>>> starting up. CPU, Disk, Memory, etc. usages are barely showing on the performance
>>>> graphs. There's nothing in the Catalina log or system.err other than showing a couple
>>>> of minute gap in the time stamp between each app as it starts up.  If I need to reboot
>>>> again later, it boots again in about 30 sec as expected.
>>>>
>>>> First question... any ideas off the top of your head that might be causing this?
>>>>
>>>> Second question... is there any other logging I can turn on that might explain what TC
>>>> is blocking on?
>>>>
>>>> Thanks.
>>>>
>>>> Jerry
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Massive Startup Time after Server Reboot

Louis Zipes-2
Can you refresh me on how to capture the stack dumps you suggested?
It's been a while....

>In Task Manager find the Tomcat process, right click on it and there is an option to create a dump file.

>Also, if you have access to jconsole or other JMX tool you can connect  to it and see what is going on.

-----Original Message-----
From: André Warnier (tomcat) [mailto:[hidden email]]
Sent: Monday, May 20, 2019 4:44 AM
To: [hidden email]
Subject: Re: Massive Startup Time after Server Reboot

- - - external message, proceed with caution - - -


On 20.05.2019 00:09, Jerry Malcolm wrote:
> Just an FYI.... this server has been in production a little over a year.  This TC load
> problem only started 2-3 months ago.  So 'something' changed to cause this problem to
> manifest itself. It's been too long to try to correlate a specific server change to the
> start of this problem. But I just wanted to point out that this problem has NOT been
> around since day 1 on this server.

Just in case, since this is a Windows server ..
2-3 months ago, an update of a virus scanner ?
(of the intrusive kind which pre-checks each file that wants to be opened)
Checking may be easy or not, depending on your access : disable the virus scanner just the
time to start tomcat and check.

>
> Jerry
>
> On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
>> Rainer,
>>
>> No change with the urandom parm.  I am attaching a portion of the Catalina log.  The
>> first half shows between 8 and 15 seconds to deploy each app for a single virtual host
>> (there are no war files, the app is already exploded in the appbase dir).  I have quite
>> a few virtual hosts with several apps each.  That initial server start took 21+ minutes.
>>   I then restarted the TC service and got the ~.5 sec start per webapp shown in the 2nd
>> half of the log below.
>>
>> Can you refresh me on how to capture the stack dumps you suggested? It's been a while....
>>
>> Thx.
>>
>> Jerry
>>
>>  First start of Tomcat after server reboot
>> ---------------------------------------------------------------------
>>  [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in [8,579] ms
>> tory [C:\domains\<myDomain>.com\webapps\cis]
>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [11,486] ms
>> tory [C:\domains\<myDomain>.com\webapps\gallery]
>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in [9,204] ms
>> tory [C:\domains\<myDomain>.com\webapps\gl]
>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [8,469] ms
>> tory [C:\domains\<myDomain>.com\webapps\idmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in [8,689] ms
>> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
>> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in [6,907] ms
>> tory [C:\domains\<myDomain>.com\webapps\malcolment]
>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in [8,469] ms
>> tory [C:\domains\<myDomain>.com\webapps\notebook]
>> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in [10,189] ms
>> tory [C:\domains\<myDomain>.com\webapps\order]
>> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in [8,501] ms
>> tory [C:\domains\<myDomain>.com\webapps\payment]
>> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in [14,209] ms
>> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has finished in [9,018] ms
>>  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
>> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in [62] ms
>>
>>  Restart of Tomcat ---------------------------------------------------------------------
>> tory [C:\domains\<myDomain>.com\webapps\cis]
>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [594] ms
>> tory [C:\domains\<myDomain>.com\webapps\gallery]
>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in [547] ms
>> tory [C:\domains\<myDomain>.com\webapps\gl]
>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [562] ms
>> tory [C:\domains\<myDomain>.com\webapps\idmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in [578] ms
>> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
>> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in [547] ms
>> tory [C:\domains\<myDomain>.com\webapps\malcolment]
>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in [579] ms
>> tory [C:\domains\<myDomain>.com\webapps\notebook]
>> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in [547] ms
>> tory [C:\domains\<myDomain>.com\webapps\order]
>> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in [531] ms
>> tory [C:\domains\<myDomain>.com\webapps\payment]
>> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in [562] ms
>> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
>> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has finished in [531] ms
>>
>> On 5/18/2019 3:18 PM, Rainer Jung wrote:
>>> Most likely it hangs waiting for enough entropy for random number generator seeding.
>>>
>>> Try whether the problem goes away if you add
>>>
>>>   -Djava.security.egd=file:/dev/urandom
>>>
>>> to you process flags. If you are using older Java than Java 8 (not possible for Tomcat
>>> 9 but just in case you also have older software stacks running), then it would be
>>>
>>>   -Djava.security.egd=file:/dev/./urandom
>>>
>>> The cryptic /./ is not a typo, you would need it for Java before Java 8.
>>>
>>> If this doesn't help, then you should try capturing a few stack dumps (thread dumps)
>>> during the long startup time. People here can help interpret them.
>>>
>>> Regards,
>>>
>>> Rainer
>>>
>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>>> This is a weird one.  It started a few months ago.   I have TC 9 running on Windows
>>>> Server 16.  After I reboot the entire server, Tomcat takes forever on startup. It
>>>> normally starts in about 30 seconds.  But after a server reboot it takes up to 15
>>>> minutes... chugging along at a snail's pace starting up all of the apps on all of the
>>>> virtual hosts.  It always finally gets there with everything successfully running.
>>>> Other servers on the same box (Apache, JAMES, ISC BIND, MySQL) don't have any problem
>>>> starting up. CPU, Disk, Memory, etc. usages are barely showing on the performance
>>>> graphs. There's nothing in the Catalina log or system.err other than showing a couple
>>>> of minute gap in the time stamp between each app as it starts up.  If I need to reboot
>>>> again later, it boots again in about 30 sec as expected.
>>>>
>>>> First question... any ideas off the top of your head that might be causing this?
>>>>
>>>> Second question... is there any other logging I can turn on that might explain what TC
>>>> is blocking on?
>>>>
>>>> Thanks.
>>>>
>>>> Jerry
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

---------------------------------------
CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and may contain information that is confidential, proprietary or exempt from disclosure. If you are not the intended recipient, please contact the sender immediately. Unauthorized use or distribution is prohibited and may be unlawful.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [OT] Massive Startup Time after Server Reboot

Christopher Schultz-2
In reply to this post by info@flyingfischer.ch
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Markus,

On 5/18/19 17:39, [hidden email] wrote:
> apt-get install haveged update-rc.d haveged defaults
>
> This increases the system entropy for random generation and reduces
> boot time for Tomcat considerably.

I'm a -1 on this solution because it takes a reliable source of
entropy and converts it into an unreliable source of entropy. There is
a reason that *NIX* systems usually have both /dev/random (a good,
secure source of randomness) and /dev/urandom (a *fast* source of
randomness which is reasonably good).

When there are reasons to use /dev/random -- such as generating a new
long-lived encryption key like PGP, RSA, ECC, etc. -- it's important
that these kinds of "artificial" sources of randomness aren't used.
That's why we don't use /dev/urandom for those purposes.

But haveged basically turns /dev/random into /dev/urandom and
therefore programs have no choice.

It's better to leave hadeged out of the mix and choose the right
source of randomness for your application.

- -chris

> Am 18.05.19 um 22:18 schrieb Rainer Jung:
>> Most likely it hangs waiting for enough entropy for random
>> number generator seeding.
>>
>> Try whether the problem goes away if you add
>>
>> -Djava.security.egd=file:/dev/urandom
>>
>> to you process flags. If you are using older Java than Java 8
>> (not possible for Tomcat 9 but just in case you also have older
>> software stacks running), then it would be
>>
>> -Djava.security.egd=file:/dev/./urandom
>>
>> The cryptic /./ is not a typo, you would need it for Java before
>> Java 8.
>>
>> If this doesn't help, then you should try capturing a few stack
>> dumps (thread dumps) during the long startup time. People here
>> can help interpret them.
>>
>> Regards,
>>
>> Rainer
>>
>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>> This is a weird one.  It started a few months ago.   I have TC
>>> 9 running on Windows Server 16.  After I reboot the entire
>>> server, Tomcat takes forever on startup.  It normally starts in
>>> about 30 seconds.  But after a server reboot it takes up to 15
>>> minutes... chugging along at a snail's pace starting up all of
>>> the apps on all of the virtual hosts.  It always finally gets
>>> there with everything successfully running.  Other servers on
>>> the same box (Apache, JAMES, ISC BIND, MySQL) don't have any
>>> problem starting up.  CPU, Disk, Memory, etc. usages are barely
>>> showing on the performance graphs. There's nothing in the
>>> Catalina log or system.err other than showing a couple of
>>> minute gap in the time stamp between each app as it starts up.
>>> If I need to reboot again later, it boots again in about 30 sec
>>> as expected.
>>>
>>> First question... any ideas off the top of your head that might
>>> be causing this?
>>>
>>> Second question... is there any other logging I can turn on
>>> that might explain what TC is blocking on?
>>>
>>> Thanks.
>>>
>>> Jerry
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzi20oACgkQHPApP6U8
pFiWDQ//fJVpQ6A21DUN4OEXIWQ3Fx8x/EA0t9jgxggHkHIEREvEvLU8ZcO+XVZG
u9msAYJQWvpwmb3ohbU8UdE7z6Rez+tFEyxpkRsZZPRZF83UC4RCrFEg3p9pZMNs
lBPwfVHri414iOb6BfAj5E7vixT7chq1UxP9/rvxC0Zs8OL/rdc4iAhG/ofpCzus
Lgjnxc/3DhagztTphpr+53KsLh/ibxDnrFqVZxrh7gV/RQRtSxr9OWs/Ie9Nj2gQ
JCtOvp0YyJofe9c/Pb/WYBmmmlBt1fGm95bP1h/ahBh3kFCDt+rtKZkbbI2oh8+J
1xLpkWzWVdogQxbVfDgm9mkHRSV0XwB9gMzGvcdaeuziAf7zDZdrgYJpyeS4u9Ty
ksmU7eDsefiqTEV/Vd6oQAmWQac02lK/gX3ExarSe4+8XRt6vIu/CAjbC2Fu4OBF
+tujXyZgQwaFNvFVd4l6qKJaZ1zFyRk8E2XU/TZqlRvC9xIKN5NVm3ztHn/0ytt3
4eIYNn6vsX8QjYDipJbQ7YbpdJMnaFGyMZTm6CyjUPs5YPIIPHaA6sCO325O8IbX
cFz+WSXpVA0LVhgxjh+1gpcoQmqMS2lroPqRV7BcgSjN90tupod+N++jtCcTVy6X
7tQ8JVIAMlU4d0W3hbAGOmpxTPVRdi88LnH00WQhKlkGZy/SLvw=
=N2E6
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Suvendu Sekhar Mondal
In reply to this post by Louis Zipes-2
On Mon, May 20, 2019, 5:46 PM Louis Zipes <[hidden email] wrote:

> Can you refresh me on how to capture the stack dumps you suggested?
> It's been a while....
>
> >In Task Manager find the Tomcat process, right click on it and there is
> an option to create a dump file.
>

Well, that will create Windows memory dump.

>
> >Also, if you have access to jconsole or other JMX tool you can connect
> to it and see what is going on.
>

Agreed. Alternatively you can use jstack to generate thread dumps. Exact
command will be like: jstack -l PID_of_JVM

I'll suggest to take thread dumps 1 sec apart.

But still I am surprised to see that it happens only during entire server
reboot and from your log, subsequent TC restart is not taking long time. Is
that a right statement or I am entirely misreading it? If that is true, you
can try 'atomatic delayed start' option.

>
> -----Original Message-----
> From: André Warnier (tomcat) [mailto:[hidden email]]
> Sent: Monday, May 20, 2019 4:44 AM
> To: [hidden email]
> Subject: Re: Massive Startup Time after Server Reboot
>
> - - - external message, proceed with caution - - -
>
>
> On 20.05.2019 00:09, Jerry Malcolm wrote:
> > Just an FYI.... this server has been in production a little over a
> year.  This TC load
> > problem only started 2-3 months ago.  So 'something' changed to cause
> this problem to
> > manifest itself. It's been too long to try to correlate a specific
> server change to the
> > start of this problem. But I just wanted to point out that this problem
> has NOT been
> > around since day 1 on this server.
>
> Just in case, since this is a Windows server ..
> 2-3 months ago, an update of a virus scanner ?
> (of the intrusive kind which pre-checks each file that wants to be opened)
> Checking may be easy or not, depending on your access : disable the virus
> scanner just the
> time to start tomcat and check.
>
> >
> > Jerry
> >
> > On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
> >> Rainer,
> >>
> >> No change with the urandom parm.  I am attaching a portion of the
> Catalina log.  The
> >> first half shows between 8 and 15 seconds to deploy each app for a
> single virtual host
> >> (there are no war files, the app is already exploded in the appbase
> dir).  I have quite
> >> a few virtual hosts with several apps each.  That initial server start
> took 21+ minutes.
> >>   I then restarted the TC service and got the ~.5 sec start per webapp
> shown in the 2nd
> >> half of the log below.
> >>
> >> Can you refresh me on how to capture the stack dumps you suggested?
> It's been a while....
> >>
> >> Thx.
> >>
> >> Jerry
> >>
> >>  First start of Tomcat after server reboot
> >> ---------------------------------------------------------------------
> >>  [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
> [8,579] ms
> >> tory [C:\domains\<myDomain>.com\webapps\cis]
> >> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in
> [11,486] ms
> >> tory [C:\domains\<myDomain>.com\webapps\gallery]
> >> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
> [9,204] ms
> >> tory [C:\domains\<myDomain>.com\webapps\gl]
> >> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [8,469]
> ms
> >> tory [C:\domains\<myDomain>.com\webapps\idmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
> [8,689] ms
> >> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
> >> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> [6,907] ms
> >> tory [C:\domains\<myDomain>.com\webapps\malcolment]
> >> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in
> [8,469] ms
> >> tory [C:\domains\<myDomain>.com\webapps\notebook]
> >> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> [10,189] ms
> >> tory [C:\domains\<myDomain>.com\webapps\order]
> >> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
> [8,501] ms
> >> tory [C:\domains\<myDomain>.com\webapps\payment]
> >> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
> [14,209] ms
> >> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
> finished in [9,018] ms
> >>  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
> >> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has
> finished in [62] ms
> >>
> >>  Restart of Tomcat
> ---------------------------------------------------------------------
> >> tory [C:\domains\<myDomain>.com\webapps\cis]
> >> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in [594]
> ms
> >> tory [C:\domains\<myDomain>.com\webapps\gallery]
> >> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
> [547] ms
> >> tory [C:\domains\<myDomain>.com\webapps\gl]
> >> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in [562] ms
> >> tory [C:\domains\<myDomain>.com\webapps\idmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
> [578] ms
> >> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
> >> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> [547] ms
> >> tory [C:\domains\<myDomain>.com\webapps\malcolment]
> >> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished in
> [579] ms
> >> tory [C:\domains\<myDomain>.com\webapps\notebook]
> >> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> [547] ms
> >> tory [C:\domains\<myDomain>.com\webapps\order]
> >> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
> [531] ms
> >> tory [C:\domains\<myDomain>.com\webapps\payment]
> >> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
> [562] ms
> >> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
> >> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
> finished in [531] ms
> >>
> >> On 5/18/2019 3:18 PM, Rainer Jung wrote:
> >>> Most likely it hangs waiting for enough entropy for random number
> generator seeding.
> >>>
> >>> Try whether the problem goes away if you add
> >>>
> >>>   -Djava.security.egd=file:/dev/urandom
> >>>
> >>> to you process flags. If you are using older Java than Java 8 (not
> possible for Tomcat
> >>> 9 but just in case you also have older software stacks running), then
> it would be
> >>>
> >>>   -Djava.security.egd=file:/dev/./urandom
> >>>
> >>> The cryptic /./ is not a typo, you would need it for Java before Java
> 8.
> >>>
> >>> If this doesn't help, then you should try capturing a few stack dumps
> (thread dumps)
> >>> during the long startup time. People here can help interpret them.
> >>>
> >>> Regards,
> >>>
> >>> Rainer
> >>>
> >>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
> >>>> This is a weird one.  It started a few months ago.   I have TC 9
> running on Windows
> >>>> Server 16.  After I reboot the entire server, Tomcat takes forever on
> startup. It
> >>>> normally starts in about 30 seconds.  But after a server reboot it
> takes up to 15
> >>>> minutes... chugging along at a snail's pace starting up all of the
> apps on all of the
> >>>> virtual hosts.  It always finally gets there with everything
> successfully running.
> >>>> Other servers on the same box (Apache, JAMES, ISC BIND, MySQL) don't
> have any problem
> >>>> starting up. CPU, Disk, Memory, etc. usages are barely showing on the
> performance
> >>>> graphs. There's nothing in the Catalina log or system.err other than
> showing a couple
> >>>> of minute gap in the time stamp between each app as it starts up.  If
> I need to reboot
> >>>> again later, it boots again in about 30 sec as expected.
> >>>>
> >>>> First question... any ideas off the top of your head that might be
> causing this?
> >>>>
> >>>> Second question... is there any other logging I can turn on that
> might explain what TC
> >>>> is blocking on?
> >>>>
> >>>> Thanks.
> >>>>
> >>>> Jerry
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
> ---------------------------------------
> CONFIDENTIALITY NOTICE: This message is for intended addressee(s) only and
> may contain information that is confidential, proprietary or exempt from
> disclosure. If you are not the intended recipient, please contact the
> sender immediately. Unauthorized use or distribution is prohibited and may
> be unlawful.
>
Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Jerry Malcolm
In reply to this post by André Warnier (tomcat)
Andre,

Your theory sounds like a real possibility.  It would explain why it
only blows up the first time after a reboot.  I'll test the theory
tonight during off-peak use hours for the web sites I host.

Thx.

Jerry

On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote:

> On 20.05.2019 00:09, Jerry Malcolm wrote:
>> Just an FYI.... this server has been in production a little over a
>> year.  This TC load
>> problem only started 2-3 months ago.  So 'something' changed to cause
>> this problem to
>> manifest itself. It's been too long to try to correlate a specific
>> server change to the
>> start of this problem. But I just wanted to point out that this
>> problem has NOT been
>> around since day 1 on this server.
>
> Just in case, since this is a Windows server ..
> 2-3 months ago, an update of a virus scanner ?
> (of the intrusive kind which pre-checks each file that wants to be
> opened)
> Checking may be easy or not, depending on your access : disable the
> virus scanner just the time to start tomcat and check.
>
>>
>> Jerry
>>
>> On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
>>> Rainer,
>>>
>>> No change with the urandom parm.  I am attaching a portion of the
>>> Catalina log.  The
>>> first half shows between 8 and 15 seconds to deploy each app for a
>>> single virtual host
>>> (there are no war files, the app is already exploded in the appbase
>>> dir).  I have quite
>>> a few virtual hosts with several apps each.  That initial server
>>> start took 21+ minutes.
>>>   I then restarted the TC service and got the ~.5 sec start per
>>> webapp shown in the 2nd
>>> half of the log below.
>>>
>>> Can you refresh me on how to capture the stack dumps you suggested?
>>> It's been a while....
>>>
>>> Thx.
>>>
>>> Jerry
>>>
>>>  First start of Tomcat after server reboot
>>> ---------------------------------------------------------------------
>>>  [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
>>> [8,579] ms
>>> tory [C:\domains\<myDomain>.com\webapps\cis]
>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in
>>> [11,486] ms
>>> tory [C:\domains\<myDomain>.com\webapps\gallery]
>>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
>>> [9,204] ms
>>> tory [C:\domains\<myDomain>.com\webapps\gl]
>>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in
>>> [8,469] ms
>>> tory [C:\domains\<myDomain>.com\webapps\idmanager]
>>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished
>>> in [8,689] ms
>>> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
>>> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished
>>> in [6,907] ms
>>> tory [C:\domains\<myDomain>.com\webapps\malcolment]
>>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished
>>> in [8,469] ms
>>> tory [C:\domains\<myDomain>.com\webapps\notebook]
>>> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished
>>> in [10,189] ms
>>> tory [C:\domains\<myDomain>.com\webapps\order]
>>> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
>>> [8,501] ms
>>> tory [C:\domains\<myDomain>.com\webapps\payment]
>>> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>> [14,209] ms
>>> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
>>> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>> finished in [9,018] ms
>>>  [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml]
>>> ptor [C:\Tomcat 9.0\conf\Catalina\myridetx.net\manager.xml] has
>>> finished in [62] ms
>>>
>>>  Restart of Tomcat
>>> ---------------------------------------------------------------------
>>> tory [C:\domains\<myDomain>.com\webapps\cis]
>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished in
>>> [594] ms
>>> tory [C:\domains\<myDomain>.com\webapps\gallery]
>>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has finished in
>>> [547] ms
>>> tory [C:\domains\<myDomain>.com\webapps\gl]
>>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished in
>>> [562] ms
>>> tory [C:\domains\<myDomain>.com\webapps\idmanager]
>>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has finished
>>> in [578] ms
>>> tory [C:\domains\<myDomain>.com\webapps\itemtrack]
>>> irectory [C:\domains\<myDomain>.com\webapps\itemtrack] has finished
>>> in [547] ms
>>> tory [C:\domains\<myDomain>.com\webapps\malcolment]
>>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has finished
>>> in [579] ms
>>> tory [C:\domains\<myDomain>.com\webapps\notebook]
>>> irectory [C:\domains\<myDomain>.com\webapps\notebook] has finished
>>> in [547] ms
>>> tory [C:\domains\<myDomain>.com\webapps\order]
>>> irectory [C:\domains\<myDomain>.com\webapps\order] has finished in
>>> [531] ms
>>> tory [C:\domains\<myDomain>.com\webapps\payment]
>>> irectory [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>> [562] ms
>>> tory [C:\domains\<myDomain>.com\webapps\projectmanager]
>>> irectory [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>> finished in [531] ms
>>>
>>> On 5/18/2019 3:18 PM, Rainer Jung wrote:
>>>> Most likely it hangs waiting for enough entropy for random number
>>>> generator seeding.
>>>>
>>>> Try whether the problem goes away if you add
>>>>
>>>>   -Djava.security.egd=file:/dev/urandom
>>>>
>>>> to you process flags. If you are using older Java than Java 8 (not
>>>> possible for Tomcat
>>>> 9 but just in case you also have older software stacks running),
>>>> then it would be
>>>>
>>>>   -Djava.security.egd=file:/dev/./urandom
>>>>
>>>> The cryptic /./ is not a typo, you would need it for Java before
>>>> Java 8.
>>>>
>>>> If this doesn't help, then you should try capturing a few stack
>>>> dumps (thread dumps)
>>>> during the long startup time. People here can help interpret them.
>>>>
>>>> Regards,
>>>>
>>>> Rainer
>>>>
>>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>>>> This is a weird one.  It started a few months ago.   I have TC 9
>>>>> running on Windows
>>>>> Server 16.  After I reboot the entire server, Tomcat takes forever
>>>>> on startup. It
>>>>> normally starts in about 30 seconds.  But after a server reboot it
>>>>> takes up to 15
>>>>> minutes... chugging along at a snail's pace starting up all of the
>>>>> apps on all of the
>>>>> virtual hosts.  It always finally gets there with everything
>>>>> successfully running.
>>>>> Other servers on the same box (Apache, JAMES, ISC BIND, MySQL)
>>>>> don't have any problem
>>>>> starting up. CPU, Disk, Memory, etc. usages are barely showing on
>>>>> the performance
>>>>> graphs. There's nothing in the Catalina log or system.err other
>>>>> than showing a couple
>>>>> of minute gap in the time stamp between each app as it starts up. 
>>>>> If I need to reboot
>>>>> again later, it boots again in about 30 sec as expected.
>>>>>
>>>>> First question... any ideas off the top of your head that might be
>>>>> causing this?
>>>>>
>>>>> Second question... is there any other logging I can turn on that
>>>>> might explain what TC
>>>>> is blocking on?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Jerry
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Jerry,

On 5/20/19 14:05, Jerry Malcolm wrote:
> Andre,
>
> Your theory sounds like a real possibility.  It would explain why
> it only blows up the first time after a reboot.  I'll test the
> theory tonight during off-peak use hours for the web sites I host.

If André's theory is correct, you will probably find that 2-3 months
ago the version of your virus scanner was upgraded, and all of the
whitelisted applications (like Tomcat) were lost.

If that's the case, you can probably re-whitelist Tomcat and get your
performance back. At least until the next upgrade.

- -chris

> On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote:
>> On 20.05.2019 00:09, Jerry Malcolm wrote:
>>> Just an FYI.... this server has been in production a little
>>> over a year.  This TC load problem only started 2-3 months ago.
>>> So 'something' changed to cause this problem to manifest
>>> itself. It's been too long to try to correlate a specific
>>> server change to the start of this problem. But I just wanted
>>> to point out that this problem has NOT been around since day 1
>>> on this server.
>>
>> Just in case, since this is a Windows server .. 2-3 months ago,
>> an update of a virus scanner ? (of the intrusive kind which
>> pre-checks each file that wants to be opened) Checking may be
>> easy or not, depending on your access : disable the virus scanner
>> just the time to start tomcat and check.
>>
>>>
>>> Jerry
>>>
>>> On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
>>>> Rainer,
>>>>
>>>> No change with the urandom parm.  I am attaching a portion of
>>>> the Catalina log.  The first half shows between 8 and 15
>>>> seconds to deploy each app for a single virtual host (there
>>>> are no war files, the app is already exploded in the appbase
>>>> dir).  I have quite a few virtual hosts with several apps
>>>> each.  That initial server start took 21+ minutes. I then
>>>> restarted the TC service and got the ~.5 sec start per webapp
>>>> shown in the 2nd half of the log below.
>>>>
>>>> Can you refresh me on how to capture the stack dumps you
>>>> suggested? It's been a while....
>>>>
>>>> Thx.
>>>>
>>>> Jerry
>>>>
>>>> First start of Tomcat after server reboot
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
[C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in

>>>> [8,579] ms tory [C:\domains\<myDomain>.com\webapps\cis]
>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
>>>> in [11,486] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\gallery] irectory
>>>> [C:\domains\<myDomain>.com\webapps\gallery] has finished in
>>>> [9,204] ms tory [C:\domains\<myDomain>.com\webapps\gl]
>>>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished
>>>> in [8,469] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\idmanager] irectory
>>>> [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
>>>> [8,689] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
>>>> [6,907] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\malcolment] irectory
>>>> [C:\domains\<myDomain>.com\webapps\malcolment] has finished
>>>> in [8,469] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
>>>> [10,189] ms tory [C:\domains\<myDomain>.com\webapps\order]
>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
>>>> finished in [8,501] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>>> [14,209] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>>> finished in [9,018] ms [C:\Tomcat
>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] ptor [C:\Tomcat
>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in
>>>> [62] ms
>>>>
>>>> Restart of Tomcat
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
tory [C:\domains\<myDomain>.com\webapps\cis]

>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
>>>> in [594] ms tory [C:\domains\<myDomain>.com\webapps\gallery]
>>>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has
>>>> finished in [547] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\gl] irectory
>>>> [C:\domains\<myDomain>.com\webapps\gl] has finished in [562]
>>>> ms tory [C:\domains\<myDomain>.com\webapps\idmanager]
>>>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has
>>>> finished in [578] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\malcolment]
>>>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has
>>>> finished in [579] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\order]
>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
>>>> finished in [531] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>>> [562] ms tory
>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>>> finished in [531] ms
>>>>
>>>> On 5/18/2019 3:18 PM, Rainer Jung wrote:
>>>>> Most likely it hangs waiting for enough entropy for random
>>>>> number generator seeding.
>>>>>
>>>>> Try whether the problem goes away if you add
>>>>>
>>>>> -Djava.security.egd=file:/dev/urandom
>>>>>
>>>>> to you process flags. If you are using older Java than Java
>>>>> 8 (not possible for Tomcat 9 but just in case you also have
>>>>> older software stacks running), then it would be
>>>>>
>>>>> -Djava.security.egd=file:/dev/./urandom
>>>>>
>>>>> The cryptic /./ is not a typo, you would need it for Java
>>>>> before Java 8.
>>>>>
>>>>> If this doesn't help, then you should try capturing a few
>>>>> stack dumps (thread dumps) during the long startup time.
>>>>> People here can help interpret them.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Rainer
>>>>>
>>>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>>>>> This is a weird one.  It started a few months ago.   I
>>>>>> have TC 9 running on Windows Server 16.  After I reboot
>>>>>> the entire server, Tomcat takes forever on startup. It
>>>>>> normally starts in about 30 seconds.  But after a server
>>>>>> reboot it takes up to 15 minutes... chugging along at a
>>>>>> snail's pace starting up all of the apps on all of the
>>>>>> virtual hosts.  It always finally gets there with
>>>>>> everything successfully running. Other servers on the
>>>>>> same box (Apache, JAMES, ISC BIND, MySQL) don't have any
>>>>>> problem starting up. CPU, Disk, Memory, etc. usages are
>>>>>> barely showing on the performance graphs. There's nothing
>>>>>> in the Catalina log or system.err other than showing a
>>>>>> couple of minute gap in the time stamp between each app
>>>>>> as it starts up. If I need to reboot again later, it
>>>>>> boots again in about 30 sec as expected.
>>>>>>
>>>>>> First question... any ideas off the top of your head that
>>>>>> might be causing this?
>>>>>>
>>>>>> Second question... is there any other logging I can turn
>>>>>> on that might explain what TC is blocking on?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Jerry
>>>>>
>>>>> ------------------------------------------------------------------
- ---
>>>>>
>>>>>
To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail:
>>>>> [hidden email]
>>>>>
>>>>
>>>> -------------------------------------------------------------------
- --
>>>>
>>>>
To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail:
>>>> [hidden email]
>>>>
>>>
>>> --------------------------------------------------------------------
- -
>>>
>>>
To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzkEWoACgkQHPApP6U8
pFinLRAAttjrmJnG6lwB3pd7kT5IpBk5H/MwJdXvEX8/S/k6MLLgWqWmqQhgRQAZ
UeVqJFJFDCrgXymxXNI2f7k3saVA+2Lxafjuft9WQGKs+mTSQR0s82VG6Iko5JzL
dZxA5XD+SSU6PafTMQ1hmlVMMDeug+ONqMcUY2XFoZSV6TUk6KqjMiod74We7/b8
7PSqI4VWSS6fA4/ZoPJNRvGLXtYDu1obVX3YysMb4tSky/5OenJX6rz0mwMenY0z
F2p2G2ObNlLKUq7DuAhBcLHB4lbkEv+20Zv6hqMGPlFB8rjiOpUpL86/gfdJY29z
KGgWLAw1M4rw2n0GjVihcfYfjVBA931wWwkaXpa6cXCrcyjOcgCH+OlhIqjtoHkW
w6mM9m99LqrWRwqqfiHDDxqywdqukfGANwi1QvaPgVkd0+FwGkinh5G3P9au3cLs
JKdpGgEog1hJnjkkXWwROlozKiKjMD82V474E4mwjw7nqPI/YXXL5E8VTlhm11JB
OWDZ8yNgwA//Aw7JbGdAUPqRSsSqrKIyGYr4voDd6nhs+83R/7z27IyIxgw/kzFU
6xRFOorztK5dOR5TqPZYB67qUL39++fpdayTYDs7f5Y8guajOsWgVTrvWo3bxCkg
CxUE7V91tWJYJKlxgbFZyZp4vvSzabf4dq5fxKhPTkYH9pmHhCs=
=z1un
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Jerry Malcolm
Update.... I totally disabled Windows Real Time Virus protection. 
Apparently it was 'one' of the problems.  However...

Original startup after reboot (with WIndows Real Time Virus scan) -- ~21
minutes

With Windows Real Time Virus scan totally disabled immediately after
reboot -- ~3 minutes

Second TC startup -- ~35 seconds.

3 minutes beats 21 minutes, and I guess I can live with it.  But there's
still something else slowing it down on the first startup after system
reboot.   I'll see about getting some stack dumps.

Thanks for the help.

Jerry


On 5/21/2019 9:55 AM, Christopher Schultz wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Jerry,
>
> On 5/20/19 14:05, Jerry Malcolm wrote:
>> Andre,
>>
>> Your theory sounds like a real possibility.  It would explain why
>> it only blows up the first time after a reboot.  I'll test the
>> theory tonight during off-peak use hours for the web sites I host.
> If André's theory is correct, you will probably find that 2-3 months
> ago the version of your virus scanner was upgraded, and all of the
> whitelisted applications (like Tomcat) were lost.
>
> If that's the case, you can probably re-whitelist Tomcat and get your
> performance back. At least until the next upgrade.
>
> - -chris
>
>> On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote:
>>> On 20.05.2019 00:09, Jerry Malcolm wrote:
>>>> Just an FYI.... this server has been in production a little
>>>> over a year.  This TC load problem only started 2-3 months ago.
>>>> So 'something' changed to cause this problem to manifest
>>>> itself. It's been too long to try to correlate a specific
>>>> server change to the start of this problem. But I just wanted
>>>> to point out that this problem has NOT been around since day 1
>>>> on this server.
>>> Just in case, since this is a Windows server .. 2-3 months ago,
>>> an update of a virus scanner ? (of the intrusive kind which
>>> pre-checks each file that wants to be opened) Checking may be
>>> easy or not, depending on your access : disable the virus scanner
>>> just the time to start tomcat and check.
>>>
>>>> Jerry
>>>>
>>>> On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
>>>>> Rainer,
>>>>>
>>>>> No change with the urandom parm.  I am attaching a portion of
>>>>> the Catalina log.  The first half shows between 8 and 15
>>>>> seconds to deploy each app for a single virtual host (there
>>>>> are no war files, the app is already exploded in the appbase
>>>>> dir).  I have quite a few virtual hosts with several apps
>>>>> each.  That initial server start took 21+ minutes. I then
>>>>> restarted the TC service and got the ~.5 sec start per webapp
>>>>> shown in the 2nd half of the log below.
>>>>>
>>>>> Can you refresh me on how to capture the stack dumps you
>>>>> suggested? It's been a while....
>>>>>
>>>>> Thx.
>>>>>
>>>>> Jerry
>>>>>
>>>>> First start of Tomcat after server reboot
>>>>> -------------------------------------------------------------------
> - --
>>>>>
> [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
>>>>> [8,579] ms tory [C:\domains\<myDomain>.com\webapps\cis]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
>>>>> in [11,486] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\gallery] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\gallery] has finished in
>>>>> [9,204] ms tory [C:\domains\<myDomain>.com\webapps\gl]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished
>>>>> in [8,469] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\idmanager] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
>>>>> [8,689] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
>>>>> [6,907] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\malcolment] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\malcolment] has finished
>>>>> in [8,469] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
>>>>> [10,189] ms tory [C:\domains\<myDomain>.com\webapps\order]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
>>>>> finished in [8,501] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>>>> [14,209] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>>>> finished in [9,018] ms [C:\Tomcat
>>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] ptor [C:\Tomcat
>>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in
>>>>> [62] ms
>>>>>
>>>>> Restart of Tomcat
>>>>> -------------------------------------------------------------------
> - --
>>>>>
> tory [C:\domains\<myDomain>.com\webapps\cis]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
>>>>> in [594] ms tory [C:\domains\<myDomain>.com\webapps\gallery]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has
>>>>> finished in [547] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\gl] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\gl] has finished in [562]
>>>>> ms tory [C:\domains\<myDomain>.com\webapps\idmanager]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has
>>>>> finished in [578] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
>>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\malcolment]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has
>>>>> finished in [579] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
>>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\order]
>>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
>>>>> finished in [531] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>>>> [562] ms tory
>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>>>> finished in [531] ms
>>>>>
>>>>> On 5/18/2019 3:18 PM, Rainer Jung wrote:
>>>>>> Most likely it hangs waiting for enough entropy for random
>>>>>> number generator seeding.
>>>>>>
>>>>>> Try whether the problem goes away if you add
>>>>>>
>>>>>> -Djava.security.egd=file:/dev/urandom
>>>>>>
>>>>>> to you process flags. If you are using older Java than Java
>>>>>> 8 (not possible for Tomcat 9 but just in case you also have
>>>>>> older software stacks running), then it would be
>>>>>>
>>>>>> -Djava.security.egd=file:/dev/./urandom
>>>>>>
>>>>>> The cryptic /./ is not a typo, you would need it for Java
>>>>>> before Java 8.
>>>>>>
>>>>>> If this doesn't help, then you should try capturing a few
>>>>>> stack dumps (thread dumps) during the long startup time.
>>>>>> People here can help interpret them.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Rainer
>>>>>>
>>>>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>>>>>> This is a weird one.  It started a few months ago.   I
>>>>>>> have TC 9 running on Windows Server 16.  After I reboot
>>>>>>> the entire server, Tomcat takes forever on startup. It
>>>>>>> normally starts in about 30 seconds.  But after a server
>>>>>>> reboot it takes up to 15 minutes... chugging along at a
>>>>>>> snail's pace starting up all of the apps on all of the
>>>>>>> virtual hosts.  It always finally gets there with
>>>>>>> everything successfully running. Other servers on the
>>>>>>> same box (Apache, JAMES, ISC BIND, MySQL) don't have any
>>>>>>> problem starting up. CPU, Disk, Memory, etc. usages are
>>>>>>> barely showing on the performance graphs. There's nothing
>>>>>>> in the Catalina log or system.err other than showing a
>>>>>>> couple of minute gap in the time stamp between each app
>>>>>>> as it starts up. If I need to reboot again later, it
>>>>>>> boots again in about 30 sec as expected.
>>>>>>>
>>>>>>> First question... any ideas off the top of your head that
>>>>>>> might be causing this?
>>>>>>>
>>>>>>> Second question... is there any other logging I can turn
>>>>>>> on that might explain what TC is blocking on?
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Jerry
>>>>>> ------------------------------------------------------------------
> - ---
>>>>>>
> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail:
>>>>>> [hidden email]
>>>>>>
>>>>> -------------------------------------------------------------------
> - --
>>>>>
> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail:
>>>>> [hidden email]
>>>>>
>>>> --------------------------------------------------------------------
> - -
>>>>
> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>> ---------------------------------------------------------------------
>>
>>
> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
> -----BEGIN PGP SIGNATURE-----
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzkEWoACgkQHPApP6U8
> pFinLRAAttjrmJnG6lwB3pd7kT5IpBk5H/MwJdXvEX8/S/k6MLLgWqWmqQhgRQAZ
> UeVqJFJFDCrgXymxXNI2f7k3saVA+2Lxafjuft9WQGKs+mTSQR0s82VG6Iko5JzL
> dZxA5XD+SSU6PafTMQ1hmlVMMDeug+ONqMcUY2XFoZSV6TUk6KqjMiod74We7/b8
> 7PSqI4VWSS6fA4/ZoPJNRvGLXtYDu1obVX3YysMb4tSky/5OenJX6rz0mwMenY0z
> F2p2G2ObNlLKUq7DuAhBcLHB4lbkEv+20Zv6hqMGPlFB8rjiOpUpL86/gfdJY29z
> KGgWLAw1M4rw2n0GjVihcfYfjVBA931wWwkaXpa6cXCrcyjOcgCH+OlhIqjtoHkW
> w6mM9m99LqrWRwqqfiHDDxqywdqukfGANwi1QvaPgVkd0+FwGkinh5G3P9au3cLs
> JKdpGgEog1hJnjkkXWwROlozKiKjMD82V474E4mwjw7nqPI/YXXL5E8VTlhm11JB
> OWDZ8yNgwA//Aw7JbGdAUPqRSsSqrKIyGYr4voDd6nhs+83R/7z27IyIxgw/kzFU
> 6xRFOorztK5dOR5TqPZYB67qUL39++fpdayTYDs7f5Y8guajOsWgVTrvWo3bxCkg
> CxUE7V91tWJYJKlxgbFZyZp4vvSzabf4dq5fxKhPTkYH9pmHhCs=
> =z1un
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Suvendu Sekhar Mondal
Jerry,


On Tue, May 21, 2019, 10:55 PM Jerry Malcolm <[hidden email] wrote:

> Update.... I totally disabled Windows Real Time Virus protection.
> Apparently it was 'one' of the problems.  However...
>
> Original startup after reboot (with WIndows Real Time Virus scan) -- ~21
> minutes
>
> With Windows Real Time Virus scan totally disabled immediately after
> reboot -- ~3 minutes
>

Wow!

>
> Second TC startup -- ~35 seconds.
>
> 3 minutes beats 21 minutes, and I guess I can live with it.  But there's
> still something else slowing it down on the first startup after system
> reboot.   I'll see about getting some stack dumps.
>

May I know what is the startup type of Tomcat service? If it is
'automatic', can you please try with 'automatic delayed start'?

>
> Thanks for the help.
>
> Jerry
>
>
> On 5/21/2019 9:55 AM, Christopher Schultz wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > Jerry,
> >
> > On 5/20/19 14:05, Jerry Malcolm wrote:
> >> Andre,
> >>
> >> Your theory sounds like a real possibility.  It would explain why
> >> it only blows up the first time after a reboot.  I'll test the
> >> theory tonight during off-peak use hours for the web sites I host.
> > If André's theory is correct, you will probably find that 2-3 months
> > ago the version of your virus scanner was upgraded, and all of the
> > whitelisted applications (like Tomcat) were lost.
> >
> > If that's the case, you can probably re-whitelist Tomcat and get your
> > performance back. At least until the next upgrade.
> >
> > - -chris
> >
> >> On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote:
> >>> On 20.05.2019 00:09, Jerry Malcolm wrote:
> >>>> Just an FYI.... this server has been in production a little
> >>>> over a year.  This TC load problem only started 2-3 months ago.
> >>>> So 'something' changed to cause this problem to manifest
> >>>> itself. It's been too long to try to correlate a specific
> >>>> server change to the start of this problem. But I just wanted
> >>>> to point out that this problem has NOT been around since day 1
> >>>> on this server.
> >>> Just in case, since this is a Windows server .. 2-3 months ago,
> >>> an update of a virus scanner ? (of the intrusive kind which
> >>> pre-checks each file that wants to be opened) Checking may be
> >>> easy or not, depending on your access : disable the virus scanner
> >>> just the time to start tomcat and check.
> >>>
> >>>> Jerry
> >>>>
> >>>> On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
> >>>>> Rainer,
> >>>>>
> >>>>> No change with the urandom parm.  I am attaching a portion of
> >>>>> the Catalina log.  The first half shows between 8 and 15
> >>>>> seconds to deploy each app for a single virtual host (there
> >>>>> are no war files, the app is already exploded in the appbase
> >>>>> dir).  I have quite a few virtual hosts with several apps
> >>>>> each.  That initial server start took 21+ minutes. I then
> >>>>> restarted the TC service and got the ~.5 sec start per webapp
> >>>>> shown in the 2nd half of the log below.
> >>>>>
> >>>>> Can you refresh me on how to capture the stack dumps you
> >>>>> suggested? It's been a while....
> >>>>>
> >>>>> Thx.
> >>>>>
> >>>>> Jerry
> >>>>>
> >>>>> First start of Tomcat after server reboot
> >>>>> -------------------------------------------------------------------
> > - --
> >>>>>
> > [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
> >>>>> [8,579] ms tory [C:\domains\<myDomain>.com\webapps\cis]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
> >>>>> in [11,486] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\gallery] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\gallery] has finished in
> >>>>> [9,204] ms tory [C:\domains\<myDomain>.com\webapps\gl]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished
> >>>>> in [8,469] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\idmanager] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
> >>>>> [8,689] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> >>>>> [6,907] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\malcolment] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\malcolment] has finished
> >>>>> in [8,469] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> >>>>> [10,189] ms tory [C:\domains\<myDomain>.com\webapps\order]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
> >>>>> finished in [8,501] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
> >>>>> [14,209] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
> >>>>> finished in [9,018] ms [C:\Tomcat
> >>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] ptor [C:\Tomcat
> >>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in
> >>>>> [62] ms
> >>>>>
> >>>>> Restart of Tomcat
> >>>>> -------------------------------------------------------------------
> > - --
> >>>>>
> > tory [C:\domains\<myDomain>.com\webapps\cis]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
> >>>>> in [594] ms tory [C:\domains\<myDomain>.com\webapps\gallery]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has
> >>>>> finished in [547] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\gl] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\gl] has finished in [562]
> >>>>> ms tory [C:\domains\<myDomain>.com\webapps\idmanager]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has
> >>>>> finished in [578] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
> >>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\malcolment]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has
> >>>>> finished in [579] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
> >>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\order]
> >>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
> >>>>> finished in [531] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
> >>>>> [562] ms tory
> >>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
> >>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
> >>>>> finished in [531] ms
> >>>>>
> >>>>> On 5/18/2019 3:18 PM, Rainer Jung wrote:
> >>>>>> Most likely it hangs waiting for enough entropy for random
> >>>>>> number generator seeding.
> >>>>>>
> >>>>>> Try whether the problem goes away if you add
> >>>>>>
> >>>>>> -Djava.security.egd=file:/dev/urandom
> >>>>>>
> >>>>>> to you process flags. If you are using older Java than Java
> >>>>>> 8 (not possible for Tomcat 9 but just in case you also have
> >>>>>> older software stacks running), then it would be
> >>>>>>
> >>>>>> -Djava.security.egd=file:/dev/./urandom
> >>>>>>
> >>>>>> The cryptic /./ is not a typo, you would need it for Java
> >>>>>> before Java 8.
> >>>>>>
> >>>>>> If this doesn't help, then you should try capturing a few
> >>>>>> stack dumps (thread dumps) during the long startup time.
> >>>>>> People here can help interpret them.
> >>>>>>
> >>>>>> Regards,
> >>>>>>
> >>>>>> Rainer
> >>>>>>
> >>>>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
> >>>>>>> This is a weird one.  It started a few months ago.   I
> >>>>>>> have TC 9 running on Windows Server 16.  After I reboot
> >>>>>>> the entire server, Tomcat takes forever on startup. It
> >>>>>>> normally starts in about 30 seconds.  But after a server
> >>>>>>> reboot it takes up to 15 minutes... chugging along at a
> >>>>>>> snail's pace starting up all of the apps on all of the
> >>>>>>> virtual hosts.  It always finally gets there with
> >>>>>>> everything successfully running. Other servers on the
> >>>>>>> same box (Apache, JAMES, ISC BIND, MySQL) don't have any
> >>>>>>> problem starting up. CPU, Disk, Memory, etc. usages are
> >>>>>>> barely showing on the performance graphs. There's nothing
> >>>>>>> in the Catalina log or system.err other than showing a
> >>>>>>> couple of minute gap in the time stamp between each app
> >>>>>>> as it starts up. If I need to reboot again later, it
> >>>>>>> boots again in about 30 sec as expected.
> >>>>>>>
> >>>>>>> First question... any ideas off the top of your head that
> >>>>>>> might be causing this?
> >>>>>>>
> >>>>>>> Second question... is there any other logging I can turn
> >>>>>>> on that might explain what TC is blocking on?
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> Jerry
> >>>>>> ------------------------------------------------------------------
> > - ---
> >>>>>>
> > To unsubscribe, e-mail: [hidden email]
> >>>>>> For additional commands, e-mail:
> >>>>>> [hidden email]
> >>>>>>
> >>>>> -------------------------------------------------------------------
> > - --
> >>>>>
> > To unsubscribe, e-mail: [hidden email]
> >>>>> For additional commands, e-mail:
> >>>>> [hidden email]
> >>>>>
> >>>> --------------------------------------------------------------------
> > - -
> >>>>
> > To unsubscribe, e-mail: [hidden email]
> >>>> For additional commands, e-mail: [hidden email]
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>>
> >>>
> > To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>>
> >> ---------------------------------------------------------------------
> >>
> >>
> > To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> > -----BEGIN PGP SIGNATURE-----
> > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
> >
> > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzkEWoACgkQHPApP6U8
> > pFinLRAAttjrmJnG6lwB3pd7kT5IpBk5H/MwJdXvEX8/S/k6MLLgWqWmqQhgRQAZ
> > UeVqJFJFDCrgXymxXNI2f7k3saVA+2Lxafjuft9WQGKs+mTSQR0s82VG6Iko5JzL
> > dZxA5XD+SSU6PafTMQ1hmlVMMDeug+ONqMcUY2XFoZSV6TUk6KqjMiod74We7/b8
> > 7PSqI4VWSS6fA4/ZoPJNRvGLXtYDu1obVX3YysMb4tSky/5OenJX6rz0mwMenY0z
> > F2p2G2ObNlLKUq7DuAhBcLHB4lbkEv+20Zv6hqMGPlFB8rjiOpUpL86/gfdJY29z
> > KGgWLAw1M4rw2n0GjVihcfYfjVBA931wWwkaXpa6cXCrcyjOcgCH+OlhIqjtoHkW
> > w6mM9m99LqrWRwqqfiHDDxqywdqukfGANwi1QvaPgVkd0+FwGkinh5G3P9au3cLs
> > JKdpGgEog1hJnjkkXWwROlozKiKjMD82V474E4mwjw7nqPI/YXXL5E8VTlhm11JB
> > OWDZ8yNgwA//Aw7JbGdAUPqRSsSqrKIyGYr4voDd6nhs+83R/7z27IyIxgw/kzFU
> > 6xRFOorztK5dOR5TqPZYB67qUL39++fpdayTYDs7f5Y8guajOsWgVTrvWo3bxCkg
> > CxUE7V91tWJYJKlxgbFZyZp4vvSzabf4dq5fxKhPTkYH9pmHhCs=
> > =z1un
> > -----END PGP SIGNATURE-----
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Massive Startup Time after Server Reboot

Jerry Malcolm
It's currently automatic.  I'll try delayed-start tonight and see if
there are changes.

Thx

On 5/21/2019 1:21 PM, Suvendu Sekhar Mondal wrote:

> Jerry,
>
>
> On Tue, May 21, 2019, 10:55 PM Jerry Malcolm <[hidden email] wrote:
>
>> Update.... I totally disabled Windows Real Time Virus protection.
>> Apparently it was 'one' of the problems.  However...
>>
>> Original startup after reboot (with WIndows Real Time Virus scan) -- ~21
>> minutes
>>
>> With Windows Real Time Virus scan totally disabled immediately after
>> reboot -- ~3 minutes
>>
> Wow!
>
>> Second TC startup -- ~35 seconds.
>>
>> 3 minutes beats 21 minutes, and I guess I can live with it.  But there's
>> still something else slowing it down on the first startup after system
>> reboot.   I'll see about getting some stack dumps.
>>
> May I know what is the startup type of Tomcat service? If it is
> 'automatic', can you please try with 'automatic delayed start'?
>
>> Thanks for the help.
>>
>> Jerry
>>
>>
>> On 5/21/2019 9:55 AM, Christopher Schultz wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> Jerry,
>>>
>>> On 5/20/19 14:05, Jerry Malcolm wrote:
>>>> Andre,
>>>>
>>>> Your theory sounds like a real possibility.  It would explain why
>>>> it only blows up the first time after a reboot.  I'll test the
>>>> theory tonight during off-peak use hours for the web sites I host.
>>> If André's theory is correct, you will probably find that 2-3 months
>>> ago the version of your virus scanner was upgraded, and all of the
>>> whitelisted applications (like Tomcat) were lost.
>>>
>>> If that's the case, you can probably re-whitelist Tomcat and get your
>>> performance back. At least until the next upgrade.
>>>
>>> - -chris
>>>
>>>> On 5/20/2019 3:43 AM, André Warnier (tomcat) wrote:
>>>>> On 20.05.2019 00:09, Jerry Malcolm wrote:
>>>>>> Just an FYI.... this server has been in production a little
>>>>>> over a year.  This TC load problem only started 2-3 months ago.
>>>>>> So 'something' changed to cause this problem to manifest
>>>>>> itself. It's been too long to try to correlate a specific
>>>>>> server change to the start of this problem. But I just wanted
>>>>>> to point out that this problem has NOT been around since day 1
>>>>>> on this server.
>>>>> Just in case, since this is a Windows server .. 2-3 months ago,
>>>>> an update of a virus scanner ? (of the intrusive kind which
>>>>> pre-checks each file that wants to be opened) Checking may be
>>>>> easy or not, depending on your access : disable the virus scanner
>>>>> just the time to start tomcat and check.
>>>>>
>>>>>> Jerry
>>>>>>
>>>>>> On 5/19/2019 5:03 PM, Jerry Malcolm wrote:
>>>>>>> Rainer,
>>>>>>>
>>>>>>> No change with the urandom parm.  I am attaching a portion of
>>>>>>> the Catalina log.  The first half shows between 8 and 15
>>>>>>> seconds to deploy each app for a single virtual host (there
>>>>>>> are no war files, the app is already exploded in the appbase
>>>>>>> dir).  I have quite a few virtual hosts with several apps
>>>>>>> each.  That initial server start took 21+ minutes. I then
>>>>>>> restarted the TC service and got the ~.5 sec start per webapp
>>>>>>> shown in the 2nd half of the log below.
>>>>>>>
>>>>>>> Can you refresh me on how to capture the stack dumps you
>>>>>>> suggested? It's been a while....
>>>>>>>
>>>>>>> Thx.
>>>>>>>
>>>>>>> Jerry
>>>>>>>
>>>>>>> First start of Tomcat after server reboot
>>>>>>> -------------------------------------------------------------------
>>> - --
>>> [C:\domains\<myDomain>.com\webapps\JSPWiki.war] has finished in
>>>>>>> [8,579] ms tory [C:\domains\<myDomain>.com\webapps\cis]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
>>>>>>> in [11,486] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\gallery] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\gallery] has finished in
>>>>>>> [9,204] ms tory [C:\domains\<myDomain>.com\webapps\gl]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\gl] has finished
>>>>>>> in [8,469] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\idmanager] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\idmanager] has finished in
>>>>>>> [8,689] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
>>>>>>> [6,907] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\malcolment] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\malcolment] has finished
>>>>>>> in [8,469] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
>>>>>>> [10,189] ms tory [C:\domains\<myDomain>.com\webapps\order]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
>>>>>>> finished in [8,501] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>>>>>> [14,209] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>>>>>> finished in [9,018] ms [C:\Tomcat
>>>>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] ptor [C:\Tomcat
>>>>>>> 9.0\conf\Catalina\myridetx.net\manager.xml] has finished in
>>>>>>> [62] ms
>>>>>>>
>>>>>>> Restart of Tomcat
>>>>>>> -------------------------------------------------------------------
>>> - --
>>> tory [C:\domains\<myDomain>.com\webapps\cis]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\cis] has finished
>>>>>>> in [594] ms tory [C:\domains\<myDomain>.com\webapps\gallery]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\gallery] has
>>>>>>> finished in [547] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\gl] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\gl] has finished in [562]
>>>>>>> ms tory [C:\domains\<myDomain>.com\webapps\idmanager]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\idmanager] has
>>>>>>> finished in [578] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\itemtrack] has finished in
>>>>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\malcolment]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\malcolment] has
>>>>>>> finished in [579] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\notebook] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\notebook] has finished in
>>>>>>> [547] ms tory [C:\domains\<myDomain>.com\webapps\order]
>>>>>>> irectory [C:\domains\<myDomain>.com\webapps\order] has
>>>>>>> finished in [531] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\payment] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\payment] has finished in
>>>>>>> [562] ms tory
>>>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] irectory
>>>>>>> [C:\domains\<myDomain>.com\webapps\projectmanager] has
>>>>>>> finished in [531] ms
>>>>>>>
>>>>>>> On 5/18/2019 3:18 PM, Rainer Jung wrote:
>>>>>>>> Most likely it hangs waiting for enough entropy for random
>>>>>>>> number generator seeding.
>>>>>>>>
>>>>>>>> Try whether the problem goes away if you add
>>>>>>>>
>>>>>>>> -Djava.security.egd=file:/dev/urandom
>>>>>>>>
>>>>>>>> to you process flags. If you are using older Java than Java
>>>>>>>> 8 (not possible for Tomcat 9 but just in case you also have
>>>>>>>> older software stacks running), then it would be
>>>>>>>>
>>>>>>>> -Djava.security.egd=file:/dev/./urandom
>>>>>>>>
>>>>>>>> The cryptic /./ is not a typo, you would need it for Java
>>>>>>>> before Java 8.
>>>>>>>>
>>>>>>>> If this doesn't help, then you should try capturing a few
>>>>>>>> stack dumps (thread dumps) during the long startup time.
>>>>>>>> People here can help interpret them.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Rainer
>>>>>>>>
>>>>>>>> Am 18.05.2019 um 21:55 schrieb Jerry Malcolm:
>>>>>>>>> This is a weird one.  It started a few months ago.   I
>>>>>>>>> have TC 9 running on Windows Server 16.  After I reboot
>>>>>>>>> the entire server, Tomcat takes forever on startup. It
>>>>>>>>> normally starts in about 30 seconds.  But after a server
>>>>>>>>> reboot it takes up to 15 minutes... chugging along at a
>>>>>>>>> snail's pace starting up all of the apps on all of the
>>>>>>>>> virtual hosts.  It always finally gets there with
>>>>>>>>> everything successfully running. Other servers on the
>>>>>>>>> same box (Apache, JAMES, ISC BIND, MySQL) don't have any
>>>>>>>>> problem starting up. CPU, Disk, Memory, etc. usages are
>>>>>>>>> barely showing on the performance graphs. There's nothing
>>>>>>>>> in the Catalina log or system.err other than showing a
>>>>>>>>> couple of minute gap in the time stamp between each app
>>>>>>>>> as it starts up. If I need to reboot again later, it
>>>>>>>>> boots again in about 30 sec as expected.
>>>>>>>>>
>>>>>>>>> First question... any ideas off the top of your head that
>>>>>>>>> might be causing this?
>>>>>>>>>
>>>>>>>>> Second question... is there any other logging I can turn
>>>>>>>>> on that might explain what TC is blocking on?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Jerry
>>>>>>>> ------------------------------------------------------------------
>>> - ---
>>> To unsubscribe, e-mail: [hidden email]
>>>>>>>> For additional commands, e-mail:
>>>>>>>> [hidden email]
>>>>>>>>
>>>>>>> -------------------------------------------------------------------
>>> - --
>>> To unsubscribe, e-mail: [hidden email]
>>>>>>> For additional commands, e-mail:
>>>>>>> [hidden email]
>>>>>>>
>>>>>> --------------------------------------------------------------------
>>> - -
>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>
>>>>>
>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>>
>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>> -----BEGIN PGP SIGNATURE-----
>>> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>>>
>>> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzkEWoACgkQHPApP6U8
>>> pFinLRAAttjrmJnG6lwB3pd7kT5IpBk5H/MwJdXvEX8/S/k6MLLgWqWmqQhgRQAZ
>>> UeVqJFJFDCrgXymxXNI2f7k3saVA+2Lxafjuft9WQGKs+mTSQR0s82VG6Iko5JzL
>>> dZxA5XD+SSU6PafTMQ1hmlVMMDeug+ONqMcUY2XFoZSV6TUk6KqjMiod74We7/b8
>>> 7PSqI4VWSS6fA4/ZoPJNRvGLXtYDu1obVX3YysMb4tSky/5OenJX6rz0mwMenY0z
>>> F2p2G2ObNlLKUq7DuAhBcLHB4lbkEv+20Zv6hqMGPlFB8rjiOpUpL86/gfdJY29z
>>> KGgWLAw1M4rw2n0GjVihcfYfjVBA931wWwkaXpa6cXCrcyjOcgCH+OlhIqjtoHkW
>>> w6mM9m99LqrWRwqqfiHDDxqywdqukfGANwi1QvaPgVkd0+FwGkinh5G3P9au3cLs
>>> JKdpGgEog1hJnjkkXWwROlozKiKjMD82V474E4mwjw7nqPI/YXXL5E8VTlhm11JB
>>> OWDZ8yNgwA//Aw7JbGdAUPqRSsSqrKIyGYr4voDd6nhs+83R/7z27IyIxgw/kzFU
>>> 6xRFOorztK5dOR5TqPZYB67qUL39++fpdayTYDs7f5Y8guajOsWgVTrvWo3bxCkg
>>> CxUE7V91tWJYJKlxgbFZyZp4vvSzabf4dq5fxKhPTkYH9pmHhCs=
>>> =z1un
>>> -----END PGP SIGNATURE-----
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]