Memory leak in the PKCS11 how to fix the problem

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)

Hello All,

 

I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area.

Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem?

(open this svg graph in browser only)

 

Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same.

 

Regards,

Raghav

Infrastructure engineer,

Cisco ISE.

 

 



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

malloc_6hrs.svg (401K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)

 

 

From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]>
Date: Saturday, 18 July 2020 at 6:20 PM
To: Tomcat Users List <[hidden email]>
Subject: Memory leak in the PKCS11 how to fix the problem

 

Hello All,

 

I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area.

Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem?

(open this svg graph in browser only)

 

Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same.

 

Regards,

Raghav

Infrastructure engineer,

Cisco ISE.

 

 



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

malloc_6hrs.svg (401K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)
The OS is RHEL => 7.6


From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]>
Date: Saturday, 18 July 2020 at 6:21 PM
To: Tomcat Users List <[hidden email]>
Subject: Memory leak in the PKCS11 how to fix the problem



From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]>
Date: Saturday, 18 July 2020 at 6:20 PM
To: Tomcat Users List <[hidden email]>
Subject: Memory leak in the PKCS11 how to fix the problem

Hello All,

I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area.
Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem?
(open this svg graph in browser only)

Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same.

Regards,
Raghav
Infrastructure engineer,
Cisco ISE.


Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)
Kindly reply for the below mail,
Expecting your feedback so that I can take it forward.

Thanks & Regards,
Raghav

On 18/07/20, 6:25 PM, "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> wrote:

    The OS is RHEL => 7.6


    From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]>
    Date: Saturday, 18 July 2020 at 6:21 PM
    To: Tomcat Users List <[hidden email]>
    Subject: Memory leak in the PKCS11 how to fix the problem



    From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]>
    Date: Saturday, 18 July 2020 at 6:20 PM
    To: Tomcat Users List <[hidden email]>
    Subject: Memory leak in the PKCS11 how to fix the problem

    Hello All,

    I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area.
    Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem?
    (open this svg graph in browser only)

    Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same.

    Regards,
    Raghav
    Infrastructure engineer,
    Cisco ISE.




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)
In reply to this post by Ragavendhiran Bhiman (rabhiman)
HI,

Kindly suggest what could be the issue below.

Thanks & Regards,
Raghav


    From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]>
    Date: Saturday, 18 July 2020 at 6:21 PM
    To: Tomcat Users List <[hidden email]>
    Subject: Memory leak in the PKCS11 how to fix the problem



    From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]>
    Date: Saturday, 18 July 2020 at 6:20 PM
    To: Tomcat Users List <[hidden email]>
    Subject: Memory leak in the PKCS11 how to fix the problem

    Hello All,

    I am seeing the memory leaks from tomcat apache in the following SSL path using PKCS11. Attached the flame graph of memory possible memory leaks in this area.
    Please check the attached flame graph of the memory trace. On simply a long run the memory keep on allocated in these back traces only causing the memory leak, and the polling of the async profiler for more than 6hours shows this clearly. Could you please help how to fix this problem?
    (open this svg graph in browser only)

    Note: If C_DestroyObject is not called because of finalizer accumulation is also tested by inducing the gc using the jmap command still could see the memory never gone down after the Full GC collection as well. Expecting your advice on the same.

    Regards,
    Raghav
    Infrastructure engineer,
    Cisco ISE.




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Christopher Schultz-2
In reply to this post by Ragavendhiran Bhiman (rabhiman)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ragavendhiran,

On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:

> The OS is RHEL => 7.6
>
>
> From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> Date:
> Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
> <[hidden email]> Subject: Memory leak in the PKCS11 how to
> fix the problem
>
>
>
> From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> Date:
> Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
> <[hidden email]> Subject: Memory leak in the PKCS11 how to
> fix the problem
>
> Hello All,
>
> I am seeing the memory leaks from tomcat apache in the following
> SSL path using PKCS11.
In what way are you using PKCS#11? Using a hardware crypto engine via
OpenSSL? Are you using the APR connector or are you using OpenSSL via
JSSE?

> Attached the flame graph of memory possible memory leaks in this
> area. Please check the attached flame graph of the memory trace.

Attachments are stripped from messages to this list. Can you post the
graph somewhere and include a link to a reply here?

> On simply a long run the memory keep on allocated in these back
> traces only causing the memory leak, and the polling of the async
> profiler for more than 6hours shows this clearly. Could you please
>  help how to fix this problem? (open this svg graph in browser
> only)> Note: If C_DestroyObject is not called because of finalizer
>  accumulation is also tested by inducing the gc using the jmap
> command still could see the memory never gone down after the Full
> GC collection as well. Expecting your advice on the same.

Are you suggesting that the solution will be to do something explicit
in a class finalizer? Or are you suggesting that some additional
native calls are required to clean-up.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
/kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
=yShM
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Martin Grigorov
In reply to this post by Ragavendhiran Bhiman (rabhiman)
Hi Ragavendhiran,

On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
<[hidden email]> wrote:

> Hello All,
>
>
>
> I am seeing the memory leaks from tomcat apache in the following SSL path
> using PKCS11. Attached the flame graph of memory possible memory leaks in
> this area.
>
> Please check the attached flame graph of the memory trace. On simply a
> long run the memory keep on allocated in these back traces only causing the
> memory leak, and the polling of the async profiler for more than 6hours
> shows this clearly. Could you please help how to fix this problem?
>
> (open this svg graph in browser only)
>
>
>
> Note: If C_DestroyObject is not called because of finalizer accumulation
> is also tested by inducing the gc using the jmap command still could see
> the memory never gone down after the Full GC collection as well. Expecting
> your advice on the same.
>

With AsyncProfiler '-e alloc' you can see what part of the code is
responsible for making most of the memory allocations, but it doesn't tell
you whether those objects leak or not. AsyncProfiler helps you to identify
the top allocations and if you manage to reduce them then you will reduce
the GC runs and the time they take.
To debug memory leaks you need to take a heap dump, e.g. with 'jmap
-dump:live,format=b,file=heap.bin <PID>' and analyze it. I'd recommend you
to use Eclipse MAT to do that.

Also in your flame graph I see that Netty is responsible for 49.04% of
the allocated objects and Tomcat for just 25.32%.


>
> Regards,
>
> Raghav
>
> Infrastructure engineer,
>
> Cisco ISE.
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)
Hello Martin,

Thanks for your reply
 
https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

Here is the link I have shared the flame graph.
Also kindly check my answers in red as well.

Thanks & Regards,
Raghav


On 22/07/20, 2:33 PM, "Martin Grigorov" <[hidden email]> wrote:

    Hi Ragavendhiran,

    On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
    <[hidden email]> wrote:

    > Hello All,
    >
    >
    >
    > I am seeing the memory leaks from tomcat apache in the following SSL path
    > using PKCS11. Attached the flame graph of memory possible memory leaks in
    > this area.
    >
    > Please check the attached flame graph of the memory trace. On simply a
    > long run the memory keep on allocated in these back traces only causing the
    > memory leak, and the polling of the async profiler for more than 6hours
    > shows this clearly. Could you please help how to fix this problem?
    >
    > (open this svg graph in browser only)
    >
    >
    >
    > Note: If C_DestroyObject is not called because of finalizer accumulation
    > is also tested by inducing the gc using the jmap command still could see
    > the memory never gone down after the Full GC collection as well. Expecting
    > your advice on the same.
    >

    With AsyncProfiler '-e alloc' you can see what part of the code is
    responsible for making most of the memory allocations, but it doesn't tell
    you whether those objects leak or not. AsyncProfiler helps you to identify
    the top allocations and if you manage to reduce them then you will reduce
    the GC runs and the time they take.
    To debug memory leaks you need to take a heap dump, e.g. with 'jmap
    -dump:live,format=b,file=heap.bin <PID>' and analyze it. I'd recommend you
    to use Eclipse MAT to do that.

<Raghav> "Yes with async profiler and just leaving the server without any sequence of action the memory started growing (RSS grows) when I profiled during that sequence there is no other memory allocation happening except this one.
That’s why I am suspecting this flow clearly. Samples are only through this flow only".

    Also in your flame graph I see that Netty is responsible for 49.04% of
    the allocated objects and Tomcat for just 25.32%.


    >
    > Regards,
    >
    > Raghav
    >
    > Infrastructure engineer,
    >
    > Cisco ISE.
    >
    >
    >
    >
    >
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: [hidden email]
    > For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)
In reply to this post by Christopher Schultz-2
Hi Chris,

Please see my answers inline.
Also shared the svg graph here.

https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

Kindly reply.


Thanks & Regards,
Raghav

On 20/07/20, 11:08 PM, "Christopher Schultz" <[hidden email]> wrote:

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA256

    Ragavendhiran,

    On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:
    > The OS is RHEL => 7.6
    >
    >
    > From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> Date:
    > Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
    > <[hidden email]> Subject: Memory leak in the PKCS11 how to
    > fix the problem
    >
    >
    >
    > From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> Date:
    > Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
    > <[hidden email]> Subject: Memory leak in the PKCS11 how to
    > fix the problem
    >
    > Hello All,
    >
    > I am seeing the memory leaks from tomcat apache in the following
    > SSL path using PKCS11.
    In what way are you using PKCS#11? Using a hardware crypto engine via
    OpenSSL? Are you using the APR connector or are you using OpenSSL via
    JSSE?
  <Raghav>  It is being used via the OpenSSL. I am not sure how to identify it is via the APR or hardware.

    > Attached the flame graph of memory possible memory leaks in this
    > area. Please check the attached flame graph of the memory trace.

    Attachments are stripped from messages to this list. Can you post the
    graph somewhere and include a link to a reply here?

<Raghav>Attached again using the dropbox.

    > On simply a long run the memory keep on allocated in these back
    > traces only causing the memory leak, and the polling of the async
    > profiler for more than 6hours shows this clearly. Could you please
    >  help how to fix this problem? (open this svg graph in browser
    > only)> Note: If C_DestroyObject is not called because of finalizer
    >  accumulation is also tested by inducing the gc using the jmap
    > command still could see the memory never gone down after the Full
    > GC collection as well. Expecting your advice on the same.

    Are you suggesting that the solution will be to do something explicit
    in a class finalizer? Or are you suggesting that some additional
    native calls are required to clean-up.

<Raghav> I called the GC via jmap -histo:live but still the weak references not cleaned up the RSS never reduced. This means that some where the memory being held in native that’s what my suspect. Kindly suggest from your side as well.

Thanks.


    - -chris
    -----BEGIN PGP SIGNATURE-----
    Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

    iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
    pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
    /kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
    uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
    iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
    U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
    rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
    53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
    vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
    SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
    nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
    fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
    =yShM
    -----END PGP SIGNATURE-----

    ---------------------------------------------------------------------
    To unsubscribe, e-mail: [hidden email]
    For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)
In reply to this post by Ragavendhiran Bhiman (rabhiman)
Sorry Martin,

My answers are inline.

Thanks & Regards,
Raghav

On 22/07/20, 7:20 PM, "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> wrote:

    Hello Martin,

    Thanks for your reply

    https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

    Here is the link I have shared the flame graph.
    Also kindly check my answers in red as well.

    Thanks & Regards,
    Raghav


    On 22/07/20, 2:33 PM, "Martin Grigorov" <[hidden email]> wrote:

        Hi Ragavendhiran,

        On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
        <[hidden email]> wrote:

        > Hello All,
        >
        >
        >
        > I am seeing the memory leaks from tomcat apache in the following SSL path
        > using PKCS11. Attached the flame graph of memory possible memory leaks in
        > this area.
        >
        > Please check the attached flame graph of the memory trace. On simply a
        > long run the memory keep on allocated in these back traces only causing the
        > memory leak, and the polling of the async profiler for more than 6hours
        > shows this clearly. Could you please help how to fix this problem?
        >
        > (open this svg graph in browser only)
        >
        >
        >
        > Note: If C_DestroyObject is not called because of finalizer accumulation
        > is also tested by inducing the gc using the jmap command still could see
        > the memory never gone down after the Full GC collection as well. Expecting
        > your advice on the same.
        >

        With AsyncProfiler '-e alloc' you can see what part of the code is
        responsible for making most of the memory allocations, but it doesn't tell
        you whether those objects leak or not. AsyncProfiler helps you to identify
        the top allocations and if you manage to reduce them then you will reduce
        the GC runs and the time they take.
        To debug memory leaks you need to take a heap dump, e.g. with 'jmap
        -dump:live,format=b,file=heap.bin <PID>' and analyze it. I'd recommend you
        to use Eclipse MAT to do that.

    <Raghav> "Yes with async profiler and just leaving the server without any sequence of action the memory started growing (RSS grows) when I profiled during that sequence there is no other memory allocation happening except this one.
    That’s why I am suspecting this flow clearly. Samples are only through this flow only".

        Also in your flame graph I see that Netty is responsible for 49.04% of
        the allocated objects and Tomcat for just 25.32%.


        >
        > Regards,
        >
        > Raghav
        >
        > Infrastructure engineer,
        >
        > Cisco ISE.
        >
        >
        >
        >
        >
        > ---------------------------------------------------------------------
        > To unsubscribe, e-mail: [hidden email]
        > For additional commands, e-mail: [hidden email]



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Ragavendhiran Bhiman (rabhiman)
In reply to this post by Ragavendhiran Bhiman (rabhiman)
The tomcat version is apache-tomcat-8.5.29
And RedHat Enterprise Linux 7.6

On 22/07/20, 7:28 PM, "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> wrote:

    Hi Chris,

    Please see my answers inline.
    Also shared the svg graph here.

    https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

    Kindly reply.


    Thanks & Regards,
    Raghav

    On 20/07/20, 11:08 PM, "Christopher Schultz" <[hidden email]> wrote:

        -----BEGIN PGP SIGNED MESSAGE-----
        Hash: SHA256

        Ragavendhiran,

        On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:
        > The OS is RHEL => 7.6
        >
        >
        > From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> Date:
        > Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
        > <[hidden email]> Subject: Memory leak in the PKCS11 how to
        > fix the problem
        >
        >
        >
        > From: "Ragavendhiran Bhiman (rabhiman)" <[hidden email]> Date:
        > Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
        > <[hidden email]> Subject: Memory leak in the PKCS11 how to
        > fix the problem
        >
        > Hello All,
        >
        > I am seeing the memory leaks from tomcat apache in the following
        > SSL path using PKCS11.
        In what way are you using PKCS#11? Using a hardware crypto engine via
        OpenSSL? Are you using the APR connector or are you using OpenSSL via
        JSSE?
      <Raghav>  It is being used via the OpenSSL. I am not sure how to identify it is via the APR or hardware.

        > Attached the flame graph of memory possible memory leaks in this
        > area. Please check the attached flame graph of the memory trace.

        Attachments are stripped from messages to this list. Can you post the
        graph somewhere and include a link to a reply here?

    <Raghav>Attached again using the dropbox.

        > On simply a long run the memory keep on allocated in these back
        > traces only causing the memory leak, and the polling of the async
        > profiler for more than 6hours shows this clearly. Could you please
        >  help how to fix this problem? (open this svg graph in browser
        > only)> Note: If C_DestroyObject is not called because of finalizer
        >  accumulation is also tested by inducing the gc using the jmap
        > command still could see the memory never gone down after the Full
        > GC collection as well. Expecting your advice on the same.

        Are you suggesting that the solution will be to do something explicit
        in a class finalizer? Or are you suggesting that some additional
        native calls are required to clean-up.

    <Raghav> I called the GC via jmap -histo:live but still the weak references not cleaned up the RSS never reduced. This means that some where the memory being held in native that’s what my suspect. Kindly suggest from your side as well.

    Thanks.


        - -chris
        -----BEGIN PGP SIGNATURE-----
        Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

        iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
        pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
        /kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
        uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
        iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
        U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
        rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
        53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
        vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
        SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
        nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
        fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
        =yShM
        -----END PGP SIGNATURE-----

        ---------------------------------------------------------------------
        To unsubscribe, e-mail: [hidden email]
        For additional commands, e-mail: [hidden email]


    ?B�KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB�?�?[��X��ܚX�K??K[XZ[?�?\�\��][��X��ܚX�P??�X�]?�\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[�?�??K[XZ[?�?\�\��Z?[????�X�]?�\?X�?K�ܙ�B�


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in the PKCS11 how to fix the problem

Martin Grigorov
In reply to this post by Ragavendhiran Bhiman (rabhiman)
On Wed, Jul 22, 2020, 16:58 Ragavendhiran Bhiman (rabhiman)
<[hidden email]> wrote:

> Sorry Martin,
>
> My answers are inline.
>

Take a heap dump and analyze it!


> Thanks & Regards,
> Raghav
>
> On 22/07/20, 7:20 PM, "Ragavendhiran Bhiman (rabhiman)"
> <[hidden email]> wrote:
>
>     Hello Martin,
>
>     Thanks for your reply
>
>
> https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0
>
>     Here is the link I have shared the flame graph.
>     Also kindly check my answers in red as well.
>
>     Thanks & Regards,
>     Raghav
>
>
>     On 22/07/20, 2:33 PM, "Martin Grigorov" <[hidden email]> wrote:
>
>         Hi Ragavendhiran,
>
>         On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
>         <[hidden email]> wrote:
>
>         > Hello All,
>         >
>         >
>         >
>         > I am seeing the memory leaks from tomcat apache in the following
> SSL path
>         > using PKCS11. Attached the flame graph of memory possible memory
> leaks in
>         > this area.
>         >
>         > Please check the attached flame graph of the memory trace. On
> simply a
>         > long run the memory keep on allocated in these back traces only
> causing the
>         > memory leak, and the polling of the async profiler for more than
> 6hours
>         > shows this clearly. Could you please help how to fix this
> problem?
>         >
>         > (open this svg graph in browser only)
>         >
>         >
>         >
>         > Note: If C_DestroyObject is not called because of finalizer
> accumulation
>         > is also tested by inducing the gc using the jmap command still
> could see
>         > the memory never gone down after the Full GC collection as well.
> Expecting
>         > your advice on the same.
>         >
>
>         With AsyncProfiler '-e alloc' you can see what part of the code is
>         responsible for making most of the memory allocations, but it
> doesn't tell
>         you whether those objects leak or not. AsyncProfiler helps you to
> identify
>         the top allocations and if you manage to reduce them then you will
> reduce
>         the GC runs and the time they take.
>         To debug memory leaks you need to take a heap dump, e.g. with 'jmap
>         -dump:live,format=b,file=heap.bin <PID>' and analyze it. I'd
> recommend you
>         to use Eclipse MAT to do that.
>
>     <Raghav> "Yes with async profiler and just leaving the server without
> any sequence of action the memory started growing (RSS grows) when I
> profiled during that sequence there is no other memory allocation happening
> except this one.
>     That’s why I am suspecting this flow clearly. Samples are only through
> this flow only".
>
>         Also in your flame graph I see that Netty is responsible for
> 49.04% of
>         the allocated objects and Tomcat for just 25.32%.
>
>
>         >
>         > Regards,
>         >
>         > Raghav
>         >
>         > Infrastructure engineer,
>         >
>         > Cisco ISE.
>         >
>         >
>         >
>         >
>         >
>         >
> ---------------------------------------------------------------------
>         > To unsubscribe, e-mail: [hidden email]
>         > For additional commands, e-mail: [hidden email]
>
>
>