Request for a tip with NoClassDefFoundError

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Request for a tip with NoClassDefFoundError

Michal Rysavy
Hello,

I would like to ask for any tips that can help us to solve the problem with
NoClassDefFoundError.

We have a specific problem with an application (Documentum) on Tomcat
(running as a container on OpenShift). Sometimes we have many
NoClassDefFoundError on classes that are loaded via network from the
backend application.
The biggest problem is that an error is unpredictable, it occurs only when
starting an application under high load and not always. Sometimes there are
few occurrences and sometimes there are thousands occurrences and sometimes
nothing (during the same load).
But we can't repeat the problem in a non-production environment, even with
the same or higher load of performance tests. More precisely, we had this
error on non-prod several times, but with negligible occurrence, randomly
and with different application configuration only that is not suitable for
production.

Class reported with NoClassDefFoundError is not always the same, but always
this class can be found in the appropriate jar file in the application
cache of received jars. When we had classloader verbose enabled, I think
that we saw that the class was loaded (but we must recheck this). More
precisely, at the same time the problematic class (stated as
NoClassDefFoundError) is the same, but after the next restart under high
load there is another problematic class.

We are running Tomcat 9.0.29 on OpenJDK Runtime Environment (build
1.8.0_222-b10) - with almost the default configuration.
We are planning a Tomcat & Java update but now we can't test it before
production, because we can't reproduce this error on the non-prod.

So I would like to ask if there are some tuning tips that we can try for a)
prevention of occurrence in production and b) ensuring occurence in
non-production

Many thanks for any tips.

Kind Regards,
Michal Rysavy

Some examples of occurence (other occurrences have another classes as
NoClassDefFoundError / ClassNotFoundException):

EXAMPLE 1/4
===========
com.emc.documentum.fs.rt.ServiceException:
com/fasterxml/jackson/databind/deser/std/FactoryBasedEnumDeserializer.
com.fasterxml.jackson.databind.deser.std.FactoryBasedEnumDeserializer
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:662)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
        at
com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
        at
com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
        at
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
        at
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
        at
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
        at
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
        at
com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
com/fasterxml/jackson/databind/deser/std/FactoryBasedEnumDeserializer
        at
com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createEnumDeserializer(BasicDeserializerFactory.java:1283)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:368)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at
com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:467)
        at
com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:460)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at
com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at
com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
        at
com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4179)
        at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
        at
com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
        at
cz.trask.dctm.csas.kld.elastic.common.EsConnector$1.readValue(EsConnector.java:133)
        at
com.mashape.unirest.http.HttpResponse.<init>(HttpResponse.java:101)
        at
com.mashape.unirest.http.HttpClientHelper.request(HttpClientHelper.java:139)
        at
com.mashape.unirest.request.BaseRequest.asObject(BaseRequest.java:80)
        at
cz.trask.dctm.elastic.impl.EsAbstractConnector.search(EsAbstractConnector.java:614)
        at
cz.trask.dctm.csas.kld.elastic.operation.EsSearchOperationService.searchDocumentsWS(EsSearchOperationService.java:64)
        at
cz.trask.dctm.csas.kld.elastic.util.EsUtil.esSearchDocumentsWS(EsUtil.java:158)
        at
cz.ness.documentum.cskld.ws.sbo.impl.CsRecordService.esSearchRecord(CsRecordService.java:1758)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
        at
com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
        at
com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
        at
com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
        at
com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
        at
com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
        at
com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
        at
com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
        at
com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
        at com.sun.proxy.$Proxy55.esSearchRecord(Unknown Source)
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:637)
        ... 43 more
Caused by: java.lang.ClassNotFoundException:
com.fasterxml.jackson.databind.deser.std.FactoryBasedEnumDeserializer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at
com.documentum.fc.client.impl.bof.classmgmt.URLClassLoaderEx.findClass(URLClassLoaderEx.java:49)
        at
com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.findClass(DelayedDelegationClassLoader.java:241)
        at
com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.findClass(AbstractTransformingClassLoader.java:121)
        at
com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.loadClass(DelayedDelegationClassLoader.java:147)
        at
com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.loadClass(AbstractTransformingClassLoader.java:68)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 81 more





EXAMPLE 2/4
===========
com.emc.documentum.fs.rt.ServiceException:
cz/trask/dctm/csas/kld/util/KafkaSpis
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.registerRecord(CsRecordServiceWebService.java:112)
        at sun.reflect.GeneratedMethodAccessor445.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
        at
com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
        at
com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
        at
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
        at
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
        at
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
        at
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
        at
com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
cz/trask/dctm/csas/kld/util/KafkaSpis
        at
cz.ness.documentum.cskld.ws.sbo.impl.CsRecordService.registerRecord(CsRecordService.java:556)
        at sun.reflect.GeneratedMethodAccessor446.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
        at
com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
        at
com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
        at
com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
        at
com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
        at
com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
        at
com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
        at
com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
        at
com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
        at com.sun.proxy.$Proxy55.registerRecord(Unknown Source)
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.registerRecord(CsRecordServiceWebService.java:87)





EXAMPLE 3/4
===========
com.emc.documentum.fs.rt.ServiceException:
org/apache/http/client/methods/HttpGet.
org.apache.http.client.methods.HttpGet
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:662)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
        at
com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
        at
com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
        at
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
        at
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
        at
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
        at
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
        at
com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
org/apache/http/client/methods/HttpGet
        at
com.mashape.unirest.request.BaseRequest.asObject(BaseRequest.java:80)
        at
cz.trask.dctm.elastic.impl.EsAbstractConnector.search(EsAbstractConnector.java:614)
        at
cz.trask.dctm.csas.kld.elastic.operation.EsSearchOperationService.searchDocumentsWS(EsSearchOperationService.java:64)
        at
cz.trask.dctm.csas.kld.elastic.util.EsUtil.esSearchDocumentsWS(EsUtil.java:158)
        at
cz.ness.documentum.cskld.ws.sbo.impl.CsRecordService.esSearchRecord(CsRecordService.java:1758)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
        at
com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
        at
com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
        at
com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
        at
com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
        at
com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
        at
com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
        at
com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
        at
com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
        at com.sun.proxy.$Proxy55.esSearchRecord(Unknown Source)
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:637)
        ... 43 more
Caused by: java.lang.ClassNotFoundException:
org.apache.http.client.methods.HttpGet
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at
com.documentum.fc.client.impl.bof.classmgmt.URLClassLoaderEx.findClass(URLClassLoaderEx.java:49)
        at
com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.findClass(DelayedDelegationClassLoader.java:241)
        at
com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.findClass(AbstractTransformingClassLoader.java:121)
        at
com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.loadClass(DelayedDelegationClassLoader.java:147)
        at
com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.loadClass(AbstractTransformingClassLoader.java:68)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 63 more




EXAMPLE 4/4
===========
com.emc.documentum.fs.rt.ServiceException:
com/mashape/unirest/http/utils/Base64Coder
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.searchRecord(CsRecordServiceWebService.java:442)
        at sun.reflect.GeneratedMethodAccessor733.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
        at
com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
        at
com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
        at
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
        at
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
        at
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
        at
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
        at
com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
        at
com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
        at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
        at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
com/mashape/unirest/http/utils/Base64Coder
        at
com.mashape.unirest.request.HttpRequest.basicAuth(HttpRequest.java:72)
        at
com.mashape.unirest.request.HttpRequestWithBody.basicAuth(HttpRequestWithBody.java:73)
        at
cz.trask.dctm.elastic.impl.EsAbstractConnector.search(EsAbstractConnector.java:611)
        at
cz.trask.dctm.csas.kld.elastic.operation.EsSearchOperationService.searchDocumentsWS(EsSearchOperationService.java:64)
        at
cz.trask.dctm.csas.kld.elastic.util.EsUtil.esSearchDocumentsWS(EsUtil.java:165)
        at
cz.ness.documentum.cskld.ws.sbo.impl.CsRecordService.searchRecord(CsRecordService.java:1292)
        at sun.reflect.GeneratedMethodAccessor734.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
        at
com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
        at
com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
        at
com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
        at
com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
        at
com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
        at
com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
        at
com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
        at
com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
        at com.sun.proxy.$Proxy55.searchRecord(Unknown Source)
        at
com.ness.documentum.csas.ws.v01.recordservice.CsRecordServiceWebService.searchRecord(CsRecordServiceWebService.java:417)
        ... 42 more
Reply | Threaded
Open this post in threaded view
|

Re: Request for a tip with NoClassDefFoundError

Paul Carter-Brown-2
NoClassDefFoundError means that the class in question threw an exception
during its static initialisation phase. Then later on whenever some code
tries to access the class statically or create an instance the JVM will
throw a NoClassDefFoundError. This is not the same as a class not found
which relates to classloading issues.

To debug a NoClassDefFoundError, look at the logs when the class was first
used and see if it threw an error initialising. Also look at the source of
the class and see if any static blocks or static variables could fail e.g.:

public class MyClass {
private static String someString = doSomething();

private static String doSomething() {
  // access some server . If network is down an exception would be thrown
  throw new RuntimeException();
}
}

The above code would cause a NoClassDefFoundError whenever someone tries to
use MyClass.

Paul


On Wed, Sep 23, 2020 at 10:00 AM Michal Rysavy <[hidden email]> wrote:

> Hello,
>
> I would like to ask for any tips that can help us to solve the problem with
> NoClassDefFoundError.
>
> We have a specific problem with an application (Documentum) on Tomcat
> (running as a container on OpenShift). Sometimes we have many
> NoClassDefFoundError on classes that are loaded via network from the
> backend application.
> The biggest problem is that an error is unpredictable, it occurs only when
> starting an application under high load and not always. Sometimes there are
> few occurrences and sometimes there are thousands occurrences and sometimes
> nothing (during the same load).
> But we can't repeat the problem in a non-production environment, even with
> the same or higher load of performance tests. More precisely, we had this
> error on non-prod several times, but with negligible occurrence, randomly
> and with different application configuration only that is not suitable for
> production.
>
> Class reported with NoClassDefFoundError is not always the same, but always
> this class can be found in the appropriate jar file in the application
> cache of received jars. When we had classloader verbose enabled, I think
> that we saw that the class was loaded (but we must recheck this). More
> precisely, at the same time the problematic class (stated as
> NoClassDefFoundError) is the same, but after the next restart under high
> load there is another problematic class.
>
> We are running Tomcat 9.0.29 on OpenJDK Runtime Environment (build
> 1.8.0_222-b10) - with almost the default configuration.
> We are planning a Tomcat & Java update but now we can't test it before
> production, because we can't reproduce this error on the non-prod.
>
> So I would like to ask if there are some tuning tips that we can try for a)
> prevention of occurrence in production and b) ensuring occurence in
> non-production
>
> Many thanks for any tips.
>
> Kind Regards,
> Michal Rysavy
>
> Some examples of occurence (other occurrences have another classes as
> NoClassDefFoundError / ClassNotFoundException):
>
> EXAMPLE 1/4
> ===========
> com.emc.documentum.fs.rt.ServiceException:
> com/fasterxml/jackson/databind/deser/std/FactoryBasedEnumDeserializer.
> com.fasterxml.jackson.databind.deser.std.FactoryBasedEnumDeserializer
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:662)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
>         at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
>         at
>
> com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
>         at
>
> com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
>         at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
>         at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
>         at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
>         at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
>         at
> com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
>         at
>
> com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
>         at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
>         at
>
> com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
>         at
> com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>         at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
>         at
>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
>         at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
>         at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>         at
>
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
>         at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
>         at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
>         at
>
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>         at
>
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
>         at
> org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
>         at
> org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> com/fasterxml/jackson/databind/deser/std/FactoryBasedEnumDeserializer
>         at
>
> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createEnumDeserializer(BasicDeserializerFactory.java:1283)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:368)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
>         at
>
> com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:467)
>         at
>
> com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:460)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
>         at
>
> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
>         at
>
> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
>         at
>
> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4179)
>         at
>
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
>         at
>
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2992)
>         at
>
> cz.trask.dctm.csas.kld.elastic.common.EsConnector$1.readValue(EsConnector.java:133)
>         at
> com.mashape.unirest.http.HttpResponse.<init>(HttpResponse.java:101)
>         at
>
> com.mashape.unirest.http.HttpClientHelper.request(HttpClientHelper.java:139)
>         at
> com.mashape.unirest.request.BaseRequest.asObject(BaseRequest.java:80)
>         at
>
> cz.trask.dctm.elastic.impl.EsAbstractConnector.search(EsAbstractConnector.java:614)
>         at
>
> cz.trask.dctm.csas.kld.elastic.operation.EsSearchOperationService.searchDocumentsWS(EsSearchOperationService.java:64)
>         at
>
> cz.trask.dctm.csas.kld.elastic.util.EsUtil.esSearchDocumentsWS(EsUtil.java:158)
>         at
> cz.ness.documentum.cskld.ws
> .sbo.impl.CsRecordService.esSearchRecord(CsRecordService.java:1758)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
>         at
>
> com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
>         at
>
> com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
>         at
>
> com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
>         at
>
> com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
>         at
>
> com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
>         at
>
> com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
>         at com.sun.proxy.$Proxy55.esSearchRecord(Unknown Source)
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:637)
>         ... 43 more
> Caused by: java.lang.ClassNotFoundException:
> com.fasterxml.jackson.databind.deser.std.FactoryBasedEnumDeserializer
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.URLClassLoaderEx.findClass(URLClassLoaderEx.java:49)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.findClass(DelayedDelegationClassLoader.java:241)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.findClass(AbstractTransformingClassLoader.java:121)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.loadClass(DelayedDelegationClassLoader.java:147)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.loadClass(AbstractTransformingClassLoader.java:68)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 81 more
>
>
>
>
>
> EXAMPLE 2/4
> ===========
> com.emc.documentum.fs.rt.ServiceException:
> cz/trask/dctm/csas/kld/util/KafkaSpis
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.registerRecord(CsRecordServiceWebService.java:112)
>         at sun.reflect.GeneratedMethodAccessor445.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
>         at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
>         at
>
> com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
>         at
>
> com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
>         at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
>         at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
>         at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
>         at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
>         at
> com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
>         at
>
> com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
>         at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
>         at
>
> com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
>         at
> com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>         at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
>         at
>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
>         at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
>         at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>         at
>
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
>         at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
>         at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
>         at
>
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>         at
>
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
>         at
> org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
>         at
> org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> cz/trask/dctm/csas/kld/util/KafkaSpis
>         at
> cz.ness.documentum.cskld.ws
> .sbo.impl.CsRecordService.registerRecord(CsRecordService.java:556)
>         at sun.reflect.GeneratedMethodAccessor446.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
>         at
>
> com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
>         at
>
> com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
>         at
>
> com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
>         at
>
> com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
>         at
>
> com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
>         at
>
> com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
>         at com.sun.proxy.$Proxy55.registerRecord(Unknown Source)
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.registerRecord(CsRecordServiceWebService.java:87)
>
>
>
>
>
> EXAMPLE 3/4
> ===========
> com.emc.documentum.fs.rt.ServiceException:
> org/apache/http/client/methods/HttpGet.
> org.apache.http.client.methods.HttpGet
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:662)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
>         at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
>         at
>
> com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
>         at
>
> com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
>         at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
>         at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
>         at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
>         at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
>         at
> com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
>         at
>
> com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
>         at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
>         at
>
> com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
>         at
> com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>         at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
>         at
>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
>         at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
>         at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>         at
>
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
>         at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
>         at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
>         at
>
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>         at
>
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
>         at
> org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
>         at
> org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/http/client/methods/HttpGet
>         at
> com.mashape.unirest.request.BaseRequest.asObject(BaseRequest.java:80)
>         at
>
> cz.trask.dctm.elastic.impl.EsAbstractConnector.search(EsAbstractConnector.java:614)
>         at
>
> cz.trask.dctm.csas.kld.elastic.operation.EsSearchOperationService.searchDocumentsWS(EsSearchOperationService.java:64)
>         at
>
> cz.trask.dctm.csas.kld.elastic.util.EsUtil.esSearchDocumentsWS(EsUtil.java:158)
>         at
> cz.ness.documentum.cskld.ws
> .sbo.impl.CsRecordService.esSearchRecord(CsRecordService.java:1758)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
>         at
>
> com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
>         at
>
> com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
>         at
>
> com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
>         at
>
> com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
>         at
>
> com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
>         at
>
> com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
>         at com.sun.proxy.$Proxy55.esSearchRecord(Unknown Source)
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.esSearchRecord(CsRecordServiceWebService.java:637)
>         ... 43 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.http.client.methods.HttpGet
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.URLClassLoaderEx.findClass(URLClassLoaderEx.java:49)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.findClass(DelayedDelegationClassLoader.java:241)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.findClass(AbstractTransformingClassLoader.java:121)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.loadClass(DelayedDelegationClassLoader.java:147)
>         at
>
> com.documentum.fc.client.impl.bof.classmgmt.AbstractTransformingClassLoader.loadClass(AbstractTransformingClassLoader.java:68)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>         ... 63 more
>
>
>
>
> EXAMPLE 4/4
> ===========
> com.emc.documentum.fs.rt.ServiceException:
> com/mashape/unirest/http/utils/Base64Coder
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.searchRecord(CsRecordServiceWebService.java:442)
>         at sun.reflect.GeneratedMethodAccessor733.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:246)
>         at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:146)
>         at
>
> com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:257)
>         at
>
> com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:93)
>         at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)
>         at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)
>         at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)
>         at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)
>         at
> com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:243)
>         at
>
> com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:471)
>         at
> com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:244)
>         at
>
> com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:135)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:129)
>         at
>
> com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:160)
>         at
> com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:75)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
>         at
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>         at
>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>         at
>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
>         at
>
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
>         at
>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
>         at
>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>         at
>
> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
>         at
>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
>         at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
>         at
>
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>         at
>
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
>         at
> org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591)
>         at
> org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:49)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> com/mashape/unirest/http/utils/Base64Coder
>         at
> com.mashape.unirest.request.HttpRequest.basicAuth(HttpRequest.java:72)
>         at
>
> com.mashape.unirest.request.HttpRequestWithBody.basicAuth(HttpRequestWithBody.java:73)
>         at
>
> cz.trask.dctm.elastic.impl.EsAbstractConnector.search(EsAbstractConnector.java:611)
>         at
>
> cz.trask.dctm.csas.kld.elastic.operation.EsSearchOperationService.searchDocumentsWS(EsSearchOperationService.java:64)
>         at
>
> cz.trask.dctm.csas.kld.elastic.util.EsUtil.esSearchDocumentsWS(EsUtil.java:165)
>         at
> cz.ness.documentum.cskld.ws
> .sbo.impl.CsRecordService.searchRecord(CsRecordService.java:1292)
>         at sun.reflect.GeneratedMethodAccessor734.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ReflectionServiceInvoker.invoke(ReflectionServiceInvoker.java:40)
>         at
>
> com.emc.documentum.fs.rt.impl.tx.TxInvocationHandler.invoke(TxInvocationHandler.java:29)
>         at
>
> com.emc.documentum.fs.rt.context.impl.SoapServiceInvocationHandler.invoke(SoapServiceInvocationHandler.java:50)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ContextCacheCleanupHandler.invoke(ContextCacheCleanupHandler.java:28)
>         at
>
> com.emc.documentum.fs.rt.context.impl.ServiceContextCleanupHandler.invoke(ServiceContextCleanupHandler.java:31)
>         at
>
> com.emc.documentum.fs.rt.context.impl.HandlerChainExceptionInvocationHandler.invoke(HandlerChainExceptionInvocationHandler.java:35)
>         at
>
> com.emc.documentum.fs.rt.context.impl.UcfServiceInvocationHandler.invoke(UcfServiceInvocationHandler.java:53)
>         at
>
> com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
>         at
>
> com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
>         at com.sun.proxy.$Proxy55.searchRecord(Unknown Source)
>         at
> com.ness.documentum.csas.ws
> .v01.recordservice.CsRecordServiceWebService.searchRecord(CsRecordServiceWebService.java:417)
>         ... 42 more
>