Aspose.Slides for Java: Thread Deadlock in Production for Presentation Files

ankit11088 · August 31, 2021, 5:14pm

We are facing an outage on production due to Thread blockages for converting presentation files to html format.

We don’t know what is the cause of this issue but after a certain conversions, this issue is happening on AWS EC2 instances and the instances are no longer processing any traffic due to every thread blocked in here.

I wanted to attach a thread dump but it doesn’t allow me to upload one.

We think we are seeing a sort of infinite loop here. Please see the traces.

http-nio-8087-exec-1

PRIORITY :5

THREAD ID :0X00007FE1D40D6000

NATIVE ID :0X54

NATIVE ID (DECIMAL) :84

STATE :BLOCKED

stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at com.aspose.slides.ac3.do(Unknown Source)

waiting to lock <0x00000006055857f8> (a java.lang.Object)
at com.aspose.slides.ac3.do(Unknown Source)
at com.aspose.slides.lk.do(Unknown Source)
at com.aspose.slides.ac3.if(Unknown Source)
at com.aspose.slides.Presentation.do(Unknown Source)
at com.aspose.slides.Presentation.save(Unknown Source)
at com.gce.lms.document.extractor.PPTContentExtractor.extract(PPTContentExtractor.java:52)
at com.gce.lms.document.extractor.PPTContentExtractor.extract(PPTContentExtractor.java:20)
at com.gce.lms.document.service.DocumentConversionService.convertDocument(DocumentConversionService.java:104)
at com.gce.lms.document.service.DocumentConversionService$$FastClassBySpringCGLIB$$9ab52226.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:100)
at com.amazonaws.xray.spring.aop.XRayInterceptorUtils.conditionalProceed(XRayInterceptorUtils.java:15)
at com.gce.lms.common.awsxray.aop.XRayInspector.processXRayTrace(XRayInspector.java:41)
at com.amazonaws.xray.spring.aop.BaseAbstractXRayInterceptor.traceAroundMethods(BaseAbstractXRayInterceptor.java:49)
at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/DelegatingMethodAccessorImpl.java:43**)**
at java.lang.reflect.Method.invoke(java.base@11.0.3/Method.java:566**)**
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.gce.lms.document.service.DocumentConversionService$$EnhancerBySpringCGLIB$$b8250240.convertDocument()
at com.gce.lms.document.controller.DocumentServiceController.runConversion(DocumentServiceController.java:89)
at jdk.internal.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.3/DelegatingMethodAccessorImpl.java:43**)**
at java.lang.reflect.Method.invoke(java.base@11.0.3/Method.java:566**)**
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at brave.servlet.TracingFilter.doFilter(TracingFilter.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.gce.lms.document.security.DocumentRequestFilter.doFilterInternal(DocumentRequestFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.gce.lms.common.filter.Slf4jMDCFilter.doFilterInternal(Slf4jMDCFilter.java:36)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.gce.lms.common.filter.GraphQLFilter.doFilterInternal(GraphQLFilter.java:65)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at brave.servlet.TracingFilter.doFilter(TracingFilter.java:87)
at org.springframework.cloud.sleuth.instrument.web.LazyTracingFilter.doFilter(TraceWebServletAutoConfiguration.java:141)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.amazonaws.xray.javax.servlet.AWSXRayServletFilter.doFilter(AWSXRayServletFilter.java:153)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
locked <0x00000006167722e0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.3/ThreadPoolExecutor.java:1128**)**
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.3/ThreadPoolExecutor.java:628**)**
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(java.base@11.0.3/Thread.java:834**)**

Such as this, there are 30 threads waiting for an object lock.

Why would this happen?

this thread is blocking all the other 29 threads

at java.util.AbstractMap.get(java.base@11.0.3/AbstractMap.java:176) at com.aspose.slides.internal.ey.long.this(Unknown Source)

Looks like bug in aspose lib. AbstractMap is not thread safe
Hashmap can get stuck in infinite loop

Please let me know what you guys think of it.
It is critical for us.Screenshot 2021-08-31 at 10.21.25 PM.jpg (287.7 KB)
Screenshot 2021-08-31 at 10.21.46 PM.png (211.4 KB)

andrey.potapov · August 31, 2021, 7:25pm

@ankit11088,
Thank you for posting the query. Please check the problem using the latest version of Aspose.Slides. If the issue persists, please share the following:

presentation files
code example reproducing the deadlock

You can compress any files into a ZIP archive and upload it or share a link to the files in file storage.

ankit11088 · September 1, 2021, 4:04am

Attaching the thread dump zip file.threadDump.tdump-254-62-(8-21).zip (10.9 KB)

We are not sure about which files result in this.
There are a lot of files processed and slowly the threads starts locking up / blocked.
Eventually, the instance stops processing or taking any new connections as the thread pool is exhausted. All threads are blocked.

Please check the thread dump and let me know what you think is going on there.

Any help is appreciated.
Thank you.

andrey.potapov · September 1, 2021, 8:31am

@ankit11088,
You should check your application code for compliance with the following article:
Multithreading in Aspose.Slides

ankit11088 · September 1, 2021, 8:42am

Thanks for the reply Andrey.

Can you go through this code snippet? I think it is in compliance with the MultiThreading requirements.

    public ExtractedContent extract(ExtractedContent content) throws IOException {
        CopyStream copyStream = new CopyStream();
        // Instantiate a Presentation object that represents a presentation file
        Presentation pres = new Presentation(new ByteArrayInputStream(content.getBytes()));

        try {
            // setting html options
            HtmlOptions options = new HtmlOptions();
            options.setHtmlFormatter(
                    HtmlFormatter.createCustomFormatter(new CustomFormattingController()));

            INotesCommentsLayoutingOptions inotesOpts = options.getNotesCommentsLayouting();
            inotesOpts.setNotesPosition(NotesPositions.BottomFull);

            // save the slides to output stream
            pres.save(copyStream, SaveFormat.Html, options);

            return ExtractedContent.builder().contentLength(copyStream.size())
                .inputStream(copyStream.toInputStream()).build();
        } finally {
            pres.dispose();
            copyStream.close();
        }
    }

The Presentation object is created in the method and disposed as well in same method.
Let me know if you see any issues in this.

Thanks.

andrey.potapov · September 1, 2021, 9:47am

@ankit11088,
This code snippet does not use multithreading. Unfortunately, I cannot use it to investigate this case because the code snippet contains unknown classes (ExtractedContent, CopyStream, CustomFormattingController). The following code example works fine on my side:

Presentation pres = new Presentation(inputStream);

try {
    HtmlOptions options = new HtmlOptions();

    INotesCommentsLayoutingOptions inotesOpts = options.getNotesCommentsLayouting();
    inotesOpts.setNotesPosition(NotesPositions.BottomFull);

    pres.save(outputStream, SaveFormat.Html, options);
} finally {
    pres.dispose();
}

You should isolate the problem and share presentation samples and a comprehensive code example to reproduce the problem on our side.