Aspose components use the .NET MemoryStream

tahir.manzoor · June 23, 2016, 10:04am

Hi Brian,

Thanks for your patience. It is to inform you that our product team has completed the work on issue (WORDSNET-13730) and has come to a conclusion that this issue and the undesired behavior you're observing is actually not a bug in Aspose.Words. So, we are closing this issue as 'Not a Bug'.

Brian THOMAS:

You'll see the app builds to target x86 rather than x64 or Any CPU. This is to reflect the fact that the customers reporting this issue are still using 32-bit machines, and we cannot insist they move to x64.

Reason is that 32 bit computer has limit 2GB memory for process. Please check following link.

Fundamentals of Garbage Collection

You are searching for the border of the “OutOfMemoryException” by creating “stream” objects and when rest of the available memory is not enough for creating document object you get the exception. Where and how locate huge volumes of the application data is the architecture questions of the application level, Aspose.Words cannot control it.

Brian THOMAS:

But it only actually returns one of the Lists, so the Garbage Collector should free up the other List.

Correct clean up required to call “Dispose” method for the streams and all resources will be collected by GC when CLR choose the right time for it.

Brian_THOMAS · June 23, 2016, 10:54am

Hi Tahir,

But if it isn't a bug what is it? I don't accept that it's not a bug. Please reopen the case.

I reported this behaviour against Aspose.Words and Aspose.Email - it's the same issue in both places.

I accept it was MS not you that wrote the MemoryStream class that throws the exception. But you chose to implement your classes using it. MemoryStream's propensity to throw the OutOfMemory exception is well known (just google). So the aspose implementation is in error because it deliberately uses a faulty .NET class.

Bear with me here:

In the source code for the test app I supplied, I allocate 2GB in 100KB chunks.
Then I allow the GC to reclaim half of it. So there is now 1GB available - it's just that it's very fragmented - the largest contiguous chunk is 100KB.
The problem is that aspose class implementations use .NET MemoryStream which expects the backing byte array to be contiguous.
In this example, Aspose.Words calls MemoryStream to allocate a chunk larger than 100KB and MemoryStream throws an exception because no such free chunk exists.
The fix, I would suggest, is to replace MemoryStream with a Stream that does not rely on a contiguous backing byte array. I already suggested the class I am using - several others are available.

The sample code I supplied is necessarily contrived - I cannot supply the actual code the customer is using - they have 32-bit Windows with about half a dozen large apps running concurrently. They intermittently see the OutOfMemory exception thrown by MemoryStream inside aspose. The supplied code allows that defect to be reproduced at will.

Please respond - it is very important to us that this issue is resolved.

tahir.manzoor · June 24, 2016, 10:45am

Hi Brian,

Thanks for sharing the detail. If you just read a text file having 100kb file size using File.ReadAllText method in your application, the System.OutOfMemoryException is throw.

However, we have logged the shared detail in our issue tracking system for further investigation. We will inform you via this forum thread once there is any update available on this issue.

List<MemoryStream> streams = ConsumeMemory(chunks);
File.ReadAllText(@"c:\temp\input.txt");

tahir.manzoor · June 24, 2016, 10:59am

Hi Brian,

Further to my last post, we are getting Aspose.Words.FileCorruptedException exception while using following code snippet. Could you please execute the application again and share the exception message and stack here for our reference?

List<MemoryStream> streams = ConsumeMemory(chunks);

using (FileStream fs = new FileStream(@"c:\temp\in.txt", FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
    string raw = "This is just a short string";
    byte[] buff = Encoding.UTF8.GetBytes(raw);

    fs.Write(buff, 0, buff.Length);
    fs.Position = 0;

    Aspose.Words.Document doc = new Aspose.Words.Document(fs);
    Console.WriteLine("Success!");
}

Brian_THOMAS · June 24, 2016, 11:45am

I pasted the sample snippet you provided into the existing sample code, and then built and stepped the console app with an empty command line inside VisualStudio.

As expected, ConsumeMemory threw this First Chance exception:

 System.OutOfMemoryException occurred
  HResult=-2147024882
  Message=Exception of type 'System.OutOfMemoryException' was thrown.
  Source=mscorlib
  StackTrace:
       at System.IO.MemoryStream..ctor(Int32 capacity)
  InnerException:

I carried on stepping from there with [F10], and ConsumeMemory displayed this message:

ConsumeMemory requested 20000 100KB chunks. Allocation failure occurred when idx = 17201

I carried on stepping the code, and the line that constructs the Aspose.Words.Document threw this exception:

System.OutOfMemoryException occurred
  HResult=-2147024882
  Message=Exception of type 'System.OutOfMemoryException' was thrown.
  Source=Aspose.Words
  StackTrace:
       at     .                   ..ctor(Int32  )
  InnerException:

I carried on stepping and got this exception:

System.TypeInitializationException occurred
  HResult=-2146233036
  Message=The type initializer for '    ' threw an exception.
  Source=Aspose.Words
  TypeName=    
  StackTrace:
       at     . (Int32  )
       at   ..ctor(Int32  )
  InnerException: System.OutOfMemoryException
       HResult=-2147024882
       Message=Exception of type 'System.OutOfMemoryException' was thrown.
       Source=Aspose.Words
       StackTrace:
            at     .                   ..ctor(Int32  )
            at     ..cctor()
       InnerException:

I carried on stepping and got this exception:

System.TypeInitializationException occurred
  HResult=-2146233036
  Message=The type initializer for 'Aspose.Words.Properties.BuiltInDocumentProperties' threw an exception.
  Source=Aspose.Words
  TypeName=Aspose.Words.Properties.BuiltInDocumentProperties
  StackTrace:
       at Aspose.Words.Properties.BuiltInDocumentProperties..ctor()
       at Aspose.Words.Document..ctor(Boolean  )
  InnerException: System.TypeInitializationException
       HResult=-2146233036
       Message=The type initializer for '    ' threw an exception.
       Source=Aspose.Words
       TypeName=    
       StackTrace:
            at     . (Int32  )
            at   ..ctor(Int32  )
            at   ..ctor()
            at   ..ctor(Boolean  )
            at Aspose.Words.Properties.BuiltInDocumentProperties..cctor()
       InnerException: System.OutOfMemoryException
            HResult=-2147024882
            Message=Exception of type 'System.OutOfMemoryException' was thrown.
            Source=Aspose.Words
            StackTrace:
                 at     .                   ..ctor(Int32  )
                 at     ..cctor()
            InnerException:

When I next pressed [F10] to step the code, execution jumped to the catch block in Program.cs and it dumped those final exception details at the console.

I repeated the debugging steps having put the value 17500 on the command line. This time I got a different exception:

System.OutOfMemoryException occurred
  HResult=-2147024882
  Message=Exception of type 'System.OutOfMemoryException' was thrown.
  Source=mscorlib
  StackTrace:
       at System.IO.MemoryStream.set_Capacity(Int32 value)
  InnerException:

and when I stepped the code I got this:

    occurred
  HResult=-2146233088
  Message=Cannot extract
  Source=Aspose.Words
  StackTrace:
       at     . (String  , Stream  , String  )
  InnerException: System.OutOfMemoryException
       HResult=-2147024882
       Message=Exception of type 'System.OutOfMemoryException' was thrown.
       Source=mscorlib
       StackTrace:
            at System.IO.MemoryStream.set_Capacity(Int32 value)
            at System.IO.MemoryStream.EnsureCapacity(Int32 value)
            at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
            at     .  (Stream  )
            at     . (String  , Stream  , String  )
       InnerException:

This shows the difference between operating when there is no large available memory chunk at all and when there is a very small amount available.

tahir.manzoor · June 27, 2016, 3:14am

Hi Brian,

Thanks for sharing the detail. We will inform you via this forum thread once there is any update available on this issue. Thanks for your patience.

Brian_THOMAS · August 16, 2016, 4:35am

This case hasn't been updated for a while. It's still a major issue for us. One of our customers reported this error this morning. This isn't a new issue - it's a duplicate of behaviour logged with you already:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at #=qR3ogK9Uz$6IS6YavueREB6elWvd5mwknmaeSVlW9dMEggA1rHMCPBbD_0HAgbRvT.#=qJCkjlmB2nq_NIb_OOwBT6w==(Stream #=q1xBxpPWH8GVxk2Kp_4s4Pw==, Stream #=qwaaRHGhiYIDnpTAu0ipcDA==)
   at #=qsKuflGDN3fZCLOr01hIIocjFaCzdsuS1JqDDepouZ58gd4dvXK3NvKRllaatJwlP.#=q7TT3NPuyu6rlytXEPOOq$g==(MemoryStream #=q1xBxpPWH8GVxk2Kp_4s4Pw==, Boolean #=qy5wLr18XuMx6BZnflvYVwA==, Int32& #=qP_kG$pP_dcCSMMf9ujc5kw==)
   at #=qsKuflGDN3fZCLOr01hIIocjFaCzdsuS1JqDDepouZ58gd4dvXK3NvKRllaatJwlP.#=q7TT3NPuyu6rlytXEPOOq$g==(MemoryStream #=q1xBxpPWH8GVxk2Kp_4s4Pw==, Boolean #=qy5wLr18XuMx6BZnflvYVwA==)
   at #=qsKuflGDN3fZCLOr01hIIocjFaCzdsuS1JqDDepouZ58gd4dvXK3NvKRllaatJwlP.#=qVdHu6ElWQwXpMNTAaTNw8Q==(#=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qTLRAluB4Zj3MOKAZ1qKuRQ==, #=qbPfGgcQZm7XmxUdKwn78NGPKAThoaikPx9HzVS6JcQPEXSgXlHhyI44D_$solIzy #=qwj0e5S1RnJLzDFCfm24$BQ==)
   at #=qsKuflGDN3fZCLOr01hIIocjFaCzdsuS1JqDDepouZ58gd4dvXK3NvKRllaatJwlP.#=qVdHu6ElWQwXpMNTAaTNw8Q==(#=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qTLRAluB4Zj3MOKAZ1qKuRQ==, #=qbPfGgcQZm7XmxUdKwn78NGPKAThoaikPx9HzVS6JcQPEXSgXlHhyI44D_$solIzy #=qwj0e5S1RnJLzDFCfm24$BQ==)
   at #=qsKuflGDN3fZCLOr01hIIocjFaCzdsuS1JqDDepouZ58gd4dvXK3NvKRllaatJwlP.#=qVdHu6ElWQwXpMNTAaTNw8Q==(#=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qTLRAluB4Zj3MOKAZ1qKuRQ==, #=qbPfGgcQZm7XmxUdKwn78NGPKAThoaikPx9HzVS6JcQPEXSgXlHhyI44D_$solIzy #=qwj0e5S1RnJLzDFCfm24$BQ==)
   at #=qsKuflGDN3fZCLOr01hIIocjFaCzdsuS1JqDDepouZ58gd4dvXK3NvKRllaatJwlP.#=qVdHu6ElWQwXpMNTAaTNw8Q==(#=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qTLRAluB4Zj3MOKAZ1qKuRQ==, #=qbPfGgcQZm7XmxUdKwn78NGPKAThoaikPx9HzVS6JcQPEXSgXlHhyI44D_$solIzy #=qwj0e5S1RnJLzDFCfm24$BQ==)
   at #=qsKuflGDN3fZCLOr01hIIocjFaCzdsuS1JqDDepouZ58gd4dvXK3NvKRllaatJwlP.#=q_0ysF7aAbHu1SLO4VR_rIw==(Stream #=qlm4fUWFfaV9euvVrpbjw7A==)
   at #=qQE60BbY$Zs5OqBGSGNzfwIshmEJoDA7HXayFZYKFUidb_nJ$PDJmPaOuSx1Uta4Y.#=qskrjcvWyQMBDOics2wUhoA==(#=qfOrRPIRhOULwBRkH1dEbJk6JSg4MvZgUKnoj38H$yqPfLGClqnbEV0Hhoao7d5Sr #=qMqKplACKurufsYdT_eXhNQ==, #=qIHFqGprmUS$WyGsYOH_MioW6lbxUXBTzmZ9VSPpwuFXohQ6b9K6KQX01gK777JiG #=qLDlwzpT8wDb61_MEVBJKvw==, #=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qjqGISNVJPJaVqR_8R7$$sg==)
   at #=qXoFPe7tt_RiuaNI_mj_UKw6um4pnGMWUvoOrJJ844KqgcDPpDRQSh0XpMDfir1wG.#=qsAhVZMMQXwBsRYzjySlc_w==(#=qRy6JeJf7dHQOCspx8K3sE5MP8kZLHclzJccpxvOkPM$bu4A2jB$gSu4pRGB2zJJH #=qYxIPCavr1aqFpIMpNFUiqg==, #=qfOrRPIRhOULwBRkH1dEbJk6JSg4MvZgUKnoj38H$yqPfLGClqnbEV0Hhoao7d5Sr #=qMqKplACKurufsYdT_eXhNQ==, #=qIHFqGprmUS$WyGsYOH_MioW6lbxUXBTzmZ9VSPpwuFXohQ6b9K6KQX01gK777JiG #=qLDlwzpT8wDb61_MEVBJKvw==, #=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qjqGISNVJPJaVqR_8R7$$sg==)
   at #=ql4uZD8C73$vAg$pAM9tvcHQf6qu221KRILZkO7NbXuCU23551HGIVjaVpngMowjZ.#=qsAhVZMMQXwBsRYzjySlc_w==(#=qRy6JeJf7dHQOCspx8K3sE5MP8kZLHclzJccpxvOkPM$bu4A2jB$gSu4pRGB2zJJH #=qYxIPCavr1aqFpIMpNFUiqg==, #=qfOrRPIRhOULwBRkH1dEbJk6JSg4MvZgUKnoj38H$yqPfLGClqnbEV0Hhoao7d5Sr #=qOljHGBbkaYJLbHv32gdHKA==, #=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qjqGISNVJPJaVqR_8R7$$sg==)
   at #=qeK8_LtDU0$Lf2J352wZnP1BuzM6GYiqjOifXlSYkC6lFe_ONa_yeU1knmtUzC4OE.#=qsAhVZMMQXwBsRYzjySlc_w==(#=qRy6JeJf7dHQOCspx8K3sE5MP8kZLHclzJccpxvOkPM$bu4A2jB$gSu4pRGB2zJJH #=qYxIPCavr1aqFpIMpNFUiqg==, #=qfOrRPIRhOULwBRkH1dEbJk6JSg4MvZgUKnoj38H$yqPfLGClqnbEV0Hhoao7d5Sr #=qMqKplACKurufsYdT_eXhNQ==, #=qIHFqGprmUS$WyGsYOH_MioW6lbxUXBTzmZ9VSPpwuFXohQ6b9K6KQX01gK777JiG #=qLDlwzpT8wDb61_MEVBJKvw==, #=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qjqGISNVJPJaVqR_8R7$$sg==)
   at #=ql4uZD8C73$vAg$pAM9tvcHQf6qu221KRILZkO7NbXuCU23551HGIVjaVpngMowjZ.#=qsAhVZMMQXwBsRYzjySlc_w==(#=qRy6JeJf7dHQOCspx8K3sE5MP8kZLHclzJccpxvOkPM$bu4A2jB$gSu4pRGB2zJJH #=qYxIPCavr1aqFpIMpNFUiqg==, #=qfOrRPIRhOULwBRkH1dEbJk6JSg4MvZgUKnoj38H$yqPfLGClqnbEV0Hhoao7d5Sr #=qOljHGBbkaYJLbHv32gdHKA==, #=qXKOiAG0TeAbXQuhahetODo0bafajYUCZM5VAVi5UjCa1O3QwATyMsW9V9nJRJpGU #=qjqGISNVJPJaVqR_8R7$$sg==)
   at Aspose.Email.Outlook.MapiMessageReader.ReadMessage()
   at Aspose.Email.Outlook.MapiMessage.FromFile(String path)

In this particular case, it's a 32-bit Windows 7 machine with 3GB installed and 665 MB free memory although the largest free block is just 15 MB which is why MemoryStream throws the exception. The point is - the amount of memory available is more than enough to open the email - it's just that MemoryStream fails unless it can satisfy the allocation using a single block, and no such block exists.

Please would you advise your progress addressing this issue?

tahir.manzoor · August 16, 2016, 2:21pm

Hi Brian,

Thanks for your inquiry. Unfortunately, your issue is not resolved yet. We have asked from our product team about the ETA of this issue. As soon as any information is shared by them, we will be more than happy to share that with you.

Thank you for your patience and understanding.

Brian_THOMAS · September 5, 2016, 3:51am

Please would you update the status of this issue?

tahir.manzoor · September 5, 2016, 9:47am

Hi Brian,

Thanks for your inquiry. We regret to share with you that there is no update available on this issue (WORDSNET-13730) at the moment. Once there is any update available on this issue, we will inform you via this forum thread.

We apologize for your inconvenience.

Brian_THOMAS · September 5, 2016, 12:10pm

Can you tell me whether you are actually working on this? If we know that you will at some point fix it, it affects our development direction. Likewise, if we know you’ve logged it but have no intention of fixing it that helps us as well. I’d sooner know that you won’t deliver a fix rather than being given assurances that never amount to anything.

tahir.manzoor · September 6, 2016, 6:11am

Hi Brian,

Thanks for your inquiry and being patient. Unfortunately, there is no further news about this issue and we can't provide any estimates either. Keeping in view the complexity of this issue, I am afraid, we won't be able to fix it in near future. We will inform you via this thread as soon as this issue is resolved. We apologize for your inconvenience.

Brian_THOMAS · October 11, 2016, 10:58am

We've just had this reported again, this time on a 64-bit Windows 7 machine with 8 GB installed. Although just 550 MB is allocated, the largest free block is 9.6 MB and we get an out-of-memory exception.

The importance of this issue cannot be overstated - this is really massive for us. Please advise what new agreement would we have to sign to get you to move this up your list of pending bugs.

Shaun_REM · October 12, 2016, 4:24pm

Have you seen this Brian?

GCSettings.LargeObjectHeapCompactionMode Property

Brian_THOMAS · October 13, 2016, 8:37am

Very many thanks for the suggestion Shaun. I had already seen that page. In fact some time ago we specifically moved to a later version of .NET so we could use the feature, but it hasn't improved performance significantly - we still get the reported error.

Can I ask what's your interest in this particular issue?

kashif.iqbal · October 13, 2016, 11:52am

Hi Brian,

With reference to EMAILNET-35246, the issue is under investigation by our Product team. However, it couldn’t be completed due to the complexities involved. Further information about the status of possible rectification of this behavior may be available by the end of next week after which we’ll be in a position to share an ETA if possible. We appreciate your little more patience until then.

Shaun_REM · October 14, 2016, 10:39am

This would seem like a more suitable replacement since it subclasses MemoryStream and is written by Microsoft.

Announcing Microsoft.IO.RecycableMemoryStream | Philosophical Geek

GitHub - microsoft/Microsoft.IO.RecyclableMemoryStream: A library to provide pooling for .NET MemoryStream objects to improve application performance.

@Brain - Just found it an interesting thread and very surprised that Aspose is trying to say it’s not their problem when it clearly is and the reasons they’re giving you are pathetic.

kashif.iqbal · October 16, 2016, 4:33am

Hi Brian and Shaun,

We haven’t denied to rectify the issue as our Product teams have worked over it and the complexity involved needs more time for resolution of this issue. Often, it so happens that fixing an issue is postponed to lateral versions keeping in view that it may give rise to further problems if not tested properly.

Anyways, we have further logged these comments for our Product team and will update you here as and when some information is available in this regard.

Brian_THOMAS · November 7, 2016, 8:46am

Very nearly six months have passed since I first logged this error. Would you give us a status report, ideally with actual dates, and not, please, just an open-ended “we’re looking into it”.

This issue continues to give us major problems and we need it fixed. We are forced into contemplating re-architecting our code to try and avoid this aspose problem.

In an earlier post I asked “Please advise what new agreement would we have to sign to get you to move this up your list of pending bugs.” but there was no reply. Would you answer this time?

kashif.iqbal · November 8, 2016, 3:11am

Hi Brian,

We have inquired the status of this issue from our Product team and, unfortunately, the issue can’t be fixed for Aspose.Email for .NET in near future due to the complexity involved. You can still opt for subscribing to Priority Support by posting your inquiry in Aspose.Purchase forum but that may not be of much help in resolving the issue earlier.

We are sorry for the inconvenience caused to you in this regard.