Aspose components use the .NET MemoryStream

I had problems running my code on heavily loaded systems: I got insufficient memory exceptions thrown by the .NET MemoryStream constructor because it could not find a region of unfragmented address space large enough to satisfy the allocation. This is a well-known issue with MemoryStream (just google!)

I have now replaced every MemoryStream in my code with a new implementation. Instead of using a single byte array whose size doubles until it's big enough for the request and which may cause the LOH (Large Object Heap) to become fragmented, it has multiple small byte arrays allocated from the SOH that get stitched together to behave like one big array.

However, aspose objects are still using MemoryStream. Have you seen the insufficient memory exception? Would you consider adopting a replacement for MemoryStream that is less likely to throw these exceptions under heavy load conditions?

Hi Brian,

Can you please share which Aspose APIs are you using, your code and a few samples to reproduce the issue?

Best Regards,

Hi Muhammad,

I have attached the Visual Studio sources for a command-line app that demonstrates the reported problem.

You'll see the app builds to target x86 rather than x64 or Any CPU. This is to reflect the fact that the customers reporting this issue are still using 32-bit machines, and we cannot insist they move to x64.

The built app runs in two steps:

Step 1 - Call a function named ConsumeMemory that constructs two List collections using many 100KB MemoryStreams, It adds each even-numbered MemoryStream to one List, and each odd-numbered MemoryStream to the other List. But it only actually returns one of the Lists, so the Garbage Collector should free up the other List. This is my attempt at deliberately fragmenting the LOH (Large Object Heap).

Key point is that after ConsumeMemory returns, there is still lots of memory available, but the largest contiguous chunk is only 100KB big.

Step 2 - Load the large msg file. The msg I provided might look a bit contrived, but it is similar to the files our customers expect to be able to print. Our customers are solicitors, and they print multi-page pdf files made from scans of paper documents and large Word documents and the like.

How to demonstrate the problem

1. Run the app with no command line options. It will attempt to allocate 20000 x 100KB MemoryStreams, which will cause an exception message:

ConsumeMemory requested 20000 100KB chunks. Allocation failure occurred when idx = 18871

On my machine the allocation failed trying to allocate MemoryStream number 18871. YMMV

2. Run the exe again, but specify a number on the command line. Use a slightly reduced chunk count from the first time around, e.g. AsposeTestApp 18800. When I did this I got an exception:

Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream..ctor(Int32 capacity)
   at #=q0O7zzHnm8HpMXWxOGM7N5F2yF0s7heAiNWzl0$t9b5MhWBLD2oTJZJtS0AKOwAaq.#=qsFJfxFNgjyAsqjNaNyhTKQ==(UInt32 #=qd0lAJycU69Lb_GnugvnJeQ==, Int32 #=q3ahYqqrsj91BmlmTpFlxlw==, Boolean #=qoYNI3d_i4_JwhgDK1y3dng==)
   at #=q0O7zzHnm8HpMXWxOGM7N5F2yF0s7heAiNWzl0$t9b5MhWBLD2oTJZJtS0AKOwAaq.#=qWztQNBsuX_$XB$EDpYmdSA==()
   at #=q0O7zzHnm8HpMXWxOGM7N5F2yF0s7heAiNWzl0$t9b5MhWBLD2oTJZJtS0AKOwAaq.#=qCD3HyoO3c1W_G4TPEFPovQ==(Stream #=qs89w56tWTZS_dmT1MHeNCQ==)
   at #=q0O7zzHnm8HpMXWxOGM7N5F2yF0s7heAiNWzl0$t9b5MhWBLD2oTJZJtS0AKOwAaq..ctor(Stream #=qs89w56tWTZS_dmT1MHeNCQ==)
   at Aspose.Email.Outlook.MapiMessageReader..ctor(Stream stream)
   at Aspose.Email.Mail.MailMessage.#=qmDOctRE$xFFKVn8a4_am3w==(Stream #=qs89w56tWTZS_dmT1MHeNCQ==, MsgLoadOptions #=qHgmybkduHS8HPZm$NSK97w==)
   at Aspose.Email.Mail.MailMessage.#=qHf2Oi8A8TQbb0w37rB0PjzdikgVuhb$j4cf3Eqa48fY=(Stream #=qs89w56tWTZS_dmT1MHeNCQ==, LoadOptions #=qHgmybkduHS8HPZm$NSK97w==)
   at Aspose.Email.Mail.MailMessage.Load(Stream stream)
   at Aspose.Email.Mail.MailMessage.Load(String fileName)
   at AsposeTestApp.Program.Main(String[] args) in c:\Development\Sandbox\AsposeTestApp\Program.cs:line 32

and you can see the exception is due to MemoryStream.

Although I cannot debug the aspose code named in the stack trace, the MemoryStream will fail because it is trying to allocate a contiguous byte array larger than 100KB and there are none in the LOH because it has been fragmented.

This would not be a problem if a different Stream type was used instead, one that did its allocations from the SOH and stitched multiple small arrays together to behave like a larger one.

A small variation on the sample code would be to use this replacement for the try block in the Main function:

try
{
    string outfileName = Path.GetTempFileName();
    List streams = ConsumeMemory(chunks);
    Aspose.Email.Mail.MailMessage mm = Aspose.Email.Mail.MailMessage.Load(@"Large Message with Large attachments.msg");
    Aspose.Email.Mail.SaveOptions so = Aspose.Email.Mail.SaveOptions.CreateSaveOptions(Aspose.Email.Mail.MailMessageSaveType.MHtmlFormat);
    mm.Save(outfileName, so);
}

and if you reduce the chunks argument slightly, e.g. 16000 instead of 18000, so the Load call succeeds, it might then go on to fail on the Save call, still with OutOfMemoryException:

Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at #=qY7f_dcWb9wEzoF7NHHGncmcWngkAQcABdYlCa_rmDCs=.#=qPBQkqGESAcLtMZDpbHi_7rZTx9f699tIhgMPZdDEq4U=(Stream #=qtzBh0wu9ew2ksWhQARc5AA==)
   at #=qF$I$r4B78E74tzvFjCifgHsR4hN2AGXhAUVvKx68Dcw=.#=qvGV8R6jk_Ir0jk1hyIMzhg==()
   at Aspose.Email.Mail.Attachment.#=qvGV8R6jk_Ir0jk1hyIMzhg==()
   at Aspose.Email.Mail.MailMessage.Clone()
   at Aspose.Email.Mail.MailMessage.#=qzhD7QiFQiaCnfjkIcIbcuw==(MhtFormatOptions #=qHgmybkduHS8HPZm$NSK97w==)
   at Aspose.Email.Mail.MailMessage.Save(Stream stream, SaveOptions options)
   at Aspose.Email.Mail.MailMessage.Save(String fileName, SaveOptions options)
   at AsposeTestApp.Program.Main(String[] args) in c:\Development\Sandbox\AsposeTestApp\Program.cs:line 34

Hi Brian,

Thank you for sharing the additional details. We are investigating this issue at our end to assist you further and will soon share our feedback with you here.

Hi Brian,

Thank you for being patient.

We have further investigated this issue and were able to observe the problem at our end. We have logged the issue as EMAILNET-35246 in our issue tracking system for further investigation by our Product team. We’ll update you here once there is some information available in this regard.

Thank you for the update Kashif.

This issue is blocking our product release at the moment, so we're anxious to learn how work is progressing. Can you give me some idea whereabouts in your escalation scheme this issue is? Have you assigned this a high priority? And can you tell me when you might issue a new release containing a fix?

If no fix is available in the short term, can you give me some workaround?

Hi Brian,

The issue is under investigation by our Product team and there is no ETA available for now. Once the issue is investigated and there is some information available about it, we’ll update you here via this thread. We are sorry but there doesn’t seem to be any other work around method that can be shared at this stage with you as the API is using MemoryStream at its base.

Thank you for the update Kashif. I’ll await further updates.

You are welcome.

The MemoryStream replacement I'm using is MemoryTributary from www.codeproject.com


Hi Brian,


Thank you for providing additional information. This article is already under consideration by the the product team.

As a short-term workaround, please advise whether it would be worthwhile me trying the following:

  1. Open the msg in an Aspose MailMessage
  2. Call methods to remove all non-inline attachments
  3. Save the remaining msg content out as mhtml
  4. Allow GC to dispose of the MailMessage

My existing method sets MhtFormatOptions flags so that the generated mhtml content doesn't contain non-inline MIME content, but the Save method throws an exception because it is implemented using MemoryStream

Hi Brian,

If you are getting exception even after using the steps 1-3, then their is no method to get you the output as the error will arise anyways. Our team is working on this issue and we plan to include the fix for this version in next version of the API if the complexities are resolved.

For step 4, you can use MailMessage.Dispose() as well for cleaning purpose.

Please can you reassure me that you're not just fixing this in one place in Aspose.Email? This is because I can get an OutOfMemoryException from Aspose.Words too. The stack trace shows that again it is due to MemoryStream:

System.OutOfMemoryException occurred
  HResult=-2147024882
  Message=Exception of type 'System.OutOfMemoryException' was thrown.
  Source=mscorlib
  StackTrace:
       at System.IO.MemoryStream.set_Capacity(Int32 value)
  InnerException: 

at System.IO.MemoryStream.set_Capacity(Int32 value)
at System.IO.MemoryStream.EnsureCapacity(Int32 value)
at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at . (Stream )
at .(String , Stream , String )
at .(Stream )
at .(Stream )
at .(Stream )
at .(Stream )
at …ctor(Stream , Boolean )
at …ctor(Stream , Document , LoadOptions )
at Aspose.Words.Document.(Stream , LoadOptions )
at Aspose.Words.Document.(Stream , LoadOptions )
at Aspose.Words.Document…ctor(Stream stream, LoadOptions loadOptions)
at Aspose.Words.Document…ctor(Stream stream)
at Aspose.Words.StyleCollection.(String )
at Aspose.Words.StyleCollection.()
at Aspose.Words.StyleCollection.(LoadFormat )
at Aspose.Words.StyleCollection. ()
at Aspose.Words.StyleCollection.(Int32 , Boolean )
at Aspose.Words.StyleCollection.(StyleIdentifier , String )
at Aspose.Words.StyleCollection. ()
at Aspose.Words.Document.EnsureMinimum()
at Aspose.Words.DocumentBuilder.(Int32 , StoryType , Int32 , Int32 )
at Aspose.Words.DocumentBuilder.MoveToDocumentStart()
at Aspose.Words.DocumentBuilder.set_Document(Document value)
at Aspose.Words.DocumentBuilder…ctor(Document doc)
at .(Stream , Document , Encoding )
at Aspose.Words.Document.(Stream , LoadOptions )
at Aspose.Words.Document.(Stream , LoadOptions )
at Aspose.Words.Document…ctor(Stream stream, LoadOptions loadOptions)
. . .

Hi Brian,

Thanks for your inquiry. To ensure a timely and accurate response, please attach the following resources here for testing:

  • Your input Word document.
  • Please create a standalone console application (source code without compilation errors) that helps us to reproduce your problem on our end and attach it here for testing.

As soon as you get these pieces of information ready, we'll start investigation into your issue and provide you more information. Thanks for your cooperation.

PS: To attach these resources, please zip them and Click 'Reply' button that will bring you to the 'reply page' and there at the bottom you can include any attachments with that post by clicking the 'Add/Update' button.

The code I supplied earlier demonstrated an OutOfMemoryException thrown by an instance of MemoryStream inside the implementation of Aspose.Email.Mail.MailMessage

I am concerned that MemoryStream is widely used within aspose components other than just Aspose.Email, which is why I'm reporting it when I saw the same OutOfMemoryException thrown by Aspose.Words.

To reproduce the new exception, start with the original code, and replace this block:

Aspose.Email.Mail.MailMessage mm = Aspose.Email.Mail.MailMessage.Load(@"Large Message with Large attachments.msg");
Aspose.Email.Mail.SaveOptions so = Aspose.Email.Mail.SaveOptions.CreateSaveOptions(Aspose.Email.Mail.MailMessageSaveType.MHtmlFormat);
mm.Save(outfileName, so);

with this:

using (FileStream fs = new FileStream(outfileName, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
    string raw = "This is just a short string";
    byte[] buff = Encoding.UTF8.GetBytes(raw);
    fs.Write(buff, 0, buff.Length);
    fs.Position = 0;
    Aspose.Words.Document doc = new Aspose.Words.Document(fs);
    Console.WriteLine("Success!");
}

There is no starting Word document - I'm just using a single line of raw text in a Stream.

Having made this change, run the program with an empty command line inside VisualStudio, making sure that within its Debug | Exceptions dialog, every exception category is checked.

Since the command line is empty, the ConsumeMemory function will throw an OutOfMemoryException. Continue execution from there and there will be a second Exception, this time from the Aspose.Words.Document constructor. The internal Exception is this:

  HResult=-2146233088
  Message=Cannot extract
  Source=Aspose.Words
  StackTrace:
       at     . (String  , Stream  , String  )
  InnerException: System.OutOfMemoryException
       HResult=-2147024882
       Message=Exception of type 'System.OutOfMemoryException' was thrown.
       Source=mscorlib
       StackTrace:
            at System.IO.MemoryStream.set_Capacity(Int32 value)
            at System.IO.MemoryStream.EnsureCapacity(Int32 value)
            at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
            at     .  (Stream  )
            at     . (String  , Stream  , String  )
       InnerException: 
Hi Brian,

Thanks for sharing the detail. We have tested the scenario and have noticed that System.TypeInitializationException is thrown while loading Document after calling ConsumeMemory method. For the sake of correction, we have logged this problem in our issue tracking system as WORDSNET-13730. You will be notified via this forum thread once this issue is resolved.

We apologize for your inconvenience.

Thank you Tahir.

The source of both the exceptions I've now posted is MemoryStream. And I'm certain these two instances I've logged aren't the only places in aspose code that use MemoryStream.

The wider point I'm trying to get across to you is that any code that uses MemoryStream class will throw an OutOfMemoryException if the LOH is fragmented and the request is larger than any free block in the LOH. And I'd suggest that the way to avoid this is to code defensively and adopt a different Stream object that doesn't rely on the LOH for its backing store and so cannot throw this sort of exception, rather than trying to tackle each exception on a piecemeal basis.

Hi Brian,

Thanks for sharing the detail about MemoryStream issue. We will inform you via this forum thread once there is any update available on these issue. Please let us know if you have any more queries.