Getting OOM Error while Generating PST file using output stream

Aspose.Email Product Family
1

Hi there,

I am getting Out of Memory error every time when I ingest some large set of messages (lets say 1000 messages) to create a pst file. I am passing OutputStream instance, instead of direct file location, to create a PersonalStorage object. i.e.
OutputStream outputStream = new FileOutputStream("file_name.pst");
PersonalStorage personalStorage = PersonalStorage.create(outputStream, FileFormatVersion.Unicode);

After processing some of the messages, at some point, the program fails with OoM error near FolderInfo.addMessage() function. Looks like newly generated pst file is constantly increasing the heap space and at some point it reaches the thrashhold and fails with Out of Memory error.

Please suggest how I can avoid such error. Here is my full program -

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import com.aspose.email.EmlLoadOptions;
import com.aspose.email.FileFormatVersion;
import com.aspose.email.FolderInfo;
import com.aspose.email.MapiMessage;
import com.aspose.email.PersonalStorage;

public class TestPstExport {
  public static void main(String []args) throws IOException {
    createPSTFromOutstream();
    System.exit(0);
  }

  public static void createPSTFromOutstream() {
      InputStream inputStream = null;
      OutputStream outputStream = null;
      PersonalStorage personalStorage = null;
      MapiMessage mapiMessage = null;
      String inputFileLocation = "D:/emailSamples/testmail.eml";
      try {
        outputStream = new FileOutputStream("D:/exports/pst_export_file.pst");
        checkAsposeLicense();
        personalStorage = PersonalStorage.create(outputStream, FileFormatVersion.Unicode);
        FolderInfo exportFolder = personalStorage.getRootFolder().addSubFolder("export");
        for (int i= 1; i <= 1000; i++) {
          System.out.println("Please wait while loading eml file " + i + "...");
          // taking same file again
          inputStream = new FileInputStream(inputFileLocation);
          mapiMessage = MapiMessage.load(inputStream, new EmlLoadOptions());
          // updating subject because injecting same file again and again
          mapiMessage.setSubject(mapiMessage.getSubject() + "_00" + i);
          System.out.println("Please wait while writing messages to the PST " + i + "...");
          exportFolder.addMessage(mapiMessage);
          System.out.println("Message " + i + " is added to PST file!");
          outputStream.flush();
        }
      } catch (Exception e) {
        e.printStackTrace();
      } finally {
        personalStorage.dispose();
        personalStorage.close();
      }
  }

  public static void checkAsposeLicense() throws RuntimeException {
    System.out.println("Checking for aspose licence...");
    File licenseFile = new File("D:/license/Aspose.Email.Java.lic");
    if(!licenseFile.exists()) {
      throw new RuntimeException("PST license file does not exists.");
    }    

    new com.aspose.email.License().setLicense(licenseFile);
    System.out.println("License applied successfully!");
  }
}
2

Hello @abhy022,

We will check this out and reply to you soon. Thanks.

3

thanks @DmitryS. Looking forward to your response.

4

Hello @abhy022,

Writing a pst file is not a sequential data write, but multiple seek and overwrite.
Due to PST format during add message we need to seek to a previous position and overwrite.
Java OutputStream is not supported for seek and overwrite features and we can’t use it in PersonalStorage implementation without caching.
In case of PersonalStorage.create(OutputStream changes are cached to RAM and saved to the OutputStream after PersonalStorage close.
In case of PersonalStorage.create(“tmpFileName” RandomAccessFile is used.
The RandomAccessFile supports seek and overwrite features and can be used without overhead.

You can try using the PST API, which is optimized for big PST files:

// blockSize - The optimal block size to expand cache buffer(in bytes)
// for example blockSize 10Mb = 10*1024*1024
PersonalStorage create(OutputStream stream, int blockSize, /*FileFormatVersion*/int version)

Also you can customize PST output stream that will work without overhead using Stream implementation:

PersonalStorage create(Stream stream..

CustomStream sample:

public class CustomStream extends Stream {...

// Gets the length of the stream in bytes
getLength()

// Gets the current position within the stream.
getPosition()

// Reads a block of bytes from the current stream and writes the data to a buffer
read(byte[] buffer, int offset, int count)

// Sets the position within the current stream to the specified value.
//   loc - seek reference point
//     Begin = 0
//     Current = 1
//     End = 2
seek(long offset, int loc)

// Writes a block of bytes to the current stream using data read from a buffer
write(byte[] buffer, int offset, int count)


PersonalStorage pstFile = PersonalStorage.create(new CustomStream(), FileFormatVersion.Unicode);