Free Support Forum - aspose.com

Special characters breaking in appointments

We are experiencing an issue while parsing meetings/appointments.

Description:

  1. We created an meeting invite using outlook with different special characters. (Sample_Meeting_For_JIRA.eml)

  2. On parsing this eml file using the below code we were able to generate a .pst file, which had all the special characters improper.(Sample meeting With Forced RTF false.eml, .eml extracted from the .pst)

  3. On commenting the line mapiConversionOption.setForcedRtfBodyForAppointment(false); from the below code and generating a .pst, we found the error was no longer observed. But we cannot afford to comment this line as it breaks the html formatting for the mail. (Sample meeting With Forced RTF as default.eml)

Please refer to the uploaded .zip file for the mentioned .eml files. SampleEmails.zip (11.2 KB)

Below is a sample test case that is failing:

import static org.junit.Assert.assertTrue;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.util.Date;
import java.util.Enumeration;
import java.util.Map;
import java.util.Properties;
import javax.mail.BodyPart;
import javax.mail.Header;
import javax.mail.MessagingException;
import javax.mail.Multipart;
import javax.mail.Session;
import javax.mail.internet.MimeMessage;
import javax.mail.util.SharedByteArrayInputStream;

import org.apache.commons.io.IOUtils;
import org.junit.Test;

import com.aspose.email.AlternateView;
import com.aspose.email.Appointment;
import com.aspose.email.AppointmentLoadOptions;
import com.aspose.email.HeaderCollection;
import com.aspose.email.MailMessage;
import com.aspose.email.MapiConversionOptions;
import com.aspose.email.MapiMessage;
import com.google.common.collect.ImmutableMap;

public class EmailReaderTest {

static String sourcePath="/Sample_Meeting_For_JIRA.eml";

@Test
public void testEmailParser() throws IOException, MessagingException {
    try (InputStream is = new FileInputStream(new File(sourcePath))) {

        MapiConversionOptions mapiConversionOption = new MapiConversionOptions();

        mapiConversionOption.setPreserveOriginalAddresses(true);
        mapiConversionOption.setPreserveOriginalDates(true);
        mapiConversionOption.setPreserveSignature(true);
        mapiConversionOption.setUseBodyCompression(true);
        mapiConversionOption.setFormat(MapiConversionOptions.getUnicodeFormat().getFormat());
        mapiConversionOption.setPreserveEmbeddedMessageFormat(false);
        mapiConversionOption.setForcedRtfBodyForAppointment(false);

        MailMessage message = new MailMessage();
        message.setBodyEncoding(Charset.forName("UTF-8"));
        SharedByteArrayInputStream sh = new SharedByteArrayInputStream(IOUtils.toByteArray(is));
        Properties sessionProps = new Properties();
        sessionProps.setProperty( "mail.mime.address.strict", "false");
        Session session = Session.getInstance(sessionProps);
        MimeMessage mm = new MimeMessage(session, sh);
        Multipart multipart = (Multipart) mm.getContent();
        Map<String, Boolean> HEADER_NAMES_ADDRESSES = ImmutableMap.<String, Boolean>builder()
                .put("from", false)
                .put("sender", false)
                .put("to", true)
                .put("cc", true)
                .put("bcc", true)
                .put("reply-to", true)
                .build();
        mm.removeHeader("Content-Transfer-Encoding");
        message.setDate(new Date());
        message.setTimeZoneOffset(0);
        message.setSubjectEncoding(Charset.forName("UTF-8"));

        HeaderCollection hc = message.getHeaders();

        Enumeration<Header> headers = mm.getAllHeaders();
        while(headers.hasMoreElements()) {
            Header header = headers.nextElement();
            hc.add(header.getName(),header.getValue());
        }

        for (int i = 0; i < multipart.getCount(); i++) {
            BodyPart part = multipart.getBodyPart(i);
            //System.out.println( "Content-type:"+part.getContentType());
            String contentString = new String(IOUtils.toByteArray(part.getInputStream()),Charset.forName("UTF-8"));
            if(part.getContentType().toLowerCase().startsWith("text/html")){
                AlternateView view = AlternateView.createAlternateViewFromString(contentString,
                        Charset.forName("UTF-8"), "text/html");
                message.isBodyHtml(true);
                message.getAlternateViews().addItem(view);
               // System.out.println(" view: "+ new String(IOUtils.toByteArray(view.getContentStream()), Charset.forName("UTF-8")));
            }
            else if(part.getContentType().toLowerCase().startsWith("text/calendar")){
                AppointmentLoadOptions appointmentLoadOptions = new AppointmentLoadOptions();
                appointmentLoadOptions.setApplyLocalTZ(false);
                appointmentLoadOptions.setIgnoreSmtpAddressCheck(true);
                byte[] contentBytes = contentString.getBytes(Charset.forName("UTF-8"));
                Appointment appointment = Appointment.load(new ByteArrayInputStream(contentBytes), appointmentLoadOptions);

                message.getAlternateViews().addItem(appointment.requestApointment());
            }else if(part.getContentType().toLowerCase().startsWith("text/plain")){
                AlternateView view = AlternateView.createAlternateViewFromString(contentString,
                        Charset.forName("UTF-8"), "text/plain");
                message.getAlternateViews().addItem(view);
            }

        }

        MapiMessage mapiMsg = MapiMessage.fromMailMessage(message, mapiConversionOption);
       //System.out.println("Mapi msg: "+mapiMsg.getBodyHtml());
        assertTrue(mapiMsg.getBodyHtml().contains("→"));
    }
}

}

@curtisyamada

I have created a ticket with ID EMAILJAVA-34770 in our issue tracking system to further investigate and resolve the issue. This thread has been linked with the issue so that you may be notified once the issue will be fixed.

@mudassir.fayyaz
any updates on the issue?

@curtisyamada

I regret to share that at present the concerned issue is still unresolved. We request for your patience and will share the good news with you as soon as the issue will be fixed.

@mudassir.fayyaz Could you provide an estimate on when this issue will be resolved?

@curtisyamada

I regret to share that the issue has just recently been added in our issue tracking system and is pending for investigation at the moment in issues queue. I request for your patience and will share the good news with you as soon as the issue will addressed.

@mudassir.fayyaz
Any updates??

@curtisyamada

I regret to share that at present the concerned issue is still unresolved. We request for your patience and will share the feedback with you as soon as it will be fixed.

@mudassir.fayyaz
The status of the issue: EMAILJAVA-34770 has been updated to Resolved.
On checking the below Release notes, I can see the issue resolution is included in the new Release.
But even after updating the artifact version of aspose-email to 20.12, I still experience the same issue. Could you please let us know if there are more changes required other than the version upgrade of the artifact, to resolve the issue at our side??

@curtisyamada

We are verifying this on our end and will share feedback with you as soon as possible. Can you please share the output that you have obtained using latest Aspose.Email for Java 20.12 on your end.

The issues you have found earlier (filed as EMAILJAVA-34770) have been fixed in this update.

@mudassir.fayyaz
The test case mentioned in the first post of this thread is still failing at,
assertTrue(mapiMsg.getBodyHtml().contains(“→”));

Please let me know what is the result your team obtained, on running the mentioned Junit test-case with provided test-data

@curtisyamada

We will get back to you with feedback as soon as possible.

@curtisyamada

We have fixed issue with Calendar conversion in PST. MapiMessage contains three properties to store message body content:

The text in a MAPI message may be stored in following properties:
PR_BODY(http://msdn.microsoft.com/en-us/library/office/cc765874(v=office.15).aspx) is used to store plain text
PR_HTML(http://msdn.microsoft.com/en-us/library/office/cc842395(v=office.15).aspx) is used to store html text
PR_RTF_COMPRESSED(http://msdn.microsoft.com/en-us/library/office/cc815911(v=office.15).aspx) is used to store rtf text.

Properties have following priorities for displaying (if possible): PR_RTF_COMPRESSED or PR_HTML and then PR_BODY.

Without PR_RTF_COMPRESSED property, we cannot ensure that the message will open correctly with different versions of Outlook.

In our API we set PR_RTF_COMPRESSED property in case of HTML content. The HTML content will be converted to RTF before set. The PR_HTML property also assigned with HTML source.
Using mapiMsg.getBodyHtml() we get the PR_RTF_COMPRESSED property value converted from RTF to HTML.

We need to check PR_HTML MAPI property in the test:

...
String mapi_PR_HTML_Content = new String(mapiMsg.getProperties().get_Item(MapiPropertyTag.PR_HTML).getData(), Charset.forName("UTF-8"));
assertTrue(mapi_PR_HTML_Content.contains("→"));

Or we can delete PR_RTF_COMPRESSED property. !But we cannot ensure that the message will open correctly in Outlook:

...
mapiMsg.getProperties().remove(MapiPropertyTag.PR_RTF_COMPRESSED);
assertTrue(mapiMsg.getBodyHtml().contains("→"));