IMAP Paging stops at 16000

We have been using IMAP paging using the call

ipi = imapClient.ListMessagesByPage((int)pplCommonData.DataConstants.NumberOfMessagesToRead);

To read the first 500 messages and then call

ipi = imapClient.ListMessagesByPage(ipi.NextPage); // get the next page of the list

To get the next 500 while ipi.Items.Count > 0

We have a customer that has 32000+ messages in their inbox and it appears that when we have read 16500 ( we keep a running counter ), the next message after the ListMessagesByPage(ipi.NextPage); returns message 16000 again.

I have checked the code on our side and I cannot see any issue. We made 33 paging requests without a problem and then the code goes in a loop since we can never get past 16000 to 16500.

Thanks for any help,

@PPLAspose,

Could you please share if you are facing the same results with the latest version of Aspose.Email for .NET API? Theoretically, there is no such limit on the number of pages or messages to be retrieved in the API. We don’t have a mailbox with this much large number of messages and are currently working to populate one for further testing of the issue at our end. We’ll soon share our findings here with you after we test the same at our end.

We are using 17.6, if you think 17.7 will make any difference I can download it. We also have the same issue with no test that contains over 16500 messages. If it is unlikely that the difference between 17.6 and 17.7 would make any difference, we would prefer not to bother the customer.

Thanks

@PPLAspose,

We have checked release notes of Aspose.Email for .NET 17.7 and there are no signs of any such issue related to IMAP client available in these. Thus, we are also not sure if the latest version of the API will be of help to you in this regard. However, we tested this at our end with the latest version of the API and following code sample, and all the elements are retrieved properly with program finishing at the end. The mailbox contains some 36000 messages at our end. May be your customer should give it a test with the latest version of the API and a sample console application to see if it works at his end or not?

Sample Code

ImapClient client = new ImapClient("exchange.domain.com", "username", "password");

int itemsPerPage = 500;

client.SelectFolder(ImapFolderInfo.InBox);

List<ImapPageInfo> pages = new List<ImapPageInfo>();

ImapPageInfo pageInfo = client.ListMessagesByPage(itemsPerPage);

Console.WriteLine("Pages: " + pageInfo.TotalCount);

pages.Add(pageInfo);

int iPageCount = 1;
while (!pageInfo.LastPage)
{
    pageInfo = client.ListMessagesByPage(pageInfo.NextPage);

    pages.Add(pageInfo);

    iPageCount += 1;

    Console.WriteLine(iPageCount);
}

int retrievedItems = 0;
foreach (ImapPageInfo folderCol in pages)
    retrievedItems += folderCol.Items.Count;
Console.WriteLine(retrievedItems);

Thank you for the information and example. I will update the version and double check our code against your sample code. If it works I will update this as soon as the customer lets us know.

I see one big difference between the sample and our code. We have an try-catch block for the ipi = imapClient.ListMessagesByPage(ipi.NextPage); so we can retry the command. There might have been an error and we attempted to recover, but the ImapPageInfo data may not longer be valid.

Do you have any ideas on that scenario? Do I need to save the ImapPageInfo before issuing the listmessagesbypage and restore it before retry?

Thanks

@PPLAspose,

Could you please share your exact code sample with us that we can use for further investigation of this problem at our end? We haven’t been in a scenario where the ImapPageInfo need to be saved first in case of an error. Please share your exact code sample with us for helping us in investigating this issue further at our end.

IMAPCode.zip (3.8 KB)

@PPLAspose,

Could you also please share the definitions of all the enumerations in this code sample to make it workable at our end? Ideally, the code should raise an exception with our account as well having more than 32K messages. Also, could you please try the above code at your end and let us know if it raises exception at your end?

The exception is raised when there is a connection issue and not as a result of any condition in the ASPOSE code, as far as I can tell.

You should be able to duplicate this using your sample code. just drop the connection and reestablish it in the while (!pageInfo.LastPage) but after the nextpage fetch. That is essentially what is happening except the dispose is caused by some issue on the server as seen below…

8/1/2017 5:19:31 PM: [IMAPFetchMessage] Unable to fetch message 9606. The system reports Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host… Attempting to recover from connection failure.
8/1/2017 5:19:37 PM: [IMAPConnectionRecovery (I) called from IMAPFetchMessage] Reconnected to outlook.office365.com
8/1/2017 5:19:38 PM: [IMAPConnectionRecovery (I) called from IMAPFetchMessage] Attempting to select folder Inbox
8/1/2017 5:19:38 PM: [IMAPConnectionRecovery (I) called from IMAPFetchMessage] Folder selection complete for Inbox

while (!pageInfo.LastPage)
{
pageInfo = client.ListMessagesByPage(pageInfo.NextPage);

pages.Add(pageInfo);

iPageCount += 1;

Console.WriteLine(iPageCount);

client .Dispose();
client = new ImapClient("exchange.domain.com", "username", "password");
client.SelectFolder(ImapFolderInfo.InBox);

}

Another slight difference from the sample and our code is after the listmessagebypage, we fetch each message, not just use the ImapPageInfo collection. We need to run a set of filters and analysis on the message headers, body, attachments, times. etc. to see if it meets the forensic criteria of the examiner. In the log lines in the previous reply you can see it was the fetch message the had the connection issue. Again all this could just be a red herring!

@PPLAspose,

We are working on this query and will soon write back here with our findings. However, I believe it has something to do with the logic and not with the API. In any case, we’ll share our findings with you here.

@PPLAspose,

We included try catch block around the pageInfo.NextPage and disconnected the connection right after that for testing the problem as you mentioned. However, we are afraid to share that no such issue was reproduced at our end. Please have a look at the following code sample for your kind reference. In order to further investigate the problem, we would request you to please share a complete sample running code that we can execute at our end for reproducing the same. We’ll look into it for assisting you further.

Sample Code

 ImapClient client = new ImapClient("exchange.domain.com", "username", "password");

int itemsPerPage = 500;

client.SelectFolder(ImapFolderInfo.InBox);

List<ImapPageInfo> pages = new List<ImapPageInfo>();

ImapPageInfo pageInfo = client.ListMessagesByPage(itemsPerPage);

Console.WriteLine("Pages: " + pageInfo.TotalCount);

pages.Add(pageInfo);

int iPageCount = 1;

bool bListed = false;
while (!pageInfo.LastPage)
{
    try
    {
        pageInfo = client.ListMessagesByPage(pageInfo.NextPage);
        bListed = true;
    }
    catch (Exception ex)
    {
        if (client != null)
            client.Dispose();

        client = new ImapClient("exchange.aspose.com", "asdf", "asdfds");

        client.SelectFolder(ImapFolderInfo.InBox);
    }
    if (bListed)
    {
        pages.Add(pageInfo);

        iPageCount += 1;

        Console.WriteLine(iPageCount);
    }
}

int retrievedItems = 0;
foreach (ImapPageInfo folderCol in pages)
    retrievedItems += folderCol.Items.Count;
Console.WriteLine(retrievedItems);

Thank you for looking at this. I will build a simple console app based on the sample (using command line parameters for credentials) and have the customer try it and see if he can get through his entire inbox. I will post my sample when done and keep you posted on the customer experience.

Thanks again.

@PPLAspose,

You are welcome and feel free to share your feedback with us after testing.

Here is the modified test code. I added some of the calls we make just to be sure it was as close to the flow (without the error recovery) as possible. I can run it fine against your account, so there has to be some subtle issue in our code.

using System;
using System.Collections.Generic;
using Aspose.Email.Clients.Imap;

namespace AsposeConsoleApp1
{
class Program
{
static void Main(string[] args)
{
ImapClient ic = null;
if (args.Length > 3)
{
Int32 iPort = Int32.Parse(args[3]);
ic = new ImapClient(args[0], iPort, args[1], args[2]);
}
else
{
ic = new ImapClient(args[0], args[1], args[2]);
}

		int itemsPerPage = 500;
		int iPageCount = 0;
		int iMessageCount = 0;
		int iCurrentStartMessageNumber = 1;

		try
		{
			ImapFolderInfo imapfi = ic.GetFolderInfo(ImapFolderInfo.InBox);
			if (imapfi.TotalMessageCount > 0)
			{
				int iMessagesRemaining = imapfi.TotalMessageCount;
				ic.SelectFolder(imapfi.Name);
				List<ImapPageInfo> pages = new List<ImapPageInfo>();
				Console.WriteLine("Current page: " + iPageCount +
													" Current message range " + iCurrentStartMessageNumber + " to " + (iCurrentStartMessageNumber + itemsPerPage));
				iPageCount++;
				try
				{
					ImapPageInfo pageInfo = ic.ListMessagesByPage(itemsPerPage);
					Console.WriteLine("Total number of messages is: " + pageInfo.TotalCount);
					while (!pageInfo.LastPage)
					{
						bool firstmessage = true;
						foreach (ImapMessageInfo page in pageInfo.Items)
						{
							if (firstmessage)
							{
								try
								{
									ic.FetchMessage(page.UniqueId);
									iMessageCount++;
									iMessagesRemaining--;
									firstmessage = false;
									Console.WriteLine("First message in pageinfo.items is unique id " + page.UniqueId);
								}
								catch (Exception e1)
								{
									Console.WriteLine("Fetch message failed. Last mesage count is " + iMessageCount + " Exception: " + e1.Message);
									throw;
								}
							}
							else
							{
								break;
							}
						}
						try
						{
							pageInfo = ic.ListMessagesByPage(pageInfo.NextPage);
							iCurrentStartMessageNumber = 1 + iPageCount * itemsPerPage;
							Console.WriteLine("Current page: " + iPageCount +
																" Current message range " + iCurrentStartMessageNumber + " to " + (iCurrentStartMessageNumber + itemsPerPage) );
							iPageCount++;
						}
						catch (Exception e1)
						{
							Console.WriteLine("List Messages By Page failed. Last page count is " + iPageCount + " Exception: " + e1.Message);
							throw;
						}
					}
				}
				catch (Exception e1)
				{
					Console.WriteLine("List Messages By Page failed. Last page count is " + iPageCount + " Exception: " + e1.Message);
					throw;
				}
			}
		}
		catch (Exception e1)
		{
			Console.WriteLine("SelectFolder failed. Exception: " + e1.Message);
			throw;
		}

		Console.WriteLine(" ");
		Console.WriteLine("Press any key to continue.");
		Console.ReadKey();
	}
}

}

@PPLAspose,

We have tested this modified code with connection breakups at different points and no exception is raised at our end. If your customer could share with us a simplified version of his code that gives raise to such issue, it will help us investigate the issue further at our end. We are sorry but without reproducing the issue, we won’t be able to further investigate the issue for any further assistance. We appreciate your understanding in this regard.

We are closing this at our end. The customer used one of our conversions to get the IMAP to a PST and then used the PST as a source. It worked fine, so he is not interested in pursing this. I will open a new issue if it ever reoccurs.

@PPLAspose,

Thank you for the feedback. Please feel free to write us back if you have any other query in this regard.