Thanks for your request. Usually, Aspose.Words needs 10 times more memory than the original document size to build a DOM in the memory.
The memory should be released once the Document object is not used. Since you are using streams, please make sure that you dispose your streams when they are not used.
Also, please attach your document here for testing. I will check it on my side and provide you more information.