Thanks for your patience. I also observed that decryptFile() method doesn't throw any exception or some boolean result in case of wrong password, so I have logged an enhancement request, PDFKITJAVA-33268, in our issue tracking system. As a workaround you can use isOwnerPassword() or isUserPassword() methods of PdfViewer class for validating password and proceed accordingly.
Secondly in reference to digital certificate decryption, Unfortunately Aspose.Pdf doesn't support this feature at the moment. However, I have also logged a new feature request, PDFKITJAVA-33269, in our issue tracking system. You will be notified via this thread as soon as it gets available.
Please feel free for contact us for any further assistance.