Click here to Skip to main content
14,937,163 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I have difficulty to decrypt data being encrypted using OpenSSL, RSA and RSA_PKCS1_OAEP_PADDING padding option.

What I am doing is to load the key from Windows KSP:
m_hSystemStore = CertOpenStore(CERT_STORE_PROV_SYSTEM, 0, NULL, CERT_SYSTEM_STORE_LOCAL_MACHINE, m_storeName.c_str());

m_pCertWithKeys = CertFindCertificateInStore(m_hSystemStore, SupportedEncodings, 0, CERT_FIND_SUBJECT_STR, m_certName.c_str(), NULL);

// Obtain the private key from the certificate.
DWORD m_KeyContextSpec = 0;
CryptAcquireCertificatePrivateKey(m_pCertWithKeys, CRYPT_ACQUIRE_SILENT_FLAG | CRYPT_ACQUIRE_PREFER_NCRYPT_KEY_FLAG, NULL, &m_hKeyContextFull, &m_KeyContextSpec, &m_KeyContextMustBeReleased);

and call NCryptDecrypt like:
BCRYPT_OAEP_PADDING_INFO paddingInfo = { 0 };
DWORD cbDecryptedMessage;
BYTE* pbDecryptedMessage = NULL;

paddingInfo.pszAlgId = BCRYPT_SHA1_ALGORITHM;

// Calculate the required buffer
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, NULL, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

// After required buffer is allocated...
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, pbDecryptedMessage, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

It fails with NTE_INVALID_PARAMETER (0x80090027). I tried different flags but none of them works.

Note: All error checkings have been removed from code for readability.

The data is being encrypted with same key (public part) like:
RsaPublicEncrypt(size - 42, blk, output, Rsa, RSA_PKCS1_OAEP_PADDING)

and can be decrypted using SoftHSM successfully.

Is there any limitation for CNG to decrypt data being encrypted by OpenSSL?
Is there any idea what am I doing wrong?


What I have tried:

I did not change anything in OpenSSL side, because that part works and we can decrypt data using SoftHSM with same key, but:
* I have tried different flags in NCryptDecrypt
* Different algorithm for padding
* Different algorithm to generate the key
None of them worked so far.
Updated 16-Jul-18 23:19pm

1 solution

You should have shown the full code. So I can only guess that you might forgot to initialise cbDecryptedMessage before calling NCryptDecrypt():
pbDecryptedMessage = new BYTE[outputDataLen]; // or malloc() with C
cbDecryptedMessage = outputDataLen;
Amir Dashti 17-Jul-18 11:38am
Thanks for the reply, the buffer is allocated using malloc() function call. I just removed that part and mentioned it as a comment in the sample code.
Jochen Arndt 17-Jul-18 12:31pm
Yes, I saw and understand that comment.

But are you setting cbDecryptedMessage in the not posted code?

Because the first call to get the required buffer size succeeds, the error must be for one of those parameters ignored with the first call.

And these parameters are cbDecryptedMessage and pbDecryptedMessage; the only ones not shown to be initialised in your posted code!
Amir Dashti 17-Jul-18 19:22pm
You are totally right, my apologies, even the first call is failing...
Jochen Arndt 18-Jul-18 2:47am
Than it is most probably pEncrypted and/or encryptedLenInBytes because paddingInfo looks fine, an invalid handle would return NTE_INVALID_HANDLE, and an invalid flag would return NTE_BAD_FLAGS.

So check those.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900