Ncryptdecrypt fails to decrypt data that is encrypted by openssl with RSA_PKCS1_OAEP_PADDING

Question

0.00/5 (No votes)

See more:

I have difficulty to decrypt data being encrypted using OpenSSL, RSA and RSA_PKCS1_OAEP_PADDING padding option.

What I am doing is to load the key from Windows KSP:

m_hSystemStore = CertOpenStore(CERT_STORE_PROV_SYSTEM, 0, NULL, CERT_SYSTEM_STORE_LOCAL_MACHINE, m_storeName.c_str());

m_pCertWithKeys = CertFindCertificateInStore(m_hSystemStore, SupportedEncodings, 0, CERT_FIND_SUBJECT_STR, m_certName.c_str(), NULL);

// Obtain the private key from the certificate.
DWORD m_KeyContextSpec = 0;
HCRYPTPROV_OR_NCRYPT_KEY_HANDLE m_hKeyContextFull;
CryptAcquireCertificatePrivateKey(m_pCertWithKeys, CRYPT_ACQUIRE_SILENT_FLAG | CRYPT_ACQUIRE_PREFER_NCRYPT_KEY_FLAG, NULL, &m_hKeyContextFull, &m_KeyContextSpec, &m_KeyContextMustBeReleased);

and call NCryptDecrypt like:

BCRYPT_OAEP_PADDING_INFO paddingInfo = { 0 };
DWORD cbDecryptedMessage;
BYTE* pbDecryptedMessage = NULL;

paddingInfo.pszAlgId = BCRYPT_SHA1_ALGORITHM;

// Calculate the required buffer
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, NULL, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

// After required buffer is allocated...
NCryptDecrypt(m_hKeyContextFull, (LPBYTE)pEncrypted, encryptedLenInBytes, &paddingInfo, pbDecryptedMessage, cbDecryptedMessage, &outputDataLen, NCRYPT_PAD_OAEP_FLAG | NCRYPT_SILENT_FLAG);

It fails with NTE_INVALID_PARAMETER (0x80090027). I tried different flags but none of them works.

Note: All error checkings have been removed from code for readability.

The data is being encrypted with same key (public part) like:

RsaPublicEncrypt(size - 42, blk, output, Rsa, RSA_PKCS1_OAEP_PADDING)

and can be decrypted using SoftHSM successfully.

Is there any limitation for CNG to decrypt data being encrypted by OpenSSL?
Is there any idea what am I doing wrong?

Thanks.

What I have tried:

I did not change anything in OpenSSL side, because that part works and we can decrypt data using SoftHSM with same key, but:
* I have tried different flags in NCryptDecrypt
* Different algorithm for padding
* Different algorithm to generate the key
None of them worked so far.

Posted 16-Jul-18 13:44pm

Amir Dashti

Updated 16-Jul-18 23:19pm

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Jochen Arndt · Answer 1 · 2018-07-16T23:19:00

Solution 1

You should have shown the full code. So I can only guess that you might forgot to initialise cbDecryptedMessage before calling NCryptDecrypt():

C++

pbDecryptedMessage = new BYTE[outputDataLen]; // or malloc() with C
cbDecryptedMessage = outputDataLen;

Posted 16-Jul-18 23:19pm

Jochen Arndt

Comments

Amir Dashti 17-Jul-18 11:38am

Thanks for the reply, the buffer is allocated using malloc() function call. I just removed that part and mentioned it as a comment in the sample code.

Jochen Arndt 17-Jul-18 12:31pm

Yes, I saw and understand that comment.

But are you setting cbDecryptedMessage in the not posted code?

Because the first call to get the required buffer size succeeds, the error must be for one of those parameters ignored with the first call.

And these parameters are cbDecryptedMessage and pbDecryptedMessage; the only ones not shown to be initialised in your posted code!

Amir Dashti 17-Jul-18 19:22pm

You are totally right, my apologies, even the first call is failing...

Jochen Arndt 18-Jul-18 2:47am

Than it is most probably pEncrypted and/or encryptedLenInBytes because paddingInfo looks fine, an invalid handle would return NTE_INVALID_HANDLE, and an invalid flag would return NTE_BAD_FLAGS.

So check those.