Schedule a Demo
Blog June 21, 2021 C#, Certificates, Cryptographic Keys

Handling X509KeyStorageFlags in applications

by Vadims Podāns

Hello everyone!

While participating on StackOverflow.com, I’m observing common in-app certificate handling misuses in .NET applications and I want to share some thoughts on this. Today I would like to speak about handling X509Certificate2 object creation inside the application code, common problems in handling private key material, potential issues and how to overcome them.

Problem Description

The not-so-recent “TLS Everywhere” enforcement in Internet PKI forced software developers to use digital certificates in their applications more frequently. Many applications are no longer monolithic, they use external services to process the data and may use certificates for internal use. Common use cases covered by this blog post include:

  • Application uses client certificate to authenticate itself at remote API;
  • Application uses certificate to digitally sign or encrypt some content.

However, PKI is still hard and the number of PKI misuses in software increased accordingly and one of [many] common issues is certificate and private key handling inside .NET applications. I found that many examples of certificate usage inside applications are broken or flawed in one or another ways. Threads on public forums use copy/pastes from other boards with a little care of what exactly these flags do behind the scene.

Most problems with these use cases occur in web apps when certificate lookup/instantiation fails. For example, certificate reading works on a developer machine, but fails when the application is deployed to production. Few common threads on StackOverflow related to this subject:

Many reasons may lead to these issues. And even if application looks working, security flaws are introduced in the application when private key material is unintentionally shared with other processes or simply leaked from the system.

Application Certificate Deployment Models

There are two common deployment models for application certificates:

  • Persistent — certificate is installed in certificate store which is independent to application.

In persistent deployment model, a certificate is installed in certificate store (such as Windows Certificate Store) using out-of-band process. The installed certificate is shared with any other application that runs under same security account.

The benefits of this deployment model include simplified certificate management, it can be easily installed and managed using external mechanism, such as certificate autoenrollment and certificate lifecycle managers (CLM). CLM makes it easy to manage certificates in certificate store without having to update or re-deploy the application each time the certificate is updated.

This deployment model is the only model that supports keys stored on hardware devices, TPM or HSM (Hardware Security Module).

However, simplified management has its own downsides. There is no certificate isolation/protection between processes. Often, when application is removed, the certificate is not removed and remains on a system. Even if certificate is removed from persistent store (windows certificate store), this doesn’t necessarily removes the private key: The case of accidentally deleted user certificates. Another process under same security account or computer administrator can easily access the certificate and its private key material.

An important note: of course, the private key can be deleted when certificate is deleted from Windows Certificate Store, but it requires some sophisticated techniques and I never seen that anyone uses it.

These downsides may compromise application security. If the certificate must be protected from unauthorized use by anyone except target application, then this deployment model SHALL NOT be used. How certificate and its private key is accessed in the code? Here is a quick example:

class MyClass {
    public static X509Certificate2 GetCertificateFromStore(StoreLocation storeLocation, String searchPattern) {
        using (var store = new X509Store(StoreName.My, storeLocation) {
            store.Open(OpenFlags.ReadOnly);
            X509Certificate2Collection certs = store.Certificates.Find(findType, searchPattern, true);
            if (certs.Count > 0) {
                return certs[0];
            }

            throw new CryptographicException("Requested certificate was not found");
        }
    }
}

The code opens certificate store, looks for certificate and then use it. No secret input is provided, so any application can use same code to access certificate installed in permanent store. However, this code may fail sometimes when storeLocation parameter points to LocalMachine. It works on developer machine and fail when application is deployed to server. This is because of different security context. Unfortunately, it is quite often when developers use highest privileges on a development machine for little to no reasons. Administrators can read and write certificates from local machine store. Standard users do not have access to certificates installed in local machine store. And applications deployed to server do not often have access to certificates installed Local Machine store. First rule of thumb: do not use Local Machine store unless the application runs under local system security context. And in 99% of applications aren’t and shall not.

  • In-app — certificate is embedded in the application itself or stored as external protected file. This includes PFX files, key chains, Azure Key Vault and many others.

This certificate deployment model has opposite features than Persistent model. In-app model provides better certificate and private key security: the private key may not be shared with other applications that run under same security account and may prevent computer administrator from accessing private key material. The idea behind this is that application loads the certificate into memory only when necessary and then unloads the certificate and private key. Certificate itself is protected with some secret (simply, PFX password) which is known only to authorized applications. How the certificate and private key is accessed in this case? Here is a quick example:

class MyClass {
    public static X509Certificate2 GetCertificateFromPfx(Byte[] pfxBytes, String password) {
        return new X509Certificate2(pfxBytes, password);
    }
}

There is a parameter called password which is known only to authorized applications and which provides private key isolation between processes. Applications that do not know the password can’t access the certificate and private key. Although the code looks secure, in fact it is not in the current way. The problem is with default behavior of the X509Certificate2 constructors that import certificate from PFX. Using default constructors, such as X509Certificate2(String, String), X509Certificate2(String, SecureString), X509Certificate2(Byte[], String), X509Certificate2(Byte[], SecureString), private key is silently copied and persisted on a system:

PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"
PS C:\> $c = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $pfxBytes,$password
PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"


    Directory: C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys


Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a--s       20.06.2021.     18:55       1603 487b53ff0ac839b14243c702864666b4_e63ebd9c-34b8-4edd-89c0-303a056a6d8e


PS C:\>

See Key Storage and Retrieval article to find paths to directories where Windows store private key files.

This example reveals the fact that private key remains on a system and literally equals to persistent model. It is pretty inevident behavior which introduces security flaws with normal working code. Any simple application/script can use FileSystemWatcher to detect key file creation and immediately create a copy of the key file. Unfortunately many developers simply miss this, because the Microsoft documentation isn’t very clear and require a solid background knowledge on how cryptography works. Didn’t I say the PKI is hard?

To avoid this non-desirable behavior, a constructor with three parameters must be used. Third parameter is of type of X509KeyStorageFlags enumeration. Let’s explore options with a brief explanation:

  • DefaultKeySet — copies private key to default store. It can be specified in PFX properties (user key set or machine key set). If none specified in PFX, user key set is used. This is something you don’t want to use, because it persists the key on a disc. And if PFX specifies Machine Key Set and application doesn’t have local administrator/system permissions, the call will fail with exception.
  • UserKeySet — forces private key copy to user profile regardless what is specified in PFX properties.
  • MachineKeySet — forces private key copy to local system profile. If application doesn’t have local administrator/system permissions, the call will immediately fail.
  • Exportable — copies private key to default key set (see rules above) and makes private key exportable. This makes zero sense, because application doesn’t need to have an access to private key material. If the application require this, the application is simply wrong. Instead, the application must deal with handles provided by operating system. Nothing else.
  • UserProtected — this flag enables private key strong protection. This doesn’t work with any non-interactive applications, because raise UI popups and require user input. And in web/background applications the popup is raised in background sessions you cannot access.
  • PersistKeySet — is similar to DefaultKeySet with the exception that private key permanently persists on a disk even if you explicitly release all handles.
  • EphemeralKeySet — the only flag that doesn’t copy private key material on a disk and the only that makes sense, because it does exactly what we need — temporarily load the key from PFX and unload when finished.

Obviously, if no parameter of X509KeyStorageFlags enumeration is passed, a DefaultKeySet value is used, which indeed copies the private key to file system. Before .NET Framework 4.7.2 and .NET Core 2.0 any enumeration value or combination persisted the key on a disk and behave similar like persistent deployment model. Starting with aforementioned framework versions, there is a new enumeration value called EphemeralKeySet. If you use it, the private key file is not created on a disk:

PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"
PS C:\> $c = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $pfxBytes,$password,"ephemeralkeyset"
PS C:\> dir "C:\Users\TestUser\AppData\Roaming\Microsoft\Crypto\Keys"
PS C:\>

One could argue that starting with .NET Framework 4.5.2 you can call X509Certificate2.Dispose() method to delete temporary key file from disk, however for short period of time the file sits on a disk. And any simple application/script can use FileSystemWatcher to detect key file creation and immediately create a copy of the key file. And call X509Certificate2.Dispose() every time when you finish to deal with the certificate.

This leads to an evident solution: you SHALL use EphemeralKeySet enumeration value when load certificate with private key from PFX or Azure Key Vault, for example. If you use any other flag, then you are doing something wrong.

Summary

In this post, I outlined common certificate usage scenarios in applications, common certificate deployment options, their properties, benefits and problems. In summary, I want to provide common best practices when dealing with certificates inside applications:

  • Use persistent certificate deployment model if you store certificate and private keys on a hardware device, such as TPM or HSM.
  • Be sure if you understand the certificate and private key sharing across processes in persistent certificate deployment model.
  • Do not use LocalMachine certificate store in persistent certificate deployment model. Use only CurrentUser store.
  • Use only three-parameter constructor for X509Certificate2 class and use only EphemeralKeySet in the third parameter in in-app certificate deployment model.
  • Always call X509Certificate2.Dispose() method when finishing the certificate usage. No exceptions.

If you do not follow these simple rules — you misuse the PKI.

Happy coding!

Related Resources

  • Blog
    March 7, 2024

    PKI Insights – Avoiding PenTest Pitfalls

    Certificates, PKI, PKI Insights
  • Blog
    March 7, 2024

    Why you are getting it wrong with Certificate Lifecycle Management

    Certificate Management, Certificates, CLM
  • Blog
    February 6, 2024

    PKI Insights Recap – Microsoft Intune Cloud PKI

    BYOD, Certificates, Cloud, Enrollment, NDES

Vadims Podāns

PKI Software Architect

View All Posts by Vadims Podāns

Comments

  • Very interesting article, Vadims!
    I’ve seen a lot of applications (even the monolithic ones) fail the TLS requirements because the developers simply do not understand the functionality of the authentication/verification/chains and so on with regards to PKI.
    And to be honest: most of the developers I met, couldn’t care less about certificate authentication and encryption. They just want their code to work flawlessly.
    Most of these authentication failures I have had to investigate are not even related to the windows certificate store, because that would be too easy, right? They mostly relate to Java applets that use their own certificate store/vault.

    In my country GDPR is stringent. Due to that when performing data transfer involving personal information from the government, all network traffic must be mutually authenticated and encrypted.
    For this I’ve read a 36-page governmental procedure on how to create a personal vault for each of these java applets, and then another 50 pages on how to sign the certificate using the government CA, and another 38 pages procedure on how to import the signed certificate in the vault, finishing with 15 pages on how to generally integrate the vault in your web-application. These last pages are so generalized, that it cannot really be called a procedure.
    It always fails the first time because the developers have their TLDR moment and simply do not understand what these detailed procedures mean. Nor do they have the know-how of PKI. This always generates an extra load of work to get the application working (either in development, acceptation, or production environment)

    As you said: PKI is indeed hard!

    • Thanks for interesting comment! I’m from EU and I know how hard GDPR is. I totally agree on developers that don’t want to make things right, they want things to be working. How? Doesn’t matter. Since PKI doesn’t provide visible benefits (i.e. hard to sell to customer) any time invested in learning PKI is often considered a wasted time and money. As the result, we constantly see how even big labels are hacked because of security breaches and will continue to happen often and often.

  • Hi Vadmis,

    I would argue that using non persistent certificate storage creates a false feeling of security.
    The pfx resides on the harddisk, the application with built-in password also.
    Anyone who can access the pfx & application binaries can use the private key.
    Often the files are readable for regular users.

    And I would argue that it’s a good idea to use the localmachine store but allow key access only for relevant serviceaccounts (gMSA).
    That protects against user attacks and the protection get’s even better with a HSM

    • The PFX is protected with password and only applications that have password in PFX can access its keys. However, when key persists, any application that runs under same security context may access the private key, because it is no longer protected by PFX password. This is the difference.

  • If we use “EphemeralKeySet ” flag for azure keyvault case. Will a certificate be visible in a memory dump ?

    • yes, the certificate will be visible in a memory dump. EphemeralKeySet means that the key is stored only in memory and not saved on a disk.

  • In case of single instance of azure web app, EphemeralKeySet is working fine. But in case of multiple instances – that’s failing intermittently and throwing this error – The specified network password is not correct. Any suggestions on how this can be investigated further?

    • Unfortunately, there are too many unknowns. I would recommend to address your question on StackOverflow.

  • Thank you, great article! The EphemeralKeySet flag makes sense, but according to this Github issue https://github.com/dotnet/runtime/issues/23749, it does not work on Windows hosts :/

    If this is the case, what is the recommended approach? Store the X509Certificates to file, then try and clean them up when the application is disposed?

Leave a Reply

Your email address will not be published. Required fields are marked *