At Tandem, we often work on software that is meant to be deployed to secure environments. Sometimes that means that setting up a dev or QA environment that is “close enough” to the architecture of production can be challenging – we have to get creative! Recently, we were engaged to implement smart card authentication for an application meant to be deployed to restricted areas – but we didn’t have access to the smart card / public key infrastructure (PKI) that would allow us to test “real-life” use cases end to end.
As discussed below, self-signing certificates is a common practice when developing secure web applications to use one-way TLS (Transport Layer Security). Mutual auth is a less common use case however, and a lot of documentation tends to focus on implementation, the “how” of configuration. This blog post is for developers who want to understand more of the “what” and the “why.”
First, we’ll discuss mutual TLS, how it differs from one-way TLS, and why we’d want one over the other. Next, we’ll look at how to set up dev PKI that mocks out smart card infrastructure, with context about how our dev setup maps onto real-world organizational processes. Last, we’ll touch on some common approaches to configuring servers that enforce mutual auth.
What is Mutual TLS?
Most web devs are familiar with HTTPS – HTTP over TLS. HTTPS is generally one-way TLS. The client – often a web browser- makes a request to a web server on a secure port – often 443- to initiate a secure connection. During the following negotiation between the server and the client, the server presents a public server certificate, signed by a certificate authority (CA) that the client trusts.
These certificate authorities are usually big names like Let’s Encrypt, Amazon, GlobalSign, or VeriSign. The client will typically look to the host operating system to determine which CAs are trustworthy. Here’s the full list of CAs that OSX High Sierra trusts by default.
If the server certificate is trustworthy, the client will proceed forward with the dance. This is my favorite explanation for those who want to dig into all the steps.
Trust is configurable. If you decide that your system should trust a CA not in the operating system defaults, and you have sysadmin rights for the system, you can tell it to trust additional certificate authorities, such as one operated by your own organization. Banks do this, and so do militaries. One reason to run your own PKI is to cut costs, by cutting out the middleman. Another is to reduce attack surface, by cutting out the middleman.
Something you really do not want when you are operating a bank, for example, is for a certificate authority provisioning you certificates that say “yes, this web server is owned and operated by BigMonday Bank” to also then go ahead and hand out (accidentally or not) certificates with your name on them to sysadmins who are NOT associated with your bank. This is the WORST CASE SCENARIO but it does happen.
For similar reasons, developers don’t always have access to certificates signed by the CA that will be used in production. When they want to test their TLS configuration, they will use PKI to sign certificates themselves, and configure the systems hosting their dev servers to trust these certs when the system is running in development mode. Browsers now ship with various safeguards to prevent their users from interacting with self-signed certificates, but you can turn these off, which is helpful when testing.
So. That’s the one-way TLS story. You’re the operator of a website, you want the users of your website to trust it (via their OS trust configuration), and then once this trust is established, your users and your website will pass sensitive information back and forth via TLS. Why is it one-way? Because the client demands that the server prove who it is, but the server (during the TLS negotiation) does not require the client to prove anything about itself.
Mutual TLS adds a step: not only must the server present proof of identity to the client (“yes, I am owned and operated by BigMonday Bank, I will not steal your password and hand it to criminals”), but the client must provide proof of identity to the server, by providing the server a certificate signed by a CA that the server is configured to trust.
For end users, Mutual TLS is often called MFA or smart card auth. End users carry the certificate around on a smart card (something they have), which can be unlocked with a PIN (something they know). Think about withdrawing money from an ATM. How do you prove to the ATM that you own the account you are accessing? How does the bank prevent forgery of ATM cards? You put a card with a chip in it into the machine, and then type in your PIN. The PIN decrypts your personal client certificate stored on the ATM card, and the ATM checks that it’s a valid, trustworthy cert.
Web servers can be configured to do Mutual TLS. Server side, this is typically called client verification.
If, as a developer, you are tasked with developing a system that performs mutual TLS, you won’t always have access to the certificate signed by CA that will be used in production. There are various reasons for this, such as bureaucracy, or separation of duties concerns. If you’re in that situation, never fear! In development, you can self-sign the client certs too. Here’s how to do it.
Quick Reference
Here is a companion makefile for the discussion below.
# do this once - setup CA make create_ca make root_cert # do this for each client/user who needs a cert make private_key make csr make crt make bundle # do this for certificate revocation make crl make revoke_crt
Smart Card Story Time: What exactly are we mocking?
Please Note: in real-life/for production systems, most of the steps below are performed automatically by computers and scripts, not manually. I’m describing them as steps performed by people because like many, I connect more to stories about security where people are the principals. I’m not using the Alice and Bob convention because the world is not all Alices and Bobs.
First, we’ll start by creating a certificate authority. In a large organization, this would be the responsibility of an ops team with lots of security chops. They’ll need those chops to keep the CA keys secure. Creating the CA is easy!
While you execute the following commands from the makefile, imagine that you are a DevSecOps engineer named Amira:
$ make create_ca openssl genrsa -des3 -out ca.key 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ....+++++ ................+++++ e is 65537 (0x010001) Enter pass phrase for ca.key: Verifying - Enter pass phrase for ca.key: |
You have just generated a root key. This key will be used to create your root certificate. Anyone who has the root key and its password has the ability to generate client certificates your site will trust.
In production applications, this key and its passphrase must be stored in a secure place. Depending on the context, secure might mean “on computers that are the equivalent of a bank vault, maybe in a literal vault.” In our dev environment, since this is a CA intended for local use only, we can share the passphrase and key with others for convenience.
Now, Amira, create the root certificate. You can hit enter for most of the prompts, but it’s handy to name your root cert clearly so that later, when you tell your operating system to trust it, you can easily remember what the certificate was used for afterwards.
$ make root_cert openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.pem Enter pass phrase for ca.key: You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. ... |
Now you have a root certificate. It is called the “root” because when combined with your root key, it can be used to sign certificates.
The root cert is public. If you want to set up a server to do client verification, you’ll add this root cert to the list of certificate authorities that your server will accept. When a client presents a cert, it has to be signed by one of these CAs to be accepted.
Ok, you’ve been imagining yourself as Amira, right? CISSP, Masters degree in CS from Stanford, wondering how her security career turned out to be so bureaucratic? Stop doing that. Your name is now José, and you are a sysadmin onboarding a new employee at your Chicago office. You’re provisioning their smart card so they can use the office bathroom. The new employee has already provided you with a driver’s license, and after thirty tense minutes of debate, convinced you that they are who they say they are.
First, we want to generate a private key, unique for our new employee. You do this:
$ make private_key openssl genrsa -out dev.key 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ..+++++ .................+++++ e is 65537 (0x010001) |
Then you generate a certificate signing request. The CSR will use the key you just made to create a file that allows Amira to create a certificate that will be uniquely associated with the new employee sitting in front of you. The certificate that Amira gives you will store some metadata about the new employee, like their name, that’s defined below.
Here, set the common name to something that makes sense in your application. So if you’re provisioning a certificate to mock out Admin auth, it would make sense to call it “Admin User”:
$ make csr openssl req -new -key dev.key -out dev.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. ... ... Common Name (e.g. server FQDN or YOUR name) []:Admin User ... |
You, José, send the CSR to Amira. You’ve worked with Amira for many years, so while she didn’t check the new employees driver’s license herself, she trusts that you did. She uses the CSR, the CA, and the password to the root key to generate a signed certificate for your new employee. Amira also sets a password on the private key. Next, she creates a p12 file, bundling the cert and the key together. Then she sends the bundle back with the password to José.
(Note: I don’t actually know how the client key password is handled in real life smart card provisioning, fortunately this isn’t a guide about provisioning real smart cards. We will need to perform these steps in our mock setup, so let’s take the mental shortcut.)
$ make crt openssl x509 -req -in dev.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out dev.crt -days 825 -sha256 -extfile config.ext Signature ok subject=C = US, ST = IL, L = Chicago, O = Tandem, OU = Tandem, CN = ... Getting CA Private Key Enter pass phrase for ca.key: |
Great, now lets bundle them up, skipping the export password with two carriage returns
$ make bundle openssl pkcs12 -export -in dev.crt -inkey dev.key -out client.p12 -name "clientcert" Enter Export Password: Verifying - Enter Export Password: |
Now both you and José both know the key password for the new employee. That’s not good, that breaks non-repudiation. You really don’t want to be associated with this new employee’s activities in any way, in case they commit crimes and accuse you of stealing their smart card to do them.
Now we are back in the office, as José, sitting with the new hire. You hand them your keyboard and have them change the password to a PIN that only they know. Only after they’ve typed it will you let them know that the PIN is very important and they shouldn’t forget it. You, José, also find your job somewhat bureaucratic and are unimpressed with the lack of espionage.
You (José) store the bundle on the smart card using some magic. Then you give the smart card to the new employee, and remind them that they shouldn’t forget their PIN, but also they should not write it down anywhere. After they leave, you realize you forgot their name, so you inspect the cert, because it says all sorts of things about them, like their common name, where they work, and what the certificate can be used for:
$ openssl x509 -in dev.crt -text -noout Certificate: Data: Version: 3 (0x2) ... Signature Algorithm: sha256WithRSAEncryption Issuer: ... Validity Not Before: Jan 7 22:40:42 2021 GMT Not After : Apr 12 22:40:42 2023 GMT ... X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment ... |
Now you are Amira again. You’re meditating on your life and your choices when you realize that you’ve been provisioning all these certs, but you have no way of revoking them. Before anyone else notices, you create a certificate revocation list:
$ make crl openssl ca -config ca.conf -gencrl -keyfile ca.key -cert ca.pem -out ca.crl.pem && openssl crl -inform PEM -in ca.crl.pem -outform DER -out ca.crl Using configuration from ca.conf Enter pass phrase for ca.key: |
Now it is Tuesday of next week. You are José. HR just told you that the new employee has already quit. You call Amira to let her know. “Yes,” says Amira, “I am fully prepared for this and have been for some time.”
You are Amira. You revoke the certificate.
make revoke_crt |
In this example, certificate revocation checks are handled by a CRL file which is loaded onto the server. In practice, these lists get big, so you’ll use OSCP instead, which is a HTTP request to a web endpoint on port 80 that will check the revocation list. The OSCP endpoint will be listed on the client cert if it’s being used as part of the PKI scheme. The SSL library running on the web server will know how to do this: you just tell it where to go, or if it should use the info on the cert.
make crl
Now, if the former employee breaks into the office and tries to use the bathroom, they will be unable to access the toilet. They will swipe their smart card (which HR should have collected, but you know HR) and the bathroom server will check the expiration date (still good) and the certificate revocation list and see that the cert is no good around here anymore.
Congrats! That was the full lifecycle of authentication via PKI!
Server Configuration for Authentication
“But wait,” you (as you) say. This is all well and good, we have this mock smart card infrastructure, but we still do not have any authentication! How do I configure my web server to do Mutual TLS? Great question. It’s not the default setting.
Let’s say your web server is Apache. Here is a Very Minimal vhost configuration with only the parts related to TLS:
<VirtualHost *:443>
...
SSLCertificateFile "https://149865198.v2.pressablecdn.com/usr/local/apache2/ssl/certs/server.crt"
SSLCertificateKeyFile "https://149865198.v2.pressablecdn.com/usr/local/apache2/ssl/certs/server.key"
# Enable client verification
SSLVerifyClient require
# A list of public certs for certificate authorities that you trust.
# In this case, "trust" means: they are used to provision client certs
# for your users, and you got them from a trusted source.
SSLCACertificateFile /etc/httpd/ssl/certs/trusted_certificate_authorities.pem
# Enables certificate revocation checks
# Server will need to be able to access ocsp.yourorg.internet on port 80
SSLOCSPEnable on
SSLOCSPDefaultResponder http://ocsp.yourorg.internet
# If you are running as a reverse proxy,
# Uses mod_ssl to extract ID metadata and pass it to the backend
# via a request header, so the backend can use it to do authorization
# The full list of variables available on the cert are here:
# https://httpd.apache.org/docs/current/mod/mod_ssl.html#page-header
RequestHeader set X-USERNAME "%{SSL_CLIENT_S_DN_CN}s"
...
</VirtualHost>
|
If you’re in the Rails universe, you might be interested in rails-auth.
For Spring Boot Fans, this article might help!
Client Configuration
As mentioned above, your operating system will need to be configured to trust your local CA when you are using it in local dev, especially for end to end testing of a browser-based application. The steps vary from machine to machine, so we won’t get into it in detail here. At a high level though, you’ll probably need to do these things:
- Add the CA cert to your system keychain/keystore and manually trust it
- Add the client certs to your system keychain/keystore and manually trust them
- Configure your browser to be less protective and accept self-signed certs served by localhost
Conclusion
If this all seems really complicated, well, yes. Strong security protocols unfortunately come with complexity and usability tradeoffs. For a really excellent discussion of this protocol for authN and more, I highly recommend the 3rd edition of Ross Anderson’s Security Engineering. It is less focused on implementation details (and less focused on confusing use of the second person) but is the best text I’ve read for understanding the “why” of what we do what we do when we’re doing security.
Also! If your organization has either authN or authZ needs, reach out to Tandem! We’ve done it before, and we’re happy to do it again.
