Internal SSL Certs with Let's Encrypt

Matt Pogue -

I was reading a great article recently about subdomain enumeration services and it got me thinking about Let's Encrypt and internal domains. It's common to run a split-domain setup where there's a single domain for both Internet-available services and Active Directory, with either yourself or someone else providing the external DNS resolution and your domain controllers providing the internal DNS resolution. I've run a split-domain setup since I began working with Active Directory (AD); it makes it easier on users and administrators both.

Note: Throughout the article when I refer to "internal DNS", I'm referring to the DNS zones configured on your Active Directory domain controllers that are only accessible from your LAN and are not available from the Internet. For the purposes of this article I'm assuming that you're running a split-DNS configuration, with one copy of your DNS zone available to the Internet and second, separate, zone available only to users on your LAN. In my opinion, this setup is preferred over using a non-existent top-level domain.

In the early days of Active Directory, there was no free source of externally-signed certificates. Some of the commercial Certificate Authorities - Network Associates, Comodo, etc. - charged $100+ annually for a single cert and required physical confirmation of business ownership before issuance. Some issued wildcard certificates (sometimes requiring additional verification) and typically charged extra for it. As a result, internal websites either stayed HTTP-only or got their own copy of the wildcard cert if one was purchased. In smaller environments (like the ones most of us solo admins inhabit) the issue was limited in scope, but in larger environments not so much. A real world problem I'd run into in several larger organizations with wildcard certificates was that every team in IT seemed to have a copy of the cert and the private key. It was used to secure sites on the Internet, extranets, intranets, development servers, developer workstations, test apps, email services, the list goes on. And once a year (or every 2 - 3 years) when this certificate expired the new certificate (plus private key!) got emailed around the company more than directions to the Christmas party. And if there became a reason to revoke or reissue the certificate the process was repeated. There was absolutely no security around the cert, and most importantly the private key. Because it was deployed on so many insecure systems, the attack surface became a lush expanse, ripe for the picking.

The flip side of wildcard certificates in the early days were privately-signed (but valid) certificates issued from an Active Directory Certificate Authority (CA) running Active Directory Certificate Services (AD CS). It was relatively easy to set up and on a corporate LAN root certificate trust was easy to manage. AD CS also offered identity management through AD, lifecycle management, client authentication, and centralized management through Group Policy. The only catch? If your machines weren't Windows, weren't domain members, or weren't able to communicate with AD, you were back to square one.

For the solo admin, the AD CS infrastructure was overkill; all most of us really needed were valid SSL certificates for our internal and external domains. The wildcard cert option was not a horrible solution, but for the reasons mentioned above I'm a bit leery of issuing wildcard certs, especially for the protection of both internal and external - and development and production - systems. And spending $500 or $600/year buying certificates isn't going to break the bank, but if you're really on a shoestring budget protecting development sites isn't going to be high on your list. Enter Let's Encrypt.

For anyone not familiar, Let's Encrypt is a non-profit certificate authority that was founded in 2013 by the Internet Security Research Group whose founding sponsors included Mozilla, the EFF, Cisco, and Akamai among others. With Let's Encrypt we can now obtain valid and trusted SSL certificates for free, and with this capability, now is the time to go all-SSL for both internal and external sites. While there are other ways to obtain certificates, my preferred method is to use the letsencrypt command on Linux and export PKCS12 format if needed on Windows. Here's what you'll need:

  1. Register any DNS names that you're generating certificates for and the entries need to point to public IP addresses that are accessible on TCP/443. However, the port only needs to be available while you're requesting the certificate; it can be closed otherwise.
  2. You'll need a Linux system with nothing running on port 443 (or where you can temporarily stop the service using the port).
  3. Finally, you'll need the letsencrypt command; on Debian/Ubuntu it's as simple as apt-get install letsencrypt.

The way the process works, the Let's Encrypt servers simply look up the domain name for which you're requesting a certificate and send a request to the IP address on port 443. Using the letsencrypt app on Linux, a listener is started on TCP/443 to respond to the request. If all goes as planned the request is approved and the certificate, private key, and intermediate chain are saved to your system and ready for use. I'll walk through the process using this site - thesoloadmin.com - as an example.

  1. Assuming letsencrypt is already installed, I'll stop the webserver temporarily to request the cert: systemctl stop nginx
  2. To renew all certificates located under /etc/letsencrypt/live you would use the flag renew. However, since I only want to renew a single certificate, I'll be using the certonly flag (you would also use the certonly flag for a new certificate request). You'll want to use the directory name under /etc/letsencrypt/live/ as the domain name to renew; e.g. mine is thesoloadmin.com, although the certificate contains alternate names for thesoloadmin.blog, thesoloadmin.org, thesoloadmin.info, and thesoloadmin.net. The command to renew a single certificate is simply: letsencrypt certonly -d thesoloadmin.com.  You'll be prompted to either start a temporary webserver or place files in webroot directory; I always choose the temporary webserver option because it's the easiest. The output from the command will be similar to the following:
root@server [~]# letsencrypt certonly -d thesoloadmin.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for thesoloadmin.com
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/thesoloadmin.com-0001/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/thesoloadmin.com-0001/privkey.pem
   Your cert will expire on 2021-06-14. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

After the command finishes, you can then start your webserver again. You can view your new certificate using the openssl command:

root@server [~]# openssl x509 -in /etc/letsencrypt/live/thesoloadmin.com/cert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:3f:ec:10:11:d2:f6:8b:7a:7f:4a:13:1b:91:34:8b:fe:a7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Dec 29 19:01:03 2020 GMT
            Not After : Mar 29 19:01:03 2021 GMT
        Subject: CN = thesoloadmin.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:af:a3:48:a7:9c:a5:78:4c:81:9c:f4:59:22:82:
                    e2:a6:cd:3b:24:63:7b:70:69:aa:e1:7f:26:d8:8e:
                    99:b7:20:57:29:2c:db:ea:c5:c9:bd:25:11:e7:37:
                    68:3f:7d:6c:d9:5d:d8:55:72:50:cb:d0:ab:87:d7:
                    37:0d:d1:a2:c8:65:db:dc:e7:1f:0d:a0:9f:aa:21:
                    ef:99:9e:6d:6d:0c:57:c6:2a:98:8c:9e:58:34:ea:
                    91:63:ff:86:89:c6:58:1a:f0:8f:88:cb:c1:08:45:
                    e1:da:42:a9:a5:5c:6c:10:64:e9:3d:43:47:0c:cf:
                    e2:03:44:42:1a:3f:aa:7c:0f:84:1d:c7:ec:9c:62:
                    69:e0:3d:3d:a4:7a:e5:fd:70:e4:14:a8:c4:c1:98:
                    17:78:bf:98:22:b3:2a:15:47:5a:98:f2:1d:9f:9a:
                    5c:b6:29:7b:43:b2:aa:4d:d2:09:28:cc:e8:d0:77:
                    10:fa:d5:75:93:00:6f:5b:77:0c:e1:99:37:03:7f:
                    78:27:fb:76:70:09:36:64:4a:fc:c0:b8:39:39:29:
                    70:72:34:5b:53:e4:34:9c:10:63:d2:ea:96:26:94:
                    fc:bc:3b:c2:26:1b:68:26:ff:2a:21:ae:8d:00:b6:
                    71:b4:6f:5b:7a:af:f8:db:7a:1c:e6:5c:8f:55:47:
                    ca:d5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                5A:FD:71:F6:63:73:34:62:5D:0D:CC:08:5E:67:96:A3:85:92:59:75
            X509v3 Authority Key Identifier:
                keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6

            Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

            X509v3 Subject Alternative Name:
                DNS:soloadmin.org, DNS:thesoloadmin.blog, DNS:thesoloadmin.com, DNS:thesoloadmin.info, DNS:thesoloadmin.net, DNS:thesoloadmin.org, DNS:thesoloadmin.us
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 44:94:65:2E:B0:EE:CE:AF:C4:40:07:D8:A8:FE:28:C0:
                                DA:E6:82:BE:D8:CB:31:B5:3F:D3:33:96:B5:B6:81:A8
                    Timestamp : Dec 29 20:01:04.006 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:1F:B4:34:11:3A:8C:AD:E6:27:E1:8C:7E:
                                95:A9:CE:A5:80:86:B8:F2:93:20:C1:BC:64:EC:DE:46:
                                7E:5F:14:56:02:21:00:B9:39:E1:77:8F:49:E3:85:E2:
                                93:5A:84:24:C6:8F:39:2F:BC:30:04:E7:73:22:73:11:
                                32:E2:44:5E:0D:92:AF
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : F6:5C:94:2F:D1:77:30:22:14:54:18:08:30:94:56:8E:
                                E3:4D:13:19:33:BF:DF:0C:2F:20:0B:CC:4E:F1:64:E3
                    Timestamp : Dec 29 20:01:03.984 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:9A:78:46:0B:13:0E:60:6C:B4:B4:0C:
                                C7:2E:D3:BE:DB:39:13:94:8B:5D:A4:76:D4:9E:D4:85:
                                17:4B:EE:24:1A:02:21:00:C0:75:99:19:6D:CA:F5:1F:
                                B7:14:7B:E0:30:2C:D7:0C:6F:60:CD:89:DD:7A:18:0F:
                                CA:C1:51:D7:FE:48:F9:78
    Signature Algorithm: sha256WithRSAEncryption
         0a:27:63:6c:ea:b7:a5:47:76:51:a5:71:02:90:bf:69:80:2d:
         e0:5c:30:96:00:24:8f:94:dc:1e:d7:2f:e2:35:8e:30:10:72:
         e1:b8:01:c0:d6:b7:ab:c6:b6:82:89:d8:96:50:8b:bb:1e:01:
         a7:e8:41:dd:02:a5:dd:8f:c2:31:a4:c4:4e:0d:f6:d7:6a:6b:
         1e:2a:51:0d:15:87:85:ef:7e:38:4b:21:ee:a3:08:49:74:91:
         fe:be:e7:45:4b:7f:33:2b:5d:68:21:d2:d6:44:71:eb:d3:87:
         72:e5:92:9f:93:6a:11:17:eb:30:6d:86:7d:7b:c0:45:31:ac:
         b7:23:98:0f:1b:e1:61:c3:8d:bd:c4:4a:fe:b9:94:33:45:c8:
         71:9d:d2:81:d6:9d:d6:f1:e4:4f:c6:7a:a4:fe:90:5d:f2:33:
         33:20:32:cf:2b:59:fe:90:18:1f:75:9d:40:4e:7a:c8:86:cf:
         37:9c:cb:6e:bf:06:25:de:22:1c:56:d0:83:54:8d:23:60:03:
         15:68:60:10:32:ca:2a:87:f5:6e:7c:27:68:6e:96:05:99:d6:
         04:2e:10:89:65:63:fe:1b:cc:3b:b3:b4:00:7b:6d:03:57:e5:
         51:1a:b2:41:1a:f8:82:ec:0f:b3:9e:c6:a7:49:16:9c:92:76:
         d5:03:4b:9b

You can issue certificates for any combination of internal domains (up to a maximum of 10 per certificate) by adding multiple "-d" switches. For instance, you could combine all your development sites into a single cert, all production sites, all utility sites, etc. - whatever makes sense in your environment. Personally, I like having multiple sub-domain certs (as opposed to a single wildcard cert) for the reasons discussed above. Once you've obtained the certs you simply move them to the internal servers where they're needed. You can also export to the PKCS12 format for use on Windows, again using the openssl command: openssl pkcs12 -export -out mydomain.pfx -inkey privkey.pem -in cert.pem -certfile chain.pem. You will be prompted for a password to protect the private key contained within the output file that you will use when importing to Windows.

In a nutshell, this is by far the easiest (and cheapest) way to enable SSL for all your internal and external web sites. Although the letsencrypt package will enable automatic renewal, I personally prefer manual renewal and disable the automatic renewal via cron. In 2021, there's no excuse for not enabling SSL for all sites under your control and getting your users acclimated to seeing the lock icon everywhere while getting them out of the habit of accepting certificate warnings.