In our day-to-day lives as solo admins, there are times when problems are solved with duct tape and baling wire. And there's nothing wrong with that! However, we all know that duct tape works right up until it doesn't, so it's best to revisit those issues with a permanent solution as quickly as possible.
I faced such an issue recently concerning my Active Directory (AD) domain controllers. Even in a small network, having all domain controller roles and required AD services (DNS, DHCP, Global Catalog, etc.) on a single server makes me very nervous; it's always best to have a second server to both share the load and provide some redundancy. When I began working for my current employer, my counterpart who had been in the job previously was running the domain with one domain controller providing not only all the AD services mentioned above, but also acting as the primary file and print server without any type of offline backup solution in place. Obviously NOT a good situation! After immediately implementing a backup solution (using the wonderful BackupAssist product, which I'll discuss in a later post), I set out to build a second server to provide some redundancy.
First some background on the setup:
- Hardware-wise, the existing domain controller machine was running as a guest on a Proxmox Virtual Environment (PVE) host server. I'm a BIG fan of Proxmox, so no complaints there.
- The physical PVE host was a relatively new Dell Precision workstation (much like Pinocchio, not a real server) with an 8-core Xeon E5-1620 (meh), 32 GB RAM (meh again) and 2x2 TB SATA drives configured as a ZFS mirrored pool (workable for our small environment).
- The domain controller guest was Windows Server 2016 and was assigned 16 GB of RAM and 800 GB disk space.
- My predecessor had also left behind a second Dell Precision configured identically to the previous machine, but unused.
Given that I had a second physical machine that was unused, I installed the latest version of Proxmox and began to prep for my Windows Server guest installation which is where I ran into my first problem - licensing. More background - when I came into my current role, both server and workstation licensing was an absolute mess. For whatever reason, my predecessor seemed to be fond of purchasing used hardware (with no included license) and downloading from Pirate Bay rather than purchasing legitimate Windows licenses. After alerting the company owner to the situation, I found a partner from whom we could purchase Windows licenses. However, the existing Windows servers (including the DC) had been cracked to activate the server and I didn't feel that I could trust it to be the backbone of our network. At that point I decided to build two new DC's and a new file and print server and decommission the existing servers. Additionally, all the new servers would run Windows Server 2019.
Shiny new Windows Server license in hand, I began my first build. I'm assuming if you've gotten this far you're able to install Windows Server so I'll skip the installation walkthrough. After installing all available Windows updates and joining the domain, I promoted the server to domain controller (the Pete Net Live website has a nice walkthrough of this process here). After promotion is a good time to take a coffee break and let everything catch up - DNS zone transfers to the new server, etc. After the coffee break, I decided to run dcdiag to validate that everything was functioning normally at the domain level (for details on the dcdiag tool, see https://activedirectorypro.com/dcdiag-check-domain-controller-health/). Everything passed, so at this point, I built the second server, repeating the same steps and promoting to DC upon completion.
Note - After completing the second server build, I used dig for Windows (see here) to ensure that the DNS zone transfers have completed successfully and my zone(s) have been replicated to both of the new DC's.
One critical "set it and forget it" network service that often gets overlooked is DHCP. Especially in smaller networks, the domain controller is an ideal place to manage your DHCP scopes. However, it's easy to forget the service is running until after you shut down the old server and no one can get an IP! There are two different methods that I've used in a small network environment and either one will work:
- First, you can configure the scope on the new server, including enabling conflict detection (this is important!), authorize it, and then deauthorize the old server. This is the easiest method. As your client IP's expire, they will pick up an address from the new DC automatically and conflict detection prevents duplicate addresses from being assigned.
- The second method involves assigning an adjacent range of addresses - for instance, if you have the range .100 - .150 alotted to your existing scope, simply use .151 - .200 for the new scope. However, if you're like me and you're somewhat OCD about these things, then the first method may be your best bet.
At this point, we have 3 fully functional domain controllers. However, the 2016 server has the Flexible Single Master Operations (FSMO) roles so we need to remedy that before we begin the decommissioning process. The folks at The Sysadmin Channel website provide a good overview of what each of the FSMO roles are along with instructions on how to move them using Powershell here. I agree with their recommendation to divide the roles between the 2 new servers; which roles go to which server doesn't make much difference in our environment, but it can become a major deal in a multi-site large AD installation. Additionally, I like to verify visually that each of my domain controllers is also a global catalog server. The easiest way to do this:
- Open the Active Directory Users and Computers MMC console.
- Navigate to the Domain Controllers OU under your domain.
- Right-click each domain controller and choose Properties.
- Ensure that the "DC Type:" box says "Global Catalog", then click the NTDS Settings button and make sure the Global Catalog box is checked.
Now that we have both new DC's built and functioning, and we've used dcdiag against both new DC's to ensure that all tests pass, we can move on to decommission the 2016 server. However, there is one more optional step, at least in my case, that can (and should) be performed and that's to raise the domain functional level.
The domain functional level determines which version of the AD schema is in use and controls some of the available capabilities (more info here). Note, the functional level also determines which domain controller operating systems will be supported (this does not apply to member servers and workstations). In my case, for whatever reason, my predecessor was still running at the "Windows Server 2008 R2" functional level. I'm going to raise the functional level to Windows Server 2016, the latest functional level change (the 2016 functional level supports all new server versions from 2016 through Windows Server 2019 and 2022).
To check - and change - your domain's functional level, go back to the Active Directory Users and Computers MMC console, right-click on your domain name, and choose "Raise Domain Functional Level". You'll be prompted with a pop-up:
Please Note: Do NOT perform this step without having a working and tested backup of your Active Directory forest. See this document from Microsoft - the AD Forest Recovery Guide - for instructions on developing a forest recovery plan which includes backups. You've implemented a DR plan that includes something similar to this already, right?
Ok, now that we've completed our DC builds, moved our FSMO roles, and ensured that our new servers are both Global Catalogs, we're finally ready to decommission the old DC. The first step in the process is to demote the server to a simple domain member server. Microsoft has a detailed article on demoting a domain controller here and our friends at The Sysadmin Channel have a shorter checklist here. Their article assumes that the DC is running at least Windows Server 2012 (and if you're still running Windows Server 2008 or - God forbid - Windows Server 2000, email me for a consulting services quote!) and includes the Powershell command to remove AD DS services which the Microsoft article is less clear on. After demoting the server, you'll want to clean up the domain controller's metadata still remaining throughout DNS, etc. Microsoft provides a good set of instructions on removing the metadata here.
After the server has been demoted to a member server, you can then follow your standard server decommissioning process. My procedure is fairly simple - backup any server data (if applicable) and wipe the hard drives (bare metal) or simply delete the virtual disk files (for a guest VM). Being somewhat paranoid when it comes to servers, I tend to keep the server's hard drives or the virtual disk files for a few months just in case. I don't think I've ever used them, but you never know.
That's the basic process for putting that old domain controller out to pasture. Since I was going through the process, I thought a quick writeup was in order. As always, feel free to email me with questions or if you run into a snag going through the process in your own environment and I'd be happy to help out if I can.