# The Solo Admin > Welcome, fellow weary traveler! This is The Solo Admin blog - the blog for IT professionals who are flying solo or an indispensable member of a small team. Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages ### About Membership URL: https://thesoloadmin.com/about-membership/ Last updated: 2021-02-11T17:40:32.000Z UPDATE: Membership is now open at all levels - free preview, $6/monthly, $60/annual. [Click to sign up](https://thesoloadmin.com/#/portal/signup). The Solo Admin will have a members-only area launching very soon. In the membership section you will find long-form articles about network design and troubleshooting, server build and maintenance, backups, disaster recovery, security, and more - all from the standpoint of the solo IT pro on a budget. The content will be divided into sections with topics that build on each other - similar to an online or college course. In addition to the articles, I will also be offering live and interactive video sessions where members are able to ask questions as we walk through various topics - building servers, installing and configuring services, troubleshooting, etc. Members will also have access to a members-only Git repository containing templates, source code, and more, related to the topics we cover. My goal is to teach others how to design, build, and maintain a small network with all the capabilities of a large enterprise. The first section will be "Network Design - From the Ground Up" with the following topics: - Designing the Network - Designing Your IP Address Scheme - Selecting Network Hardware - Securing the Network - Foundational Structure - VLAN's, Subnetting, etc. - Laying the Foundation - Basic Datacenter Layout - Setting Up Network Hardware - Switches, Routers, Firewalls - Implementing Multiple Physical Sites - VPN's and Routing - Network Testing and Troubleshooting - Designing and Building Servers - Server Hardware Selection - Server Builds - Virtual and Bare Metal - Implementing Services - DHCP, DNS, etc. - Implementing Backups and Basic Disaster Recovery (DR) - Developing a Maintenance Plan - Implementing Server and Network Monitoring - Finishing the Build - Securing Active Directory via Group Policy - Implementing Office 365 - Creating and Securing User Accounts Finally, as a member, you will have a voice in the direction and the creative process of the blog. You can contact me any time via email - [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com) \- to provide feedback, request new topics, or ask questions about existing topics. My ultimate goal is to work together with the members to grow the content based on their feedback to provide real value. If you're interested in becoming a member, [click here to sign up](https://thesoloadmin.com/#/portal/signup). ### Members-Only Pages URL: https://thesoloadmin.com/members-only-pages/ Last updated: 2021-02-17T15:41:39.000Z The following list contains links to all Members-Only pages of the blog: - Network Design From the Ground Up - [Asking Questions](https://thesoloadmin.com/asking-questions/) ### My Resume URL: https://thesoloadmin.com/my-resume/ Last updated: 2022-06-28T18:46:02.000Z I'm currently working on an interactive version of my resume so in the interim, here's the latest PDF version for download. [ Matt Pogue Chronological Resume MASTER Matt Pogue Chronological Resume MASTER.pdf 237 KB download-circle ](https://thesoloadmin.com/content/files/2022/06/Matt-Pogue-Chronological-Resume-MASTER-2.pdf "Download") ## Posts ### Tech Note: pfSense Upgrades URL: https://thesoloadmin.com/tech-note-pfsense-upgrades/ Last updated: 2025-08-25T16:27:00.000Z If you run [pfSense](https://www.pfsense.org/?ref=thesoloadmin.com) on [Netgate](https://netgate.com/?ref=thesoloadmin.com) appliances, this Tech Note is for you. pfSense on Netgate appliances (locally; not cloud-based) has been my firewall of choice for nearly 15 years. The devices are rock solid with the security of [FreeBSD](https://www.freebsd.org/?ref=thesoloadmin.com) to boot. However, the devices use ZFS, which can be confusing at times. I have a [Netgate 2100](https://www.netgate.com/pfsense-plus-software/how-to-buy?ref=thesoloadmin.com#2100) at one of my remote sites, serving as a firewall/Wireguard client connection back to the main office. The 2100 is ideal in this role, providing all the power you need for a remote client site with 4 GB RAM, and an 8 GB MMC disk. While Netgate creates separate ZFS filesystems for `/var` (and several subdirectories), `/tmp`, and `/home`, in reality these are all on the same physical disk, which you'll note when viewing the output from the `df` command: ``` Filesystem Size Used Avail Capacity Mounted on pfSense/ROOT/default_20250825095103 4.7G 1.0G 3.6G 22% / devfs 1.0K 0B 1.0K 0% /dev pfSense/var 3.6G 3.2M 3.6G 0% /var pfSense/tmp 3.6G 228K 3.6G 0% /tmp pfSense/home 3.6G 96K 3.6G 0% /home pfSense/var/log 3.6G 3.1M 3.6G 0% /var/log pfSense/var/cache 3.6G 96K 3.6G 0% /var/cache pfSense/var/db 3.6G 2.5M 3.6G 0% /var/db pfSense/var/tmp 3.6G 104K 3.6G 0% /var/tmp pfSense/ROOT/default_20250825095103/cf 3.6G 1.3M 3.6G 0% /cf pfSense/ROOT/default_20250825095103/var_cache_pkg 3.9G 321M 3.6G 8% /var/cache/pkg pfSense/ROOT/default_20250825095103/var_db_pkg 3.6G 7.5M 3.6G 0% /var/db/pkg tmpfs 4.0M 140K 3.9M 3% /var/run ``` Wait, something's fishy here! I'm *supposed* to have 8 GB of disk space. Why do I only have 4.7? `pfSense/ROOT/default_20250825095103 4.7G 1.0G 3.6G 22% /` Prior to the upgrade, the situation was even worse, with `df` reporting only 1.8 GB of disk space total! What gives? Smartly, Netgate appliances running their pfSense+ software implement what are called "[boot environments](https://docs.netgate.com/pfsense/en/latest/backup/zfsbe/space.html?ref=thesoloadmin.com)" - ZFS snapshots taken before each upgrade. You can view the saved boot environments from the "System > Boot Environments" menu in the pfSense+ web GUI, however, the space occupied by each is misleading; for example, on the web page, it shows the snapshot from my previous upgrade as occupying 456 kb of space. In order to see the true amount of used space, you will need to view the snapshots from the command line with the command `zfs list -t snapshot`: ``` [25.07.1-RELEASE][root@wfmfw004.wwf-mgt.com]/root: zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT pfSense/ROOT/default_20250825095103@2025-08-25-09:51:12-0 308K - 1.04G - pfSense/ROOT/default_20250825095103@2025-08-25-10:11:12-0 316K - 1.04G - pfSense/ROOT/default_20250825095103/cf@2025-08-25-09:51:12-0 140K - 6.24M - pfSense/ROOT/default_20250825095103/cf@2025-08-25-10:11:12-0 140K - 6.24M - pfSense/ROOT/default_20250825095103/var_cache_pkg@2025-08-25-09:51:12-0 137M - 137M - pfSense/ROOT/default_20250825095103/var_cache_pkg@2025-08-25-10:11:12-0 88K - 120K - pfSense/ROOT/default_20250825095103/var_db_pkg@2025-08-25-09:51:12-0 1.02M - 7.45M - pfSense/ROOT/default_20250825095103/var_db_pkg@2025-08-25-10:11:12-0 1.01M - 7.44M - ``` Note the `REFER` column, which shows my previous snapshot, taken on 8/25/2025 at 09:51:12 as occupying 1.04 GB. Prior to cleaning up these snapshots, I had 4 of them from previous upgrades, occupying over 85% of my available space! Even sneakier, Zabbix was none the wiser, as my firewalls weren't reporting 8 GB of total space; as each snapshot was taken, my "total" disk size decreased! The solution? In the Boot Environments page in the pfSense+ web GUI, delete all boot environments other than the current running environment. After deletion, the `df` command will immediately reflect the increase in available disk space for the root filesystem. At this point, you can proceed with your upgrade. **Note: for upgrading pfSense+, I would recommend *at least* 1.5 GB of free space, otherwise, the upgrade will fail with an "out of space" error. After the failure, you will need to manually remove the downloaded packages from `/var/cache/pkg` or your risk an inaccessible device!** After freeing up the necessary space, I decided to upgrade from the shell via the `pfSense-upgrade -d` command. Be aware - your firewall will automatically reboot once the upgrade is complete. For all its quirks, I remain a staunch Netgate-pfSense guy (though the pfSense fork [OPNSense](https://opnsense.org/?ref=thesoloadmin.com) is a good solution as well). I want a firewall that works even if the Internet is down and that doesn't require an "always on" cloud connection in order to function. Netgate support has always been extremely knowledgeable and responsive anytime I've needed them, and I've had more than one appliance that lasted a decade+. If you're a pfSense user and you're in a jam, feel free to reach out and I'd be happy to help if I can - [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com?Subject=pfSense%20Help) ### Zabbix Server Dashboard: New CSS Challenge URL: https://thesoloadmin.com/zabbix-server-dashboard-new-css-challenge/ Last updated: 2025-06-21T15:46:19.000Z If you're a Zabbix user, unless you're SERIOUSLY behind on software updates, you're aware by now that a change in v7.0 reoriented the dashboard labels from horizontal to vertical. ![Zabbix Screenshot - Vertical Tags](https://thesoloadmin.com/content/images/2024/06/screen1.png) Zabbix Screenshot - Vertical Tags It bugged me enough that I created a [post](https://thesoloadmin.com/quick-note-zabbix-dashboard-css/) with instructions on how to change it. However, Zabbix threw me a curve in v7.2, moving from defined styles to Javascript components. It took some experimentation, but here's how to revert back to horizontal menus in Zabbix v7.2: 1. SSH to your Zabbix server system. 2. Change to the Zabbix `js` components directory: 1. `cd /usr/share/zabbix/ui/js` 3. Make a backup copy of `component.z-vertical.js`: 1. `cp component.z-vertical.js /root/zabbix_component.z-vertical.js_backup` 4. Edit `component.z-vertical.js` with the editor of your choice (vim, of course!): 1. Search for the line that reads `transform: 'rotate(270deg)'` and comment it out: 1. `//transform: 'rotate(270deg)'` 2. As of Zabbix 7.2.7, this will be line #45. 2. Remove the comma at the end of the previous line, #44; it should read: 1. `left: 0` 3. Comment out lines #88 and #89: 1. NOTE: Replace the single quotes in the lines below with a backtick character; backticks indicate code formatting in Markdown and as such, aren't displayed correctly. 2. `//this.style.width = '${this.#inner_container.scrollHeight}px;'` 3. [//this.style.height](https://thesoloadmin.com/zabbix-server-dashboard-new-css-challenge/)` = '${this.#inner_container.scrollWidth}px;'` 4. Copy and paste the commented lines, swapping the variables: 1. `this.style.width = '${this.#inner_container.scrollWidth}px;'` 2. `this.style.height = '${this.#inner_container.scrollHeight}px;'` 5. Save the file and restart Zabbix server: 1. `:wq` 2. `systemctl restart zabbix-server` 6. Double-check Zabbix's status: 1. `systemctl status zabbix-server` ![](https://thesoloadmin.com/content/images/2024/06/screen2.png) Zabbix server status from systemctl 1. Reload the Zabbix server dashboard in your browser. Your tags should now be restored to their former horizontal glory: ![](https://thesoloadmin.com/content/images/2024/06/screen3.png) Zabbix dashboard with horizontal tabs restored. Like the previous Zabbix fix, this one is pretty minor, and a relatively easy fix to boot, but hopefully some of my fellow Zabbix admins will find it useful. As always, if you have any questions, don't hesitate to reach out: [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com). ### Active Directory Authentication on Linux - A Simple Guide URL: https://thesoloadmin.com/active-directory-authentication-on-linux/ Last updated: 2025-02-24T18:03:36.000Z Continuing my previous [post](https://thesoloadmin.com/linux-server-build-walkthrough/) walking through a Linux server install with an eye toward creating a standard build, this post contains a set of easy steps to configure Active Directory (AD) authentication on Linux for your standard user accounts. This post makes a great companion piece because it's the first thing I do after completing my Linux builds. Let's get right to it, shall we? > Note: this post is specific to Debian and derivatives (verified working on Ubuntu 24.04 also), but should be fairly easily adapted to other Linux distros. ## Step 1 - Create a Backup After completing your initial server build and installing the base packages, it's a great time to create a backup/snapshot, if you haven't already. ## Step 2 - Prerequisites In order to complete these steps, you'll need to ensure a few prerequisites are in place: 1. Pretty obviously, a working Active Directory domain; you'll need to know both hostname and IP address for your AD DNS server and at least one domain controller. 2. Your Linux server, configured with a static IP address, a hostname that's unique among your domain-joined computers, configured for DNS resolution against your AD DNS server. 1. On the Linux server, we'll enable `root` to login via SSH temporarily for troubleshooting/configuration purposes. 3. Create an AD group to contain the user accounts permitted to authenticate to the Linux servers. 1. (Optional) Create an AD account with permission to query AD for troubleshooting, both now and for future tasks; see my [previous article](https://thesoloadmin.com/searching-active-directory-with-ldapsearch/) about searching AD with Linux using `ldapsearch` for reference. 4. An AD account with permission to query AD, for testing/troubleshooting purposes. 5. I've uploaded sample copies of all the files referenced in this post to the Solo Admin [Github repository](https://github.com/mattpogue/thesoloadmin.com?ref=thesoloadmin.com), under the "posts/active-directory-authentication-on-linux" folder. Feel free to grab them and customize them for your environment. 1. For the purposes of this post, the AD domain will be `domain.com`, the Linux server will be called `server.domain.com`, and the AD domain controller will be called `dc.domain.com`. 2. **Do NOT keep these default values! They must be updated for your environment.** Now that the prerequisites are in place, we'll be following the steps at [https://wiki.debian.org/AuthenticatingLinuxWithActiveDirectorySssd](https://wiki.debian.org/AuthenticatingLinuxWithActiveDirectorySssd?ref=thesoloadmin.com) rather than reinvent the wheel with an identical walkthrough. Here are some notes on the process: 1. Make sure and explicitly update `/etc/hosts`, modifying the entry with your server's name to reflect its IP, instead of `127.0.0.1`. Ping by name to verify that it no longer resolves to `127.0.0.1`. ``` 192.168.0.1 server.domain.com server ``` 1. In my environment, I've replaced the `ntp` package with `ntpsec`. The `ntp.conf` file in the repo works identically for either. 2. When changing ownership after the first successful login, I've always used Domain Users for my group designation, e.g. ``` chown -R :"Domain Users" /home//.ssh ``` As I mentioned previously, having an account for AD integration can come in handy at any step of this process when trying to troubleshoot, and the excellent `ldapsearch` Linux utility is your friend. **NOTE* \-* this guide, and the linked Debian HOW-TO page, are for authentication *only*, and not SAMBA file servers/domain controllers, etc. Your server, by virtue of the final `join-ad.sh` script, will have a computer account in the domain, but it will be limited. As always, if you're stuck and would like help, shoot an email to [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com?Subject:Linux%20AD%20Auth%20Article) and I'll do what I can. 😎 ### Linux Server Build Walkthrough URL: https://thesoloadmin.com/linux-server-build-walkthrough/ Last updated: 2025-11-05T18:49:29.000Z Hello fellow admins! It's been a while, but I'm back in the saddle to discuss a foundational tool that should be in every admin's toolbox - the standard server build. My hope is that you can use this article as the basis for a standard build of your own, whether in the lab or in Production. First, a caveat - this article will cover a standard Linux build using Debian, intended for use as a server. That being said, the process could be easily modified for different Linux distributions or even Windows. If you're like me and you don't build servers enough to spend time on a standard server image, the important part is to develop a process that's easily repeatable. Let's get to it! ## **Step 1 - Configuring the Hardware** For the purpose of this walkthrough, we'll be installing Debian 12.8 on a guest virtual machine running atop Proxmox 8.2, however, the process can be applied to bare metal just as easily. Debian Linux in 2024 can handle most any hardware change you throw its way, unless it's something really exotic. However, you'll obviously want to carefully plan your disk and filesystem layout ahead of time, as a change to either usually means a rebuild; rather than letting the installer configure it automatically, I always choose manual filesystem layout. While there are about a million different ways to lay out your filesystems, there's no one right way to do it (and just as many *wrong* ways to do it). Here's what I've settled on after years of tinkering: - `/boot` \- 512 MB, ext4 filesystem - `swap` \- 1024 MB - `/` \- remaining space, ext4 filesystem, configured as an LVM logical volume on top of a single LVM volume group comprising the remaining free space. This allows for flexibility down the road. I would also recommend finalizing the hardware to the extent possible before beginning the install, as Linux has a habit of changing hardware IDs when new hardware is added; e.g. `eth0` might become `eth1` with the addition of a new network card. Also, I don't cover it here because I don't use it frequently, but Linux does support TPM and secure boot and that's definitely a decision to make before you start installing. You can *technically* change it later, but it won't be easy. Now that your hardware is prepped, let's start the install process. ## Step 2 - Installation Media When choosing your installation media from the various downloads available, you're always, always, *always* better off to start small and add on vs. starting with an "everything but the kitchen sink" install image and removing packages later (or even worse, leaving them all intact). In Debian's case, my preferred install media is the one labeled "64-bit PC netinst iso" on [the Debian website](https://www.debian.org/distrib/?ref=thesoloadmin.com). Grab the media and prep it using [Rufus](https://rufus.ie/en/?ref=thesoloadmin.com) or the install media creator of your choice. ## Step 3 - The Install I'm not going to screenshot the entire install process, but I will leave you with a few tips: - As mentioned previously, choose "Manual" partitioning, and then partition your disk(s) using the scheme you decided on in Step 1. - When creating your non-root user account, if you're planning to create a build image and/or configure external authentication, your user account should be something generic, as you'll only use it a few times. Especially don't use an account name you plan to use for external auth, as this will introduce extra steps into the process. - When choosing a Debian mirror, I usually stick with the default Debian.org mirrors. This was not always the case, but I've had issues with reliability and availability of some of the others recently. - **Most Important:** when prompted which packages to install, uncheck everything except "SSH Server"; NO desktops, NO standard packages, NO print server/web server (even if you'll need one later). This gives you the smallest possible installation as a starting point from which to add everything you need. - If you DO want to create a standard image, it should be created before moving to the next step. ## Step 4 - Post-Install Setup > **Note:** if you're planning to create an image to accompany your standard build, this is the point in the process to do it. At this point, you should have root's password set, a non-root user account created, and a hostname assigned, but still be on a dynamic IP. It's easy enough to set a "generic" hostname here, take a snapshot/backup, then change the hostname back and continue. If you do create a standard image at this point, and/or if you've built your system from a standard image, all you need to do to bring the system up to date is perform an `apt update && apt dist-upgrade`. You can then continue with the steps below. I keep my build instructions in a text file that I refer to from this point forward (a copy of which is located in The Solo Admin's [Github repository](https://github.com/mattpogue/thesoloadmin.com?ref=thesoloadmin.com) under the "posts/linux-server-build-walkthrough" folder) and I'm not going to go through the whole thing step-by-step, but I will include a few of the key points here: - My first steps are: - Configure a static IPv4 address - Modify GRUB to provide a higher screen resolution (reboot) - Configure SSH/SSHD, including `authorized_keys` for the user you created during the install, along with a customized `/etc/ssh/sshd_config` (I'll leave building a customized config as an exercise for the reader). - Run `apt-update` to get latest Debian repo, then install basic packages: ``` apt install vim wget curl net-tools iproute2 man tcpdump dnsutils grc rsync telnet ncat zip unzip mlocate sudo letsencrypt lynx pandoc htop lsof strace git nmap linuxlogo hwinfo screen tmux fio gnupg ``` - Copy all remaining standard config files, most importantly: `.vimrc` and `.bashrc` (see my [previous article](https://thesoloadmin.com/exploring-my-bashrc-in-2021/) for info and a link to my standard `.bashrc`) - Configure `vim` as my default editor: ``` update-alternatives --config editor # choose "vim.basic" from the list ``` - Disable IPv6 (this is a topic for future review, as I'm looking at IPv6) - Install and configure `btop` (I prefer to install from source using [these instructions](https://github.com/aristocratos/btop/blob/main/README.md?ref=thesoloadmin.com#with-make), but it can also be installed using `apt`). - Install other miscellaneous packages like `lets-encrypt`, `nullmailer`, etc. - After the previous steps, I reboot once more and check `dmesg` to make sure I'm not getting any boot errors. ## Step 5 - Wrap-Up and Next Steps With any luck, you've now got a Debian Linux server that's ready to be further customized as needed. My first step is to ensure that SSH login as my standard user account via public key authentication is working as expected. At this point, I also configure `sudo` and verify it's working, and verify that `root` cannot be logged into via password over SSH. If this is a Production machine, my next step is configuring Active Directory authentication, which we'll walk through in my next post. \*Note: I maintain a custom `authorized_keys` for `root` that only allows my "root" key, stored in my password safe, just in case AD auth isn't working. This step is optional, but it's saved my bacon on multiple occasions. As always, if you're stuck and need help, or if there's anything I typo'ed or left out of the process, please don't hesitate to email me - [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com). ### Quick Note: Zabbix Server Dashboard CSS Change URL: https://thesoloadmin.com/quick-note-zabbix-dashboard-css/ Last updated: 2025-03-15T17:23:39.000Z If you're reading this article, you probably already know that [Zabbix](https://www.zabbix.com/?ref=thesoloadmin.com) is one of the premier open source systems monitoring applications out there. They recently released version [7.0](https://www.zabbix.com/download?ref=thesoloadmin.com), but masochists like me have been running 7.0 betas and for quite a few months prior. The Zabbix web interface - one of my primary motivations for using the software - is pretty kick ass, allowing you to build custom dashboards and do most all of the management-related tasks. However, at some point in the 7.0-pre release cycle, they decided to change the label orientation, as shown in this screenshot: ![Zabbix Screenshot - Vertical Tags](https://thesoloadmin.com/content/images/2024/06/screen1.png) Zabbix Screenshot - Vertical Tags If you're familiar with the interface, you'll notice it immediately - the labels are all vertical instead of horizontal. Granted, it's a minor cosmetic annoyance, but it's also a super easy fix. Here's how: 1. SSH to your Zabbix server system. 2. Change to the Zabbix `include` directory: 1. `cd /usr/share/zabbix/ui/include` 3. Make a backup copy of `defines.inc.php`: 1. `cp defines.inc.php /root/zabbix_defines.inc.php_backup` 4. Edit `defines.inc.php` with the editor of your choice (vim, of course!): 1. Search for the line that reads: 1. `define('ZBX_STYLE_TEXT_VERTICAL', 'text-vertical');` 2. To search in vim command mode, type: `/ZBX_STYLE_TEXT_VERTICAL` It will be near line 2180, depending on Zabbix version. 2. Edit the line to read as follows: 1. `define('ZBX_STYLE_TEXT_VERTICAL', '');` 5. Save the file and restart Zabbix server: 1. `:wq` 2. `systemctl restart zabbix-server` 6. Double-check Zabbix's status: 1. `systemctl status zabbix-server` ![](https://thesoloadmin.com/content/images/2024/06/screen2.png) Zabbix server status from systemctl 1. Reload the Zabbix server dashboard in your browser. Your tags should now be restored to their former horizontal glory: ![](https://thesoloadmin.com/content/images/2024/06/screen3.png) Zabbix dashboard with horizontal tabs restored. As I mentioned earlier, this one is a very minor change and an easy fix to boot, but hopefully some of my fellow Zabbix admins will find it as useful as I did. As always, if you have any questions, don't hesitate to reach out: [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com). ### Supply Chain Intrigue, or The Spy Who Shagged My Repo URL: https://thesoloadmin.com/supply-chain-intrigue/ Last updated: 2024-12-12T15:32:38.000Z ***Update 1, 3/31/2024:*** Lassie Collin has [published an update](https://tukaani.org/xz-backdoor/?ref=thesoloadmin.com) on the xz backdoor on his own site, tukaani.org. The DNS CNAME "xz.tukaani.org" has been temporarily removed. Ladies and gentlemen, fellow SysAdmins, distinguished guests - we've got a serious problem on our hands. For those who are new to the party, on 3.29.2024, a backdoor was [discovered](https://open.substack.com/pub/lcamtuf/p/technologist-vs-spy-the-xz-backdoor?ref=thesoloadmin.com) in the open source package `xz`​. Luckily for us all, the malicious dev's code was sloppy and drew the attention of [Anders Freund](https://www.linkedin.com/in/andres-freund/?ref=thesoloadmin.com), a longtime PostgreSQL developer, now working at Microsoft. I was going to do a detailed timeline, but honestly, it's already being covered fantastically: - Evan Boehs - ["Everything I Know About the xz Backdoor" ](https://boehs.org/node/everything-i-know-about-the-xz-backdoor?ref=thesoloadmin.com) - lcamtuf - ["Technologist vs spy: the xz backdoor debate"](https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor?ref=thesoloadmin.com). - [Sam James](https://cmpct.info/~sam/?ref=thesoloadmin.com) ([thesamesam on Github](https://gist.github.com/thesamesam?ref=thesoloadmin.com)) - ["FAQ on the xz-utils backdoor"](https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27?ref=thesoloadmin.com) For anyone who's not up to speed, I strongly encourage you to read those links. However, the `xz` intrigue is only a symptom of a larger problem in the world of software development: securing the supply chain. In just the past few months, we've seen issues with malicious packages on both [PyPi](https://arstechnica.com/security/2024/03/pypi-halted-new-users-and-projects-while-it-fended-off-supply-chain-attack/?ref=thesoloadmin.com) and [NPM](https://www.fortinet.com/blog/threat-research/malicious-packages-hiddin-in-npm?ref=thesoloadmin.com), the main repositories of 3rd party libraries for Python and Javascript respectively. Fortunately, the malicious packages were relatively unsophisticated and were discovered before significant damage could be done, but they hint at an issue that is only going to grow in significance. The `xz` backdoor is probably the most sophisticated attack - in terms of planning and execution, if not necessarily the source code involved - that we've seen so far (*that we know of*), involving extensive planning and almost 3 years of preparation and lead time. The widespread adoption of `git` and the popularity of [Github.com](https://github.com/?ref=thesoloadmin.com) fostered a revolution in software development. Open source repositories were certainly not new; I was using Perl packages from [CPAN](https://cpan.org/?ref=thesoloadmin.com) \- arguably the first open source repository for 3rd party language libraries - in the late '90s (check out [this episode](https://corecursive.com/tdih-cpan/?ref=thesoloadmin.com) of the CoRecursive podcast for a fascinating history of CPAN). But by the mid-2000s, due to stagnation and other issues, Perl was falling out of favor. The rise of PHP, Ruby, Python, and Javscript (via jQuery and Ajax) in web development, and Python, Rust, Javascript (via Node.js), and Go on the systems side - all languages with vibrant developer communities and, most importantly, extensive built-in support for external package repositories - had begun in earnest by the late 2000s. In April 2005, in response to revocation of the open source BitKeeper license for Linux, Linus Torvalds developed `git` (see the [Wikipedia article](https://en.wikipedia.org/wiki/Git?ref=thesoloadmin.com) for a detailed history) and moved Linux kernel versioning to it that same year. In 2008, [Github.com](https://github.com/?ref=thesoloadmin.com) debuted and the revolution truly began. According to the 2023 [developer survey](https://survey.stackoverflow.co/2023/?ref=thesoloadmin.com) by [Stack Overflow](https://stackoverflow.com/?ref=thesoloadmin.com), 93.9% of respondents used `git` for version control, compared to 69.3% in 2015\. In the same survey, Javascript was the most used language by far, used by 65.8% of respondents. For someone like me who's been in technology for over 25 years and developing software as a major part of my job since 2008, the change in both languages and methods of development were nothing short of revolutionary. The increased speed of development, widespread adoption of open source, and extensive usage of 3rd party libraries also brought with it a set of problems unique in the history of software development. As a developer, I can now create sophisticated programs at light speed without having to reinvent the wheel, thanks to the use of external libraries. However, it also makes me nervous when I dig into the package source and see 2,000 folders in the `node_modules/` directory. And that's really the crux of the issue: attacks on software supply chains will only become more prevalent and harder to detect over time and it's becoming nearly impossible for even well-staffed development teams to keep a handle on their application dependencies. As we've seen before with the `leftpad` [package saga](https://medium.com/nerd-for-tech/that-time-a-guy-broke-the-internet-23c00903ad6f?ref=thesoloadmin.com), wittingly or unwittingly, a single developer can inflict a huge amount of damage. Some organizations have tried freezing their dependency chains at a certain "known good" version to combat the issue, but that's at best a stopgap solution. Ultimately, if you had a group of nation-state actors or blackhats who had both the patience and the programming chops to pull off a "low and slow" attack - gaining the trust of package maintainers by committing quality code, being active in the communities over a long period of time, and spreading malicious code out over multiple repositories such that it's only functional when all the pieces are combined on a given system - it would be nearly impossible to prevent it. As the `xz` backdoor has shown, it's entirely possible that this is already happening, and sophisticated backdoors could already be in place in packages considered "critical software infrastructure", e.g OpenSSH and/or Linux kernel modules. It's long been known that a sizable portion of the global tech infrastructure is maintained by volunteers who do it simply because they're passionate about their craft (h/t [xkcd.org](https://xkcd.com/2347?ref=thesoloadmin.com)): ![](https://thesoloadmin.com/content/images/2024/03/dependency.png) It's especially shameful the large number of tech companies who've made their vast fortunes on the backs of these developers, placating the community by hiring a few programmers or donating a token amount of money to open source projects. The open source software revolution in general (cue Cathedral & Bazaar [reference](http://www.catb.org/~esr/writings/cathedral-bazaar/?ref=thesoloadmin.com)) has been the engine powering the rise of the Internet, smartphones, and the Internet of Things, and is largely responsible for kicking off the current ongoing revolution that is AI and LLMs. But as any tech founder can relate, "move fast and break stuff" isn't really compatible with code audits and the principle of least-privilege. We've made HUGE advancements in the 21st century, including changing our entire software development model multiple times, but we've arrived at a point where our future success will depend on our ability to secure what we've built and keep it secure going forward. In my opinion, this is where the rubber meets the road in the debate about AI being used for good or evil. We're faced with a massive challenge; time will tell if we're able to rise to it. --- ## Tip of the Hat, Wag of the Finger Tip of the Hat to the sources referenced above, from which all the other information in this post flows. Big time Tip of the Hat to Anders Freund for discovering and analyzing the `xz` backdoor: And I know I'm not the only person to feel this way, but the biggest Tip of the Hat in this post rightfully belongs to ***Lassie Collin*** ([Larhzu on Github](https://github.com/Larhzu?ref=thesoloadmin.com)). In addition to being completely innocent of anything surrounding the backdoor, he thanklessly maintained an unexciting, but crucial, package that's installed in pretty much every Linux distro in existence. This included having to defend himself against complaints that he wasn't working fast enough while dealing with issues in his personal life. **Reminder**: **this was *NOT* Lassie's full time job, nor was he paid at all as far as I'm aware.** And now a Wag of the Finger to Github for closing down the `xz` repository and suspending the accounts involved - *including Lasse Collin's account* which is absolutely shameful; ## ***Github: restore his account NOW!*** A second Wag of the Finger (or maybe just "the finger") to all the armchair admins out there who will take the time to criticize open source maintainers, who - again, for those who are new to this - are not paid a single dime for their work, for not working fast enough, for not implementing whichever major feature this week that would make their (the admin's) job easier, on and on and on. Hey, fellow admins - if the package maintainers aren't working fast enough for you, why don't you learn to fscking code and *fix it yourself*. Here, let me help you get started: [![](https://thesoloadmin.com/content/images/2024/03/c_language.jpg)](https://www.amazon.com/Programming-Language-2nd-Brian-Kernighan/dp/0131103628?ref=thesoloadmin.com) --- ### Proxmox 8.0 Hang on Boot - A Fix URL: https://thesoloadmin.com/proxmox-8-0-hang-on-boot-fix/ Last updated: 2024-08-15T14:06:32.000Z Today's post is going to be a quick one, but hopefully having another source for the fix for this issue on the web will help someone who's in the same dire straits I was in! (Please note - I'm using the Proxmox "no subscription" repository; if you're an Enterprise customer, YMMV) Here's the scenario - upgrading my servers from Proxmox 7 to Proxmox 8, and Debian 11 (Bullseye) to Debian 12 (Bookworm). I run the `pve7to8` script and everything checks out, so I begin the upgrade. Some 750+ packages later and I'm ready to reboot. I'm working remotely, so I start my continuous ping, issue the `reboot` command and wait. And wait. And.....oh crap. There are few worse feelings in the world than the server not coming back up after a reboot when you're remote. However, I do have the consolation that this is at least a backup server and not one of our Production machines. After driving into the office and hooking up a monitor, here's what I've got: on boot, I get past the grub menu screen and about halfway through the boot process and the system just hangs. Nothing happening, no error messages, nothing. So I reboot the server and choose the recovery option from the menu. Now I can see that the server is hanging up somewhere around network initialization in the boot process. I begin going through the troubleshooting steps - is it maybe a hardware issue that coincidentally struck at the same time? Spoiler alert - NO! *One positive - I now know WAY more about ZFS troubleshooting than I ever wanted to know. BIG props to [Nick Chevsky](https://github.com/nchevsky?ref=thesoloadmin.com) for the [systemrescue-zfs](https://github.com/nchevsky/systemrescue-zfs?ref=thesoloadmin.com) Github repo!* I'll cut out the *12 hours* of troubleshooting that I went through and skip straight to the fix. Fortunately for me, I stumbled across the following link on the Proxmox site - [https://forum.proxmox.com/threads/proxmox-ve-8-0-released.129320/post-567264](https://forum.proxmox.com/threads/proxmox-ve-8-0-released.129320/post-567264?ref=thesoloadmin.com). The fix for the boot hang is to remove the script `/etc/network/if-up.d/ntpsec-ntpdate`. To make a long story short: 1. If your Proxmox root file system is ZFS, download and boot using the [systemrescue-zfs ISO](https://github.com/nchevsky/systemrescue-zfs/releases?ref=thesoloadmin.com). 2. Using instructions from [this link on the Ubuntu forums](https://askubuntu.com/questions/123126/how-do-i-mount-a-zfs-pool?ref=thesoloadmin.com), you'll need to do something similar to the following steps (YMMV if you configured Proxmox differently on your server): 1. `zpool import rpool` \- you may receive a warning that the pool was previously mounted on a different system. If so, use the command `zpool import rpool -f` to force-mount. Note - if the `zpool` command doesn't find your pool, use `zpool import -a` to search for pools on all connected storage. 2. Use the command `zfs get all | grep mountpoint` to locate your the root filesystem mount within the ZFS pool. On my system, it was `rpool/ROOT/pve-1`. 3. Here's the tricky part - when you import the ZFS pool, it attempted to mount any mount points found within the pool, but for me, it did NOT over mount the System Rescue's root filesystem, which is ok. We're going to temporarily change the mount point, remove the NTP script, then change it back, like so: 1. `mkdir /mnt/zfsroot` 2. `zfs unmount rpool/ROOT/pve-1` 3. `zfs set mountpoint=/mnt/zfsroot rpool/ROOT/pve-1` 4. `zfs mount rpool/ROOT/pve-1` 5. Now that your root filesystem is mounted, remove the offending file: 1. `rm /mnt/zfsroot/etc/if-up.d/ntpsec-ntpdate` 6. Now that the offending script has been removed, we need to umount the root filesystem and change our mount point back to the original location: 1. `cd /` 2. `umount /mnt/zfsroot` 3. `zfs set mountpoint=/ rpool/ROOT/pve-1` 3. With that done, you can remove the USB thumb drive and boot your Proxmox server normally. However, there is one final step you'll need to do: 1. When your server boots, the ZFS pool is going to complain about being used under another system and you'll be dumped to a `(initramfs)` prompt. Fortunately, the error tells you exactly what you need to do: `The pool can be imported, use 'zpool import -f' to import the pool. Manually import the pool and exit.` 1. `zpool import -f rpool` 2. `exit` Your server should now be booted completely. These same instructions are also applicable to systems with an LVM root filesystem. In the case of LVM, I would recommend downloading the [Proxmox 8.0 installer ISO](https://proxmox.com/en/downloads?task=callelement&format=raw&item%5Fid=690&element=f85c494b-2b32-4109-b8c1-083cca2b7db6&method=download&args[0]=60ef944c411ad4e68bb65a3ffe694e22&ref=thesoloadmin.com), select the Advanced menu, Recovery. Once booted, use `lvdisplay` to view your logical volumes, then follow step 2c above to remove the `/etc/network/if-up.d/ntpsec-ntpdate` script. One final note - after figuring out the steps above, I was able to proactively remove the `ntpsec` Debian package (`apt remove --purge ntpsec`) and replace it with `chrony` (see [this link](https://chrony.tuxfamily.org/?ref=thesoloadmin.com) for more info on `chrony`) on my Production servers. However, on first boot, one of my prod machines gave me a rescue prompt, as there was a wayward entry in `/etc/fstab` for an external hard drive where the path had changed after the Debian update (one more argument for mounting your disks by UUID!). I would strongly recommend, after you've fixed any issues and booted successfully, reboot one more time just to make sure your server comes up on its own without assistance. I know it's a pain to shut down all the VM's, but it will save you the heartache of a second failed boot process down the road. Final note - I'm a HUGE fan of Proxmox and I've been running Proxmox hosts for many years. I would be happy to help in any way I can; if you run into a Proxmox issue you can't seem to get past, send me an email - [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com). ### Users Aren't the Enemy - They're Your First Line of Defense URL: https://thesoloadmin.com/users-arent-the-enemy/ Last updated: 2025-08-26T13:00:07.000Z In IT, security is always at the back of your mind, ceaselessly reminding you how easily everything you've built can come crashing down around you. And if you don't feel this way, there's nothing like a company-wide incident to bring security sharply into focus. Info security in a large corporate environment is a diffuse thing, where audits attempt to determine levels of risk that can be accepted or mitigated, and "why do I care when there's a whole department dedicated to worrying about that stuff", right? As solo admins, we don't have the luxury of offloading security on to someone else's plate; it's a front-and-center responsibility that ranks right up there with making payroll and keeping the lights on. In a large corporation, it would take one hell of a security incident to bring the whole organization down, which is not to say that it can't happen, only that in terms of risk, total organizational compromise ranks fairly low in the hierarchy. For small and medium-sized businesses, a single incident - ransomware, server compromise, data thefts and leaks, etc. - could very well be a knockout blow from which the organization never recovers ([this article](https://www.inc.com/adam-levin/think-ransomware-cant-put-you-out-of-business.html?ref=thesoloadmin.com) from Inc. magazine provides some perspective). In today's fraught environment for the solo admin, a little peace of mind goes a long way, but peace of mind can be expensive and may not be in the budget this year. In this post, I'm going to talk about what I believe is one of the best uses of the limited resources we have available - educating and building a rapport with your users. First, an aside - very early in my career in the late 90's, I worked on the internal employee help desk for Southwestern Bell (SBC) in St. Louis, before they became AT&T again. Some of the users were getting computers for the first time in their career and called in with the absolute simplest of problems - the "plug it in, turn it on" variety. My team - mostly guys in their early to late 20's - had a grand old time laughing at 30-year veteran lineman who didn't know where the power button was. One day my boss Lloyd (a 30-year veteran himself) countered one of my comments by asking how easily I would be able to hop in a phone company truck, climb a pole, and fix a citywide outage. Or go down to the accounting department and knock out payroll for 50,000 employees, for that matter. My 19 year old self immediately got his point; all of the users we supported had their areas of specialty and they shouldn't be expected to understand my job any more than I was expected to understand theirs. I've repeated that anecdote to coworkers and subordinates many times over the years because I think it's an important lesson - our end users, for the most part, aren't IT experts and we shouldn't expect them to be (in fact, the ones who *think* they are can be the most dangerous). That's what we're here for! That doesn't mean they're not going to do stupid things at times, but one of your goals as an admin should be to prepare for the worst while hoping for the best. I honestly believe that having a strong working relationship with your users is one of the single most important security controls you can implement and with that in mind, here are some tips I've learned to help build and maintain that relationship. ## User Relationship Tip 1 - Open Door Policy On my first day at my current job, I visited all 35 employees to introduce myself and emphasize the fact that my door was always open. I wanted them to know that I don't consider any question to be dumb or not worth asking. I also let them know that I considered them to be our first line of defense against ransomware, viruses, and all around bad shit and that I took all of those things very seriously. I made it clear that I wanted them to report anything suspicious to me so I could investigate. I've also made myself available if they had questions regarding their home setups too. That doesn't mean that they all get free PC support (I've had to draw that line a few times), but since my employer doesn't provide home equipment for every user I genuinely need to know what kind of equipment my users are running at home and what types of issues they're running in to if and when they're accessing work stuff. Especially during Covid, many of my users needed to work from home and I had to make do with what they had. That also doesn't mean that I'll allow any random home equipment to connect to our VPN, as I'll discuss later, but even if they weren't doing work stuff from home, I didn't want them to be running virus-infected computers if for no other reason than for the good of the Internet at large. Plus, I wanted to get them in the habit of paying attention to what's happening on their computers so they'll notice when something is different. That can be the difference between stopping an infection at a single PC versus cleaning up your entire office. Finally, I tried to be as accessible as possible. If you're not a people person, work on becoming one. I've had users stop by my office to inquire about cryptocurrency, let me know they're now running a Helium mining hot spot, get my thoughts on what type of computer they should get for their kids going to college, and all sorts of tech-related stuff. By letting them know that I'm not the unapproachable "IT guru on the hill" that some admins turn themselves into, I've been able to be proactive about all kinds of problems - not just security related - because a user thought it was important enough to mention to me. That won't happen if they're afraid to enter your office. ## User Relationship Tip 2 - Educate, Train, Reinforce My dad tells a story about when he was asked to teach an adult Continuing Education college course in the late 80's on computer basics. At that point in the late 80's the computer revolution was just beginning to transform many occupations. This was before user friendliness, in the days of the command line and the floppy disk, and many of the folks attending his class looked at computers with a mixture of suspicion and trepidation. They were watching their jobs being taken over by a technology they barely understood and some of them were nervous and even hostile. The first night of class, my dad had everyone come up to the front of the class where he had a computer (at the time, very likely a shiny [IBM XT](https://en.wikipedia.org/wiki/IBM%5FPersonal%5FComputer%5FXT?ref=thesoloadmin.com)) on a desk where all the attendees could walk 360° around it and see how the components attached. He explained to them what each component did, and after everyone had a chance to look, they returned to their desks, each of which was equipped with a computer, powered off. Lesson 1 that night was - very simply - "power it on". His goal was to take the "mystery" out of it by showing his students that while it was a complex machine, it was just a machine (insert car repair analogy here). By doing those simple exercises, he reduced his students' anxiety and made them much more receptive to learning. Fast forward to the present and anyone who's worked in a business environment knows at least the basics of how to operate a computer. However, I'm continually amazed that even most high school and college age students have no clue how a computer actually *functions* \- meaning, what's inside and what each part does. By the same token, you'll find that most of your users have almost no understanding of how email works, why ransomware is a threat, or why one web browser is any better worse than another (or most likely, even what a web browser IS). It's absolutely **imperative** that you train your users. I've implemented an annual security training program that all users are required to attend and I supplement that training throughout the year by sending out short emails addressing threats that they may have heard about on the news or that I've been asked about. But in each year's class - much like the beginning of the school year for teenagers - I always reinforce the basics. I try not to go too far into the weeds, but I will pick one or two current threats or current news stories (ransomware has dominated for the last few years) and sprinkle them throughout the serious stuff: how to identify spam/phishing emails, how to pick secure passwords and store them in [KeePassXC](https://keepassxc.org/?ref=thesoloadmin.com), how to spot bad/malicious web content, and what to do when something fishy happens on their computer (UNPLUG!). And the most important rule of all - report any suspicious occurrence to me. I try to keep the class to an hour with questions at the end and I've managed to get even the most meeting-averse users to pay attention. Bottom line: educate your users and train them by reinforcing that education throughout the year (especially informally through emails). ## User Relationship Tip 3 - Don't Be a Babysitter For most people (myself included), it feels very creepy to spy on your coworkers. Prior to this wonderful new era where remote work is common, small businesses tended to fall on one side of the fence or the other without a lot of gray area in the middle. Either everyone is in the office all the time or everyone works somewhere else (and maybe there isn't a physical office). My boss is old school - if you're not in the office, you're not working. When Covid hit, it upended this relationship. We were limited to 10 people or less in the office for long stretches at a time. People had to quarantine when they weren't sick. My boss and I discussed the situation and I explained it to him exactly as I'm explaining it here - don't be a babysitter. We settled on generating some activity reports from our internal project management application and a couple of Exchange 365 activity reports. Obviously not enough to track a person's every move when they're working at home, but enough so that you could tell if a person had generated any activity on a given day. There are a few reasons why this tip is so important: - All that goodwill relationship building you've put in with your users will be worthless if they feel that you're spying on them. The old "just following orders" chestnut won't help you much when an "us versus them" mentality takes hold. - Trust plays a big role in any relationship - I'm asking my users to be my eyes and ears and report back to me when they see suspicious activity. No one's going to do that if they feel you're looking at *them* suspiciously. - Especially in small businesses, when a user isn't doing their job, it becomes obvious pretty quickly. We're trusting our managers to manage their staff, and we're also expecting them to trust their subordinates. If someone is at home and they step away from their computer for a few minutes, how is that any different than a user getting up from their desk to chat with a coworker, get a cup of coffee, etc., in the office? It's not any different and we should acknowledge that. - In the United States we already work too much. This is [one example](https://biteable.com/blog/work-from-home-statistics/?ref=thesoloadmin.com), but there are many other polls and studies showing that we work more hours from home, not less. Additionally, we're now invading our users' private lives more than ever. Cory Doctorow has a *fantastic* [article](https://pluralistic.net/2022/08/21/great-taylors-ghost/?ref=thesoloadmin.com#solidarity-or-bust) on his blog about how surveillance in the guise of "productivity monitoring" has invaded our homes. This needs to be drastically scaled back. - Finally, people are resourceful. If you go too far with the monitoring and blocking, your users will begin to develop ways to bypass your controls. I would much rather my users browse Facebook on their lunch break on a system I have complete control over than to try and find ways to defeat that control. There's absolutely no need for an adversarial relationship with your users. All that being said, this doesn't mean we're going to operate like the wild west. I monitor all traffic pretty extensively, my users' computers are locked down, and I aggressively block spam, executable attachments, and other types of malware before it gets to my users. Which leads me to tip #4.... ## User Relationship Tip 4 - Guide Behavior with Controls After reading the previous tip you might be asking whether I'm advocating for a free-reins environment and that's absolutely **not** the case! I'm hoping that if you're reading this post you have the sentience to distinguish between technical controls and outright surveillance, but if not, here's the guide that I would use: whenever you're considering a new monitoring system or additional product to enforce corporate policy, ask yourself if you're comfortable with it being done to you. If it makes you uncomfortable it will make your users uncomfortable too. As a general rule of thumb, avoid any app that requires a user to be recorded - video, audio, keystrokes. That's creepy. Your controls should be as unobtrusive as possible. For example, by default, your spam filter should block ALL file types and only allow the ones your users will use - PDF, DOC/DOCX, XLSX, PNG/JPG/BMP, etc. You'll find that there's generally a small list of extensions that are legitimate. We also spam filter pretty aggressively and I've trained my users how to add exceptions in Barracuda and/or request an addition to the global whitelist. At implementation we had quite a few adds all at once but now I add about 2 - 3 exceptions per month. I control our desktops and laptops with Group Policy and a hardening script that's part of our standard build. Finally, my firewall allows ports 80 and 443 out from the user IP block along with a couple of exceptions. All other outbound traffic is blocked. When my users connect to our VPN, I route all their outbound traffic through my firewall so I can monitor it. If you're serious about security, your posture should always, always, always start with "default deny" and go from there. Outside of those controls, I don't feel the need to spy on my users or log their keystrokes because I'm not the morality police and I could never hope to wade through all that data in the first place. I've taken intentional steps to limit the damage that could be done if something nefarious were to slip through and I choose to spend the rest of my time making sure that I could completely recover from a disaster - whether physical or digital - *when* it happens (not *if)*. Which leads me to the final tip.... ## User Relationship Tip 5 - Don't Be a Jerk This is good advice for your personal life too. It's 2022\. There are hundreds of thousands of IT people in the world and you're not any more special or entitled than any of the rest of us. I exchanged emails with a PE-certified engineer this week who worked on components of the Apollo spacecraft. Unless you're THAT cool, keep it to yourself. I do my best to be humble and genuinely friendly to everyone I work with because they're my 2nd family. Your office may not be like that but there's no reason for snooty hostility either. In IT our main purpose should be to keep the business's technological parts and pieces running smoothly, utilize that same technology to improve productivity, and solve whatever problems our users and customers may encounter. If you enjoyed this post, leave a comment below or better yet, click the "Subscribe" button to get new posts in your Inbox. Thank you for reading! ### A New Direction URL: https://thesoloadmin.com/a-new-direction/ Last updated: 2024-08-15T14:06:50.000Z After a lot of soul searching, I've decided to take The Solo Admin blog in a new direction. I'm still passionate about my intended audience - solo IT practitioners, members of small IT departments, employees of MSP's who focus on small business, or even perhaps employees who were drafted to perform IT duties when there was no one else available to perform them. I fall into this category; I'm the sole IT person for a small business of 35 employees and I know there *must* be many others like me out there. I've had amazing mentors and opportunities throughout my career and I want to give something back to the industry that has provided me with a rewarding career for so many years now. I know there are many folks for whom working in a small company translates to being new to IT and at an early stage in their careers. In those instances, without a mentor and other team members to learn from and help share the load, it can be a struggle just to keep your head above water. In other instances, perhaps you're an experienced manager of a small team of folks. Or maybe you're like me and you're in a unique situation where you're well into your career and chose your position because it requires a huge diversity of knowledge, including both administration and programming, and you were tired of the large corporate "same job every day" grind. Whatever the case, I know you're out there! If you're one of my small band of subscribers (thank you so very much if you are!) or just someone who's stopped in from time to time to check out a post, then you know that I've mainly done technical "how-to" type posts like my [Batch Downloading with Python](https://thesoloadmin.com/batch-downloading-with-python/) post. And while I've enjoyed doing those, in general, I don't feel they're the best purpose for the blog. In most of them, I ended up linking to the excellent work that other folks had already done and I felt that I wasn't contributing very much in the overall scheme of things. So I really felt I was at a crossroads. I love doing this blog and I want to continue to grow and find new readers, but what I've done so far has amounted to little more than a whisper in a windstorm. The feeling became overwhelming that I needed to go a different way. In April of this year, I authored a post titled [The Solo Developer](https://thesoloadmin.com/the-solo-developer/). Rather than provide a step-by-step walk through, the post was written as a set of general guidelines a solo developer could follow. To my surprise, it was - and still is - far and away the most read post on the site; nothing else has even come close. Seeing this result was the confirmation I needed that the blog's articles should be more high-level, focusing more on general topics and sharing the knowledge I've gained from over 25 years in the industry. ![](https://thesoloadmin.com/content/images/2022/08/post_stats-1.png) ****Analytics from TheSoloAdmin.com. Note the spike in traffic at the end of April.** The posts on the blog - for the most part - will still be very technical because my passion lies in the day-to-day, hands-on (which explains why I'm 25 years into my career and *still* haven't moved to management), but I'm also going to be writing about high-level skills - building user and management relationships, security processes and training, integrating cutting edge tech on a shoestring budget, etc. You'll still find code snippets, whole scripts, config files, and possibly even a short how-to if the situation calls for it. I'll still share things that I work on that I feel others could make use of; for example, I have a set of configuration management scripts that I've used for years that I've often thought about combining into a "mini Ansible" and maybe I'll turn it into a short series of posts and a Github repo. The bottom line is if you've read any of my posts and enjoyed them, thank you for taking the time; it means a lot to me. Take a minute to subscribe or add my feed to your RSS reader (I personally recommend [Tiny Tiny RSS](https://tt-rss.org/?ref=thesoloadmin.com)) and I promise I will do my best to make it worth your time. I also promise that if you provide your email address to the blog, I will never, ever, EVER sell or use your info in any way other than to send you blog-related stuff. So stay tuned for some new content coming soon! ### Setting Up Wazuh for Host-Based Monitoring URL: https://thesoloadmin.com/setting-up-wazuh-for-host-based-monitoring/ Last updated: 2022-07-19T19:20:53.000Z [CodeProject](http://www.codeproject.com/?ref=thesoloadmin.com) Our modern world is a dangerous place, fraught with peril, seemingly with bad actors at every turn. Yet we must continue forth somehow! What to do, what to do? For the solo admin with a limited budget, in this article (and an article or two to follow) we'll take a look at [Wazuh](https://wazuh.com/?ref=thesoloadmin.com) for endpoint security. I'll note here that The Solo Admin blog isn't affiliated or partnered with anyone; the software and hardware that I write about is from personal interest and use only. So let's dive in. First off, since the only official Debian flavor that Wazuh supports is Ubuntu, I spun up a new Proxmox VM with 4 CPU cores, 8 GB of RAM, and 50 GB disk space. According to the [Wazuh server requirements](https://documentation.wazuh.com/current/quickstart.html?ref=thesoloadmin.com), this is the bare minimum for 1 - 25 agents. In my environment, it should be sufficient for my needs, but if not, my virtual server host does have some extra head room I can allocate later on. As an aside, one of the things that bugs me is Ubuntu's lack of a "small" install source for their server version, along the lines of Debian's "network install" ISO which clocks in at a svelte 380 MB. I was unable to find anything comparable for Ubuntu 22.04, however, I was able to find a small install for Ubuntu 20.04 at [this link](http://archive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/current/legacy-images/netboot/mini.iso?ref=thesoloadmin.com). For the build, I'm not going to walk step by step through the Ubuntu install; simply perform a standard install using whatever process you prefer (in the future I may write up the standard build process that I follow for all Linux systems that includes package installation and configuration after the initial build). One step I always perform on Ubuntu systems is to disable automatic updates; linuxnightly.com has a detailed walk-through of the process [here](https://linuxnightly.com/how-to-disable-automatic-updates-in-ubuntu/?ref=thesoloadmin.com). The reason I recommend disabling automatic updates is because I've been burned by Linux (and Windows) updates too many times. On my servers, I install updates manually - `apt update && apt dist-upgrade` \- and record my updates into a log file per-server (yes, I know `apt` keeps its own log too) so that I can easily troubleshoot a broken app or a server that won't boot. Of course with virtual servers, I take a snapshot so that if the problem is REALLY bad, I can rollback, but what's the fun in that? --- ### Aside - Removing Ubuntu "Cruft" One of the main reasons I dislike Ubuntu servers is the amount of "cruft" they force on you right out of the gate. In this instance, I'm talking about netplan and systemd-resolved. I vastly prefer the `/etc/network/interfaces` style of configuration - it's *simple*. It also covers your DNS resolution; no need to spread files all over `/etc`. So 2 additional post-build steps I took were to disable Netplan using [these instructions](https://askubuntu.com/questions/1031709/ubuntu-18-04-switch-back-to-etc-network-interfaces?ref=thesoloadmin.com) and disable `systemd-resolved` with: ``` systemctl disable systemd-resolved rm /etc/resolv.conf ``` Ensure that you create a new `/etc/resolv.conf`, using your settings from `/etc/network/interfaces`: ``` nameserver 8.8.8.8 nameserver 1.1.1.1 ... ``` --- So that now that we've built the server, performed our standard post-build configuration, and taken a snapshot, it's time to upgrade to Ubuntu 22.04\. For this process, I got the basic steps from linuxize.com [here](https://linuxize.com/post/how-to-upgrade-to-ubuntu-22-04/?ref=thesoloadmin.com). It's pretty straightforward, but I would strongly recommend performing an update of this magnitude from the console (or from the VM server's virtual console) versus over SSH, both so you have virtual terminal support for troubleshooting and so you don't need to worry about disconnects. Note: In preparation for the update, I did need to edit the file `/etc/update-manager/release-upgrades` and set `Prompt=normal` in order to move to 22.04\. For me, the upgrade took close to 2 hours but YMMV. After package installs and a reboot, let's double-check the server version with `lsb_release`: ``` root@infosecserver:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04 LTS Release: 22.04 Codename: jammy ``` One detour from the basic Wazuh install - I'm going to install Postgresql server to enable database storage for alerts. By default, it looks like Wazuh stores alerts in a file, but I like the option for database storage so I'm going to try it out. If I run into performance issues, I'll revert back to file-based alert storage. Simple Postgresql server install instructions: ``` # Add Postgresql repo echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/postgresql.list # Install the Postgresql GnuPG signing key wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc > /etc/apt/trusted.gpg.d/postgresql-official.asc # Update and install apt update && apt install postgresql # Install the Postgresql development libraries apt install libpq-dev ``` Now we're finally ready to get down to the meat of the matter - Wazuh server installation! Since we have a supported installation, we're going to use the Quickstart installation process from [here](https://documentation.wazuh.com/current/quickstart.html?ref=thesoloadmin.com). Basically, it's just: ``` curl -sO https://packages.wazuh.com/4.3/wazuh-install.sh && sudo bash ./wazuh-install.sh -a ``` While it's installing, I would encourage you to open a new terminal window a start a tail follow of the install log file, `/var/log/wazuh-install.log`. ``` root@infosecserver [/usr/local/src]# ./wazuh-install.sh -a 19/07/2022 11:32:46 INFO: Starting Wazuh installation assistant. Wazuh version: 4.3.5 19/07/2022 11:32:46 INFO: Verbose logging redirected to /var/log/wazuh-install.log 19/07/2022 11:32:51 INFO: --- Dependencies ---- 19/07/2022 11:32:51 INFO: Installing apt-transport-https. 19/07/2022 11:32:53 INFO: Installing software-properties-common. 19/07/2022 11:33:04 INFO: Wazuh repository added. 19/07/2022 11:33:04 INFO: --- Configuration files --- 19/07/2022 11:33:04 INFO: Generating configuration files. 19/07/2022 11:33:06 INFO: Created wazuh-install-files.tar. It contains the Wazuh cluster key, certificates, and passwords necessary for installation. 19/07/2022 11:33:06 INFO: --- Wazuh indexer --- 19/07/2022 11:33:06 INFO: Starting Wazuh indexer installation. 19/07/2022 11:33:54 INFO: Wazuh indexer installation finished. ...... 19/07/2022 11:36:24 INFO: --- Summary --- 19/07/2022 11:36:24 INFO: You can access the web interface https:// User: admin Password: 19/07/2022 11:36:24 INFO: Installation finished. ``` Now that the server install is complete, I deployed my first agent using the instructions found [here](https://documentation.wazuh.com/current/installation-guide/wazuh-agent/wazuh-agent-package-linux.html?ref=thesoloadmin.com#deploy-a-wazuh-agent). After install, I logged into the Wazuh console and sure enough, there's my agent. I also validated that my recent SSH session into the agent system was detected and logged. ![](https://thesoloadmin.com/content/images/2022/07/wazuh_home-1.png) So we now have a working Wazuh installation with a successful Linux agent install. I will now start working on the bane of every security admin's existence - alert tuning/false positive elimination. I will also be deploying Windows server agents and I will write up both processes - tuning and agent deployment - in the near future. Stay tuned! ### The Solo Developer URL: https://thesoloadmin.com/the-solo-developer/ Last updated: 2022-08-11T22:01:24.000Z [CodeProject](http://www.codeproject.com/?ref=thesoloadmin.com) Throughout my career I've worked on multiple engagements where I was tasked with cleaning up after a fellow developer who - after anything from writing a few lines of code to automate a process to writing a full blown application - disappeared, leaving the client company in desperate straits. While I was tempted at times to "punish" the client for their ignorance, after evaluating the situation I typically realized that it was not the client who was at fault, but rather the developer who had failed to properly set expectations and deliver a finished product to the customer. In my current full-time position, I spend more than 50% of my time writing code in support of our in-house project management application. The current version (with some major modifications over the years) was put into production in 2010 and still performs beautifully. However, the world has moved on in the interim, so most of my development efforts at this point are dedicated to rewriting the application to bring it into the modern era. The point of all this is that I harbor a very low opinion of those long-gone developers I mentioned in the first paragraph. The ones who couldn't be bothered to deliver source code (where appropriate), use a version control system, or even prepare the most basic documentation; and while I am a team of one, I vowed that anyone who came behind me would not suffer the way I have. So in this post, I will outline the steps I've taken to make my small development world a better place for those who will someday follow in my footsteps. **Step 1 - Version Control** Before writing any code, you should decide on a version control system and implement it. Personally, I prefer [Subversion](https://subversion.apache.org/?ref=thesoloadmin.com) as it contains all the features I need - file diff between versions, easy branch/tag capabilities, and compatibility with all operating systems. On Windows, I use the [TortoiseSVN](https://tortoisesvn.net/?ref=thesoloadmin.com) client, and on Linux the command line client. If you prefer to host your repositories on Windows, [VisualSVN Server](https://www.visualsvn.com/server/?ref=thesoloadmin.com) provides a nice graphical interface to manage your repositories (although there is a cost associated with it). In addition, the [WebSVN project](https://websvnphp.github.io/?ref=thesoloadmin.com) provides a beautiful web interface to your repositories. That being said, any well-established VCS will do the trick, from [CVS](https://www.nongnu.org/cvs/?ref=thesoloadmin.com) to [Git](https://about.gitlab.com/?ref=thesoloadmin.com). Typically, I structure my code repositories with a `tags/` directory containing my release versions and a `trunk/` directory that contains my current working version. A new release gets its own version-numbered folder under `tags/`. I've found that this is typically enough; however, feel free to use whatever structure makes sense. The structure you choose is far less important than the fact that you're using version control. I also document each commit, tying as much as possible into my bug/issue tracking software (next section). While it's important to document and commit your code, don't hesitate to get creative and use your VCS for anything where version/change history is important. For instance, I have a second repository that contains all database-related code and documentation, including: - A SQL script to recreate the database schema - A backup copy of the database with any seed records that are required to perform a "default" install of the application - e.g. an "admin" user account in the users table, pre-populated quantity/measurement tables, etc., and the SQL script(s) to create it - Any miscellaneous SQL scripts that relate to the project, such as data import scripts, performance/troubleshooting scripts, etc. Finally, I also keep a "devops" repository with copies of all my generic server and workstation config files, miscellaneous scripts, server update logs, and anything else I feel is relevant. This gives me an easy way to keep my scripts, files, documentation, and change logs together. **Step 2 - Issue Tracking** Even as a solo developer, it's a good idea to maintain an issue tracking database as a supplement to the documentation in your VCS. My personal favorite for the last several years is [Redmine](https://www.redmine.org/?ref=thesoloadmin.com), however, I've also had success with [Mantis](https://www.mantisbt.org/?ref=thesoloadmin.com) and The Bug Genie (now known as [Pachno](https://www.pach.no/?ref=thesoloadmin.com), so YMMV). Redmine provides easy integration with my Subversion repositories; linking to an issue is as easy as putting "Issue #123" in my commit log which provides an automatic link to the issue when the repository is viewed in Redmine. Redmine also allows me to track time spent per issue, customize my categories and priorities, and autogenerate change logs and wiki entries. Between Redmine and my repository documentation, I can easily tell when a feature was added as well as tying a feature to a specific change in my source code. With its support for multiple projects, I can also use a single Redmine installation to track work for multiple clients. **Step 3 - IDE** Without starting a holy war over which IDE is preferable, I think it's safe to say that for C#/.NET development, [Visual Studio](https://visualstudio.microsoft.com/?ref=thesoloadmin.com) is going to provide the best bang for the buck, especially considering that my version of choice is Visual Studio Community (free). Since my project management application is composed of several C# class libraries on the backend and and .NET MVC web frontend, the full version of Visual Studio is my primary IDE. However, for almost everything else - including Python, PHP, shell scripting, and Powershell to name a few - I'm now using [Visual Studio Code](https://code.visualstudio.com/?ref=thesoloadmin.com). With its ever expanding universe of plugins and themes, settings sync via Microsoft, support for Git and other version control systems, and cross platform compatibility, it's hard to beat! As a fallback, on my Windows systems, I *always* have [Notepad++](https://notepad-plus-plus.org/?ref=thesoloadmin.com) installed as well. From editing config files to viewing code snippets to debugging JSON from the browser, you really can't beat it. I also use it for making quick edits to scripts and HTML. Regardless of which IDE you prefer, the bottom line is to find one that does what you want and supports the features you need. Learn it from top to bottom and stick with it. Over the years, I've tried just about everything from [Atom](https://atom.io/?ref=thesoloadmin.com) to [Jetbrains](https://www.jetbrains.com/?ref=thesoloadmin.com) and they've all had some features I liked. However, at the end of the day, I just didn't have a reason to switch, especially since most of my coding is in the Microsoft world. **Step 4 - Miscellaneous Applications** Finally, I want to mention some of the apps I use that are an indispensable part of my toolkit. There are many, many more tools out there that are just as deserving of praise as these and I encourage you as a solo developer to always be on the lookout for apps that can increase your productivity, automate repetitive tasks, or provide those killer features that you can't get anywhere else. Most - if not all - of these applications have alternatives, some of which are probably better, but I'm a big believer in simplicity and stability. Again, that being said, I'm always keeping an eye out for new stuff; constant learning is one of the main requirements of this job. If you're not willing to put in the time or the effort to continually learn and improve upon your skills, you're in the wrong profession! Here are some of the apps I use frequently to support my development tasks: - [OpenDBDiff](https://github.com/OpenDBDiff/OpenDBDiff?ref=thesoloadmin.com) \- Diff tool for SQL databases. I use this frequently to make sure that my changes are sync'ed between my Development and Production databases. Quick, easy, and performs as advertised. - [Postman](https://www.postman.com/?ref=thesoloadmin.com) \- API testing tool. I recently worked on a project building out an API and Postman was exactly what I needed to test both locally, in Development, and in Production. The free version is enough for me, but the paid version provides extra features. - [pgAdmin](https://www.pgadmin.org/?ref=thesoloadmin.com) \- the only GUI you need for working with PostgreSQL databases. - [HTTPToolkit](https://httptoolkit.tech/?ref=thesoloadmin.com) \- a relative newcomer to the toolbox, HTTP Toolkit allows you to intercept HTTP and HTTPS traffic from within the browser. Similar to [Fiddler Classic](https://www.telerik.com/fiddler/fiddler-classic?ref=thesoloadmin.com), another great tool I've used over the years. **Step 5 - Philosophy** In conclusion, I want to talk a little bit about the philosophy behind the decisions I make as a solo developer. As I mentioned earlier in the post, I take a dim view of any developer who holds clients hostage by not providing source code or documentation on a project, hijacks domain names, or otherwise operates in a way that gives the rest of us a bad name. In order to differentiate yourself from the bad actors in our profession, here are a few suggestions from my personal experience: - Whether you're in a full time position or a contract/freelance position, determine upfront what's expected of you in terms of timeline, pay scale/rate, and work output. It should be decided upfront who will own the source code you produce; generally, in a freelance situation, documented source code (including VCS repositories) is part of my final deliverable. If that's not going to be the case, then think about offering some type of [source code escrow](https://en.wikipedia.org/wiki/Source%5Fcode%5Fescrow?ref=thesoloadmin.com) so that future developers can enhance and improve on your original code in the event that you're not available. - Document, document, document! Utilize your VCS and issue tracker to generate "developer docs" in addition to the usage documentation you provide the client. Future developers who have to work on your stuff will thank you for it. - Build a solid and stable development environment. This is more of a personal recommendation. As I mentioned, in our profession, we should always be learning and on the lookout for tools that can enhance our productivity or make our lives easier. However, I've found that deciding on a core set of tools and learning them inside and out has made the biggest boost to my productivity. Especially your IDE - figure out how you work best and go with it. If you prefer VIM and the terminal, then go that way. If you're more of a GUI person, then go that direction. Whichever direction you go, commit to it and learn it. These are some of the tips and techniques I've developed over the years as a solo developer. I'm sure there are many other ways to get the same job done and that's kind of the point - pick what works best for you and stick with it! If you have any questions about this post, feel free to email me - [matt@thesoloadmin.com](mailto:matt@thesoloadmin.com). ### No, It's Not Always DNS URL: https://thesoloadmin.com/no-its-not-always-dns/ Last updated: 2024-03-27T19:29:02.000Z In today's environment of always-on, always-available Internet connectivity, network troubleshooting has become somewhat of a lost art. However, there are those of us still around who remember this was not always the case. Network troubleshooting was - and is - a crucial skill for both admins and developers and you may find it coming in handy when you least expect it. In this article, we'll explore network troubleshooting up and down the stack. Let's start by talking a bit about the "stack", a.k.a. network "layers". In the late 90's, networking was far from the "set it and forget it" technology it has become. Several of the companies I worked for early in my career had dial-up Internet access primarily dedicated to email and Internet access on the desktop was a luxury, especially in smaller companies. However, you weren't missing much, as there wasn't that much corporate world-related content out there yet. When I got into IT in 1997, the certification game was at its peak, led by Microsoft, Novell (for you youngsters reading this, they're the company that Microsoft stole Active Directory from - [https://en.wikipedia.org/wiki/Novell](https://en.wikipedia.org/wiki/Novell?ref=thesoloadmin.com)), and Cisco, and one thing common in all the certifications was the [OSI Model](https://osi-model.com/?ref=thesoloadmin.com) and its layers: ![](https://thesoloadmin.com/content/images/2022/04/tcpip_osi_mapping.jpg) OSI and TCP/IP Models While the OSI model is still around, the TCP/IP model with its four layers instead of seven seems to be more relevant to today's environment so it's the model I'll be using for this article. The four layers of the TCP/IP model are: - Network Interface, a.k.a. Physical (encapsulating layers 1 and 2 of the OSI model) - Internet (encapsulating all the "layer three" protocols and IP routing) - Transport (TCP, UDP) - Application (everything else) While being more rare today than in the past, it's very useful to know how to identify the layers in which various technologies operate and to understand at a high level how data flows through the layers of the stack. Here's how it was explained to me: - At the sending end, as application data is prepared to be sent over the network, a packet is created by adding headers as it moves down the stack, finally to the physical layer where transmission occurs (whether on a hard line, wireless, satellite etc.). - At the receiving end, the packet makes its way up the stack and headers are discarded until the data contained within the packet is all that's left. For instance, let's look at an TCP packet with its headers as an example: ![](https://thesoloadmin.com/content/images/2022/03/packet_diagram.jpg) Packet Diagram Without going further into detail, now that we have a basic understanding of the stack, we can talk about the tools available to troubleshoot problems at the corresponding layers. As we go, I'll point out the various tools you can use on both Windows and Linux operating systems. ***Note: The following commands must be run in either an elevated command prompt/Powershell session in Windows or a root terminal in Linux.*** **Layer 1 - Network Interface** Here in the 21st century, we tend not to put too much thought into the physical layer because it's the layer we typically expect to "just work". However, as a solo admin, the physical network layer is in our jurisdiction, so we need to be familiar with it. Like most corporate offices, the majority of my users are connected to the network via [Cat 5e](https://en.wikipedia.org/wiki/Category%5F5%5Fcable?ref=thesoloadmin.com). For the purposes of today's discussion, we'll assume that you've already tested the cable and determined there are no physical issues, so we need to check things at the network adapter in the computer. On Windows, open an elevated command prompt and run the command `netstat -e`: ![](https://thesoloadmin.com/content/images/2022/03/netstat_command.png) You're focusing on the "Errors" statistic; if this number is above zero, then you're experiencing physical layer issues. Over the years, I've seen physical layer errors manifest as timeouts, dropped packets, and other weirdness in the higher layers. If you're experiencing odd issues that seem to defy explanation, look for errors at the physical layer. On Linux you can get a quick overview with the command `ifconfig` and more detailed statistics with the command `netstat -s`: ![](https://thesoloadmin.com/content/images/2022/03/ifconfig_command-1.png) ifconfig/netstat Commands on Linux With regard to wireless networking, on Windows, you can generate a (very) detailed wireless networking report by issuing the command `netsh wlan show wlanreport`: ![](https://thesoloadmin.com/content/images/2022/03/wlan_report.png) Although the WLAN report includes more than just the layer one statistics, it does provide a detailed view of them as well. On Linux, the command `iwconfig` provides a detailed overview of wireless status, including bit rate, link quality, etc. Working our way up from the physical connection, but before we get to layer 2 and IP routing, we have the protocols that make up layer 2 of the OSI model or the "data link" layer. This layer can be problematic in a small network environment, especially if you're dealing with older hardware. It's also important to keep in mind the [difference between a hub and a switch](https://byjus.com/gate/difference-between-hub-and-switch/?ref=thesoloadmin.com), both when troubleshooting and when purchasing new hardware. Unless you have some crazy reason I can't fathom, don't ever buy a hub. Come on people - it's the 21st century; don't use networking tech from the 90's! Some of the more common issues I've run into when troubleshooting layer 2 issues are: - ARP caching on the switch (typical solution: reboot to clear the cache, or clear from within the management interface). - Incorrect VLAN configuration; in general, VLAN's aren't needed in small networks. However, they can be useful for providing a degree of isolation within a single hardware device. - Although technically a layer 1 issue, link speed and duplex configuration can be a BIG problem if not configured properly. For example, if you have auto-negotiation disabled on the switch and you're not properly configuring your link speed and duplex settings, you will generally experience horrible performance, sometimes manifesting as errors when running `netstat` / `ifconfig` (previous section). On both Windows and Linux you can view the local machine's ARP cache via the `arp` command: ![](https://thesoloadmin.com/content/images/2022/04/arp.png) ARP Command Output **Layer 2 - Internet** Most everyone reading this article has probably performed troubleshooting at layer two at some point, since this is the layer in which `ping` and friends operate, but hopefully you'll come away with some new techniques and/or tools after reading this section. As an admin, it's easy to overlook this layer when troubleshooting, but over the years, I've encountered a number of issues here. Note - in the OSI model, IP networking is considered layer 3 so if/when you hear someone talking about "layer 3", chances are they're talking about this layer. Since almost everyone should be familiar with `ping`, I won't spend too much time on it. However, there are a number of ping enhancements and substitutions that are available that you should be aware of. On Windows, there are the Powershell `Test-Connection` and `Test-NetConnection` commands. These commands blur the line between layer 2 and layer 3 of the TCP/IP model, but they provide a number of useful options, especially `Test-NetConnection`. The Microsoft documentation reference for `Test-NetConnection` is [here](https://docs.microsoft.com/en-us/powershell/module/nettcpip/test-netconnection?view=windowsserver2022-ps&ref=thesoloadmin.com). Also functioning at layer 2 of the TCP/IP model is IP routing. In a small network environment, you most likely won't be dealing with backbone routing protocols, however, a working knowledge of how routing works is crucial when troubleshooting. For instance, in my small network environment we have two satellite office locations connected through [OpenVPN](https://openvpn.net/community/?ref=thesoloadmin.com) tunnels (creating two additional internal subnets that need to be routed properly), and remote users also connected via OpenVPN clients (creating a fourth internal subnet). In the network I just described, the subnets are assigned as follows ([CIDR](https://en.wikipedia.org/wiki/Classless%5FInter-Domain%5FRouting?ref=thesoloadmin.com) notation): - 10.0.0.0/24 - HQ office LAN - 10.0.1.0/24 - Remote Office #1 - 10.0.2.0/24 - Remote Office #2 - 10.0.254.0/24 - Remote VPN Users Do I really need 254 addresses for each location? No, I don't. However, by using full class-C networks, I have plenty of room for growth and I don't have to do any subnetting math in my head (if you find yourself needing to do subnet math, my favorite IP subnet calculator website is [here](https://www.subnet-calculator.com/?ref=thesoloadmin.com)). When troubleshooting routing issues, the old standby tool `traceroute` (`tracert` on Windows) is your friend; however, routing is one of those things that as a solo admin, you'll rarely have to deal with. Unless you're a sadist, you'll want to handle all routing at the firewall, assign your hosts a default gateway, and call it a day. For instance, let's look at the output from a trace from my local machine to one of the remote subnet hosts (10.0.1.254): ![](https://thesoloadmin.com/content/images/2022/04/tracert.png) Tracert Output Not very exciting - I go through my default gateway (the firewall), through the VPN tunnel and out the other side. As far as routing is concerned, my advice is to keep it simple - your Internet gateway will be the default route for your firewalls and if your VPN tunnels are configured correctly, the firewalls will handle the VPN routes for you. However, knowing how routing functions can come in very handy when troubleshooting these types of issues; the guru99.com website has a primer on IP routing [here](https://www.guru99.com/ip-routing.html?ref=thesoloadmin.com) that I would strongly recommend if you're not familiar with all the concepts. I might cover routing loops, TTL, and other vagaries of routing in a separate post (or email me if you have questions). **Layer 3 - Transport** The Transport layer is generally consistent in both the OSI model (as layer [4](https://osi-model.com/transport-layer/?ref=thesoloadmin.com)) and the TCP/IP model and contains transport protocols, the most common of which are TCP and UDP. From layer 4 up, both the protocols and the troubleshooting techniques are more complex. Examining TCP vs. UDP, it's handy to know the difference between the two in terms of function and general usage. The howtogeek.com website has a [very nice writeup](https://www.howtogeek.com/190014/htg-explains-what-is-the-difference-between-tcp-and-udp/?ref=thesoloadmin.com) of the differences between TCP and UDP. It's at this point that we will begin to rely on full blown applications such as [Wireshark](https://www.wireshark.org/?ref=thesoloadmin.com) and [Nmap](https://nmap.org/?ref=thesoloadmin.com) to assist us with troubleshooting. (As an aside, one of my favorite general IT interview questions asks the candidate to describe the TCP three-way handshake, e.g. SYN -> SYN/ACK -> ACK). One of my go-to commands on Linux is the `netcat` command, alias `nc`. The `nc` command is ideal for determining whether a TCP port is open and can send and receive data. For example, we can determine if www.google.com is responding by issuing the command `nc -v www.google.com 80`. If we receive a successful connection message, we can send the request `GET / HTTP/1.0` to request a copy of the root (home) page and we should receive a response back: ![](https://thesoloadmin.com/content/images/2022/04/netcat.png) Netcat Command Output Detailed usage of programs like Wireshark and Nmap could fill multiple posts so I'll leave that for another time, but any admin worth his or her salt will take the time to learn how to use both of these programs. They will both serve you well in your day-to-day troubleshooting and have helped me solve many problems over the years. **Layer 4 - Application** Finally we get to the top - layer 4, a.k.a. the Application layer (layers 5, 6, and 7 of the OSI model - Session, Presentation, and Application respectively). This is where all of our upper level protocols - DNS, HTTP/S, SMTP, SMB, NFS, etc. - reside. Since I could write a series of posts on any one of these protocols, I'll illustrate two examples that I use very frequently. First up, in reference to this post's title, sometimes DNS is the problem, in which case the `dig` command (part of the `bind9-dnsutils` package on Debian, downloadable on Windows - instructions [here](https://phoenixnap.com/kb/dig-windows?ref=thesoloadmin.com)) is THE tool for the job. For example, if we wanted to view all the DNS TXT records for thesoloadmin.com domain, we could do it easily via the command `dig thesoloadmin.com txt`: ![](https://thesoloadmin.com/content/images/2022/04/dig.png) Dig Command Output Julia Evans has a very nice `dig` tutorial on her blog [here](https://jvns.ca/blog/2021/12/04/how-to-use-dig/?ref=thesoloadmin.com). I use `dig` all the time, especially when I'm adding/updating DNS records to ensure that propagation has completed. I also use it internally to troubleshoot a variety of Active Directory issues, as AD is intimately linked with DNS. Finally, I'm going to walk through the steps used to send an email via `telnet`. I've used this particular example probably as much as anything else in this article over the years (with the exception, obviously, of `ping`). When troubleshooting mail delivery, this is a handy sequence of commands to know. We'll keep it very basic and assume that we're testing against an internal server and our IP address is allowed to connect and send mail. The sequence is as follows: - `telnet mail.thesoloadmin.com 25` \- Initiate a connection to the mail server on port TCP/25. - `ehlo your.hostname.com` \- Issue the EHLO command to identify the connecting host. The server will respond with a list of capabilities. - `mail from: ` \- The "mail from:" command begins the sequence to send a message. The server will respond with `250 Sender accepted`, an error indicating that authorization is required, or possibly another message. - `rcpt to: ` \- The "rcpt to:" command defines the recipient. Again, the server will respond with either a successful message, an error indicating that relaying is denied, or possibly another message. - `data` \- If the previous commands have all been successful, the "data" command indicates that the following lines will contain the message data. As shown in the screenshot below, the SMTP protocol remains very basic - enter "Subject:" to set the subject line of the message, press enter, and then begin typing the message body. After completing the message body, go to a new line and enter a single "." followed by another carriage return. The single period signifies the end of the message data. ![](https://thesoloadmin.com/content/images/2022/04/mail_via_telnet.png) Sending Mail via Telnet Having obtained both the Cisco CCNA and SANS GCFW certifications early in my career provided me with a solid foundation in both networking and firewalls and imparted a lot of skills that I still use on a regular basis. Whether you're a solo admin running your own small network environment, a developer spinning up and managing containers, or a server admin working with VM's, network troubleshooting is a critical skill that you'll probably use more often than you realize. If you have questions or comments about any of the topics in this post, feel free to email me - [matt@thesoloadmin.com](mailto: matt@thesoloadmin.com). ### Batch Downloading With Python URL: https://thesoloadmin.com/batch-downloading-with-python/ Last updated: 2022-08-11T22:01:55.000Z [CodeProject](https://www.codeproject.com/?ref=thesoloadmin.com) Today one of my user teams won their bid for a new project with one of our best customers (yay!). Our projects always start with purchase orders and when the customers don't have EDI or an API available (most of them), our project expediters have to download the customer's purchase orders and manually enter them into our system (boo!). All the reasons why this is so are a topic for another day. Today's problem is that our new project already has 242 purchase orders that need to be downloaded and entered into our project management app, so at the very least I can help retrieve the PDF's from which they'll do the data entry. First, a little background - this particular customer is using a new system called [Modern PO](https://www.modernpo.com/?ref=thesoloadmin.com) to track their PO's. The site is nice - a simple and straightforward user interface and basically what you'd expect from a modern form-based website. ![](https://thesoloadmin.com/content/images/2022/03/modern_po_home-1.png) Purchase Orders List Page Each PO has the PO ID field linked to its details page and each details page contains a set of tabs along the top. The only tab we're concerned with is "View", as this tab contains a "Download" button which will give us a PDF version of the purchase order. ![](https://thesoloadmin.com/content/images/2022/03/modern_po_details_page.png) Purchase Order Details - "View" Tab I thought perhaps there would be a site API that would allow me to easily request the PO's I needed, but no such luck. So the task today is to write a script that can access the PO list page and for each PO, access its details page, "View" tab, and save the file linked to the "Download" button. Not super simple but not rocket science either. For tasks like this, my GOTO language (ha!) is Python - [https://python.org](https://python.org/?ref=thesoloadmin.com), specifically Python 3 as there's no sense writing new code in Python 2 at this point. In this article I want to walk through the process that I use to actually develop a script like this, for anyone who hasn't done a whole lot of scripting and/or may be intimidated by this type of project. Rather than just giving you something you can copy-paste (although you will have that also), I want to show the steps that it takes to build up your own scripts. Before you know it you'll have your own script library to draw from. I go through basically the same process regardless of the language. *NOTE: The remainder of the article assumes at least a working knowledge of Python. If you've never written anything in Python before, the [learnpython.org](https://www.learnpython.org/?ref=thesoloadmin.com) site is a great place to start.* So first off, go ahead and download the finished script so you can follow along: [ Batch downloader batch\_downloader.py 2 KB download-circle ](https://thesoloadmin.com/content/files/2022/03/batch%5Fdownloader.py "Download") If you're on Windows, go ahead and open up your preferred Linux distro in WSL (for this post, I'm using [Debian](https://www.microsoft.com/en-us/p/debian/9msvkqc78pk6?ref=thesoloadmin.com#activetab=pivot:overviewtab)) and make sure you have Python 3 installed with the command `python3 --version` ![](https://thesoloadmin.com/content/images/2022/03/Fig1.png) If you get a "command not found", you'll need to install Python. In Debian/Ubuntu, a simple `apt-get install python3` should do the trick. Starting with a blank slate when scripting, it's good practice to include the interpreter as the first line in your script. This allows you to run it from the command line without needing to specify the path to Python each time. Just to make sure, I use the `which` command to verify the path to my executable, in this case: ``` (debian) mpogue@darkstar [~]$ which python3 /usr/bin/python3 (debian) mpogue@darkstar [~]$ ``` Now that we have Python 3's path, the first line of our script should read: `#!/usr/bin/python3` For those of you who don't know, the "#!" symbol at the beginning of the line is called the "shebang" or "hashbang" symbol. An excellent explanation of the symbol and why you should include the interpreter line can be found at linuxhandbook.com - [](https://linuxhandbook.com/shebang/?ref=thesoloadmin.com)[https://linuxhandbook.com/shebang/](https://linuxhandbook.com/shebang/?ref=thesoloadmin.com). Next up in your Python scripts, we need to import all the required modules we need (details of the Python import system [here](https://docs.python.org/3/reference/import.html?ref=thesoloadmin.com)): ```Python from bs4 import BeautifulSoup import urllib3.request import re ``` 1\. [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/?ref=thesoloadmin.com) is a FANTASTIC Python library for parsing and dissecting HTML documents. If not for its abilities, this project would no longer have been classified as "little". 2\. [urllib3](https://urllib3.readthedocs.io/en/stable/?ref=thesoloadmin.com) is pretty much the defacto HTTP client library for Python 3 and is used throughout much of Python's core, including pip. 3\. [re](https://docs.python.org/3/library/re.html?ref=thesoloadmin.com) is Python's built-in regular expression library. My guilty admission? I've struggled with regular expressions since the day I wrote my first line of code and will continue to struggle with them until I keel over at the keyboard. Now let's start diving into the script itself. First up, I declared a variable for the site's base URL. Not only does it save me from having to type it multiple times, it also makes it easier to change in the future if/when I end up reusing the script. I also created a variable to hold the "SessionId" cookie that needs to be submitted with each request. Finally, I instantiated a new `PoolManager()` object; [according to the urllib3 docs](https://urllib3.readthedocs.io/en/stable/user-guide.html?ref=thesoloadmin.com), the PoolManager object "handles all of the details of connection pooling and thread safety so that you don’t have to". ``` # The site's base URL base_url = 'https://www.whateverwhatever.com' # Session Cookie, retrieved from the browser after authenticating successfully. We'll need to submit this with each request. session_cookie = '_SessionId=cmKMxrYDzpULXcMbwtuYDNjzdRCWdGS9xgOPUQtbyAdjyu4LvlPylF3ICxVj3V7NSs%2BliTKtRCRNbEc8BhzCdMGeHWJyyT8n0NEaJ7DKU20TzsuD9FZtMbH5od4xhKrE96vlqDvuPEYegbPtL14Of%2BZZsCI4jXCRRcSk%2FojgBYg%2Bwf%2FICDk3MM5STbkkLvWFXR8PK0Xvg6DBy0mnzR2t2jBh7mPijOLFiFRiVriwze8Xkci2QDmziMrclTxHCMWCqjERGFs4wxwJ9f%2BiWq1Y7CEZx5W5GmEyrwRRUJwVGu2dk%2Bfr82Jr3L09GKB1--y9bG00G4CVBMovga--3DplyhIxzHXKYRswjnQfBw%3D%3D' # New instance of the PoolManager() http = urllib3.PoolManager() ``` Before we go any further, now is a good time to fire up an interactive Python shell in your WSL terminal. From the command line, simply run `python3` and you'll be dropped into an interactive shell where you can run Python code. The interactive shell is yet another reason why Python is a fantastic language to work with. After entering the shell and executing the lines from the script so far, you should see something like this: ![](https://thesoloadmin.com/content/images/2022/03/python_shell-1.png) Python Interactive Shell As an aside, I've been using Visual Studio Code to write my Python scripts for the last year or 2\. It's really hard to be the syntax highlighting/code completion it provides, but with Python, you can code in anything from Notepad (no idea why you would) to any of the wonderful IDE's out there. Support for the language is pretty much universal across all operating systems, which is another reason why it's great for this type of task. If you're looking at what we've done so far with the script, and you're not overly familiar with web-related programming you might be asking "where did the `session_cookie` variable come from"? Here's how I determined the value to pass for my session variable. In Firefox, I installed the [Cookie Manager extension](https://github.com/Rob--W/cookie-manager?ref=thesoloadmin.com) which allows me to quickly and easily view all cookies for a given website. Typically, most sites are going to set a cookie containing session details and the modernpo.com site is no different. After going to the site and logging in, click the Cookie Manager button and choose "Open Cookie Manager for the Current Page". The displayed "Name" and "Value" boxes give you the values you need to use. ![](https://thesoloadmin.com/content/images/2022/03/cookie_manager.png) Firefox Cookie Manager Extension Next up in our script, we're going to create a new request variable using our `PoolManager` instance, along with a new instance of the BeautifulSoup `html.parser`: ``` # Create the request req = http.request('GET', base_url + '/my-company/purchase_orders?page=1', headers={'Cookie':session_cookie}) # Create the HTML parser soup = BeautifulSoup(req.data, 'html.parser') ``` Notice that we're sending the HTTP header "Cookie:" containing the `session_cookie` variable we defined previously. This is the point where - if you're running the script line-by-line in the Python shell - you can start to do some troubleshooting. For instance, after you create the `req` variable, it will hold an `HTTPResponse` object containing the full HTML document that you can view by printing the `.data` property of the object like so: ``` >>> print(req.data) b'\n\n \n Modern Purchase Order\n \n\n \n\n \n \n \n\n \n \n \n\n \n\n \n \t