As usual I have an aversion to creating “pet” VMs, and as such I decided to deploy my Forgejo runners on RKE2. Sticking to the Rancher philosophy of “cattle not pets” (a philosophy I strongly agree with), I decided to go this route. Of course deploying Forgejo runners in Kubernetes comes with its own issues, mainly, unlike GitLab, Forgejo can not create pods in a cluster natively. The recommended way to deploy runners in Kubernetes is via DinD (Dock in Docker), this means my workflows will be in containers, in an DinD container (on RKE2), in a container (on Harvester, also RKE2). It really is containers all the way down!
To save you from thinking too hard, dear reader, the answer is: incredibly lazy, I am pretty sure that’s how I got into this career in the first place. As the lazy man I am, I have become tired of remembering to restart my deployments, and spell check my posts when I make changes to this site. So I did what any proper lazy person would do, I spent more time automating my tasks, rather than just being careful, and diligent.
As much as I love Proxmox, if it is anything, it is basic. This is a double edge sword of course, it is rock solid and stable (foreshadowing), and it is also exceptionally basic with few features I want, especially as a Kubernetes user.
I had been planning to move from Proxmox to Harvester (again) for a little while, however this was accelerated by a bug1 I happened across when upgrading from v8 to v9. The Ceph cluster became unstable post upgrade, with the manager services seg faulting in a continuous loop. Turns out this was an issue with the python interpreter2 and before the solution became available my Ceph cluster toppled over and became totally unresponsive, and I certainly played a role in the final failure (whoops). At this time I decided to simply tear down the Proxmox cluster as a whole and begin setting Harvester up.
I recently needed to flash some Sonoff S31 power switches, and realized I forgot to write down what I did and how to do it. Thankfully the process (thanks to ESPHome) is exceptionally easy. So this is a quick write up to explain the disassembly and flashing process.
Disassembly
First pull the grey power button plate off the S31 switch, this can be done with a small flathead screwdriver.
In the previous post (Installing NixOS on BTRFS: Part 1) I realized only after installing NixOS that I had forgot to encrypt my installation, so I have decided to reinstall. This is a small follow up to that post.
Setting up the storage
Like before I opted for BTRFS RAID-0 with two 1TB NVME SSDs, and I made a small change, I will not be using a swap partition this time around. After some consideration I don’t see the value in adding swap to my HTPC system.
It has been many months since I had decided to update my HTPC running NixOS. Sadly, failing to update the HTPC machine as new NixOS versions became available left my machine’s OS version too far back and now no longer compatible with the current version of my NixOS config. I was left with many errors when attempting to upgrade and some may not have been resolvable due to some packages requiring newer compilers that were not available on the system. After a couple hours of debugging I realized this was not worth the effort and decided to simply scrap the system and reinstall. This new install will be just about the simplest install possible, so it is a good starting point.
If like me you have a very small Synapse deployment but have joined some very large rooms in the past and left since, you may have your server reaching out still to those previous rooms servers. I discovered this when I noticed my IDS/IPS was catching connections outbound to certain GeoIP restricted locations (Iran, Russia, China, Saudi Arabia, etc), thankfully most of the destination ports were 8448 (the default synapse port) so it was fairly obvious what the service was. At first I attempted to take a look at the database and I saw about 10k lines worth of destinations Synapse was reaching out to, I had joined some large rooms so this was not shocking. This is generating many false positives and I left these rooms a long time ago so there is no need for my servers to be communicating with them. Below is how I cleared the rooms from my server.
In my previous post “Backup Server Woes” I finally setup an S3 backup box, with some trials and tribulations of course. Sadly through this process I learned a couple things about my backup strategy I was currently using and they weren’t good. Previously I was using K10, K10 has a nice little web UI I can proxy through Rancher MCM, which makes management nice as the UI never gets exposed other than through the Rancher UI.
So whats wrong with K10? Technically there’s not really anything wrong with K10, but the combination of K10 + Ceph seems to be the problem. K10 creates a PVC to offload data to your destination, which isn’t abnormal, but its the way it does this that confuses and annoys me.
It occurred to me that I am long past due for deploying an actual backup box for the lab. I have for years now relied either on a third party (S3), external HDDs, or my desktop (what else will I do with 20TB of ZFS?), to temporarily hold data before moving it back to what ever cluster or application I am going to redeploy. This mean I actually haven’t had a true reliable device dedicated to backups, ever. While my site here may not show it (yet), in the past year I have redeployed my lab four times so far. This is going from bare metal rke2, harvester, harvester (again), to Proxmox, and before that in the years prior the story was roughly the same. At this point I think I have the janky way of redeploying a cluster down to a science, but its time for that to change.
As discussed in the previous post I set out with redeploying my cluster and intended on enabling my cluster with backups. This time around I wanted to try out Kasten K10 (from Veeam), I fully admit at times all I want is a GUI. I just want to take a quick peak and make sure everything is good to go. This is why this time around I decided to give K10 a shot, Velero works well enough but the eye candy caught my attention.
