Companion post for the cluster video. The video covers the same ground on screen in about fourteen minutes: watch it here.

During my first few projects as a control systems engineer, virtual machines weren't common in control systems yet. We'd have operating systems crash and software get corrupted, and we'd have to rebuild from scratch or restore from backups. It always happened at the worst time: mid-startup, or while the plant was running, and it took a while to fix.

So it was great when virtualization came around. We could build the operating system in a virtual machine, snapshot it regularly, and if anything happened, roll back to a better time. But if a virtual machine is running on one physical server and that server dies, every virtual machine on it goes down. So to achieve high availability, we cluster the physical machines. Instead of one server running your virtual machines, you have three or more, and if one of them dies, its virtual machines start back up on the others.

Putting one together is pretty complex, though: multiple servers, the networking between them, and a lot of software that all has to be configured right. So if I'm handing over a full turnkey cluster, I need to prove it works the way it should, and the only way to test redundancy is to break it on purpose. This is how a cluster is put together, layer by layer, and how we test every layer before it ships.

The layers

A cluster is made up of layers of technology. At the bottom is the hardware, the servers themselves, which we call nodes. Then the network that connects them, the storage that's shared between them, and the compute layer that runs your virtual machines. On top of that are your workloads. And under all of it, power.

Which technology goes in each layer depends on the platform. Our most recent clusters are built either on Linux, with Ceph or LINSTOR for storage, Incus for the virtual machines, and Kubernetes, or on Windows, with Storage Spaces Direct and Hyper-V. Which one you end up on is really a matter of client preference. If I'm making the recommendation, I prefer Linux with all open source, because I can't stand spending money on licenses (among other reasons), and Windows Server Datacenter is very expensive. If you want Kubernetes, or you're building your own software, Linux is great. Windows makes sense if you're mostly using Windows technology like Active Directory, which you're probably already using for IT, and traditional industrial software is all Windows and sometimes needs SQL Server.

The nodes

A node is just a server, and we build clusters with at least three of them. Most clusters for SCADA are small: three, four, maybe five nodes. Each one has two power supplies, its own drives, and network ports for two separate purposes: two trunk ports out to the switches, which carry several networks over one cable, and storage ports that go straight to each of the other nodes.

We size the cluster for the workload: every virtual machine, Kubernetes if we need it, the database, the whole list. Then we add up what each piece of software needs and make sure everything can run at the same time, even if we lose a node.

The network

On a three-node cluster we have three switches, a switch per node. The switches carry everything except storage: management, your virtual machines talking to each other, and connections out to the plant. Each node's two trunk ports are 10 gig uplinks, each one to a different switch, so if one cable or one switch fails, the node can still reach the other nodes. The other two 10 gig ports on each switch connect the switches to each other in a ring.

On that ring we run ERPS, Ethernet Ring Protection Switching, a vendor-neutral ring protocol that's mechanically similar to Spanning Tree. Three switches cabled in a ring make a loop, and without a ring protocol, traffic would circle that loop and multiply until it floods the network: a broadcast storm. So ERPS blocks one of the links, and if another link fails, it opens the blocked one and traffic goes the other way around. The difference from Spanning Tree is speed: ERPS switches over in under 50 milliseconds, against 30 to 50 seconds for classic Spanning Tree, or under a second to a few seconds for Rapid Spanning Tree.

Then we break it. Pull one of the cables between the switches, and traffic across the ring drops for about 60 milliseconds, right around what we expect. Cut the power to a whole switch, and it's half a second before ERPS has rerouted around it.

Storage

In the early years of virtualization, we'd use a NAS or a SAN: one central storage box that every server depended on. Since then, hyper-converged solutions have come out, where you use the storage that's built into the cluster nodes, and now there are very good open source options like LINSTOR and Ceph. Hyper-converged storage writes every piece of data to more than one node, so any one node can disappear and your virtual machines still have their disks. That's also what lets a virtual machine restart on another node: its disk is already there.

When all the nodes are in one rack, the storage cables run in a mesh straight between the nodes at 25 gig. Nothing sits in between, like a switch, so storage doesn't care about a switch failure or have to reconverge when one goes. If I pull a drive out of a running node, the pool degrades, nothing loses a ping, and nine seconds after it's back in, it's healthy. Pull a storage cable, and nothing degrades at all.

Compute, and why you need three

The hypervisor, Hyper-V on Windows or Incus on Linux, runs your virtual machines and keeps track of which ones are running on which node. Our typical cluster has an Ignition server, a database, Active Directory or its Linux equivalents, monitoring with Grafana, Prometheus and Alertmanager, maybe a virtual router or firewall, and usually an engineering workstation for the PLC software.

If a node dies, its virtual machines get restarted on the nodes that are left. To do that, every node checks in with the others about every second. When a node stops answering, the rest can't tell if it's dead or just cut off from them, so each node gets one vote, and only a group holding more than half the votes is allowed to act. That majority is called quorum. With three nodes, if one drops out, the other two still hold two of three votes.

With two nodes, if the link between them breaks, each one has exactly half, and neither has a majority. If they both kept going, you could get the same virtual machine running on both nodes, writing to the same disk, and that's how you get data corruption. That's called split brain. So you need a third vote: a third node, or a small witness somewhere else. The same tie comes up when a cluster is split across two locations, and the answer is the same: a tiebreaker in a third place.

To test failover on a Windows cluster, we cut the power to a node running a domain controller, and it was back up on another node in 76 seconds. On a Linux cluster, a SCADA virtual machine came back on another node in 79 seconds. It isn't instant, because the cluster first has to decide the node is gone, and then the virtual machine boots from scratch. For planned maintenance we don't wait for that: we live-migrate the virtual machine while it's running, and on our Linux cluster the pause was about a second and a half, without losing a ping.

Kubernetes, and the four-minute bug

On the Linux clusters, Kubernetes runs in virtual machines on the hypervisor. It gives anything native Linux, like Nautilus, health checks: if a workload crashes or freezes, it's restarted, and if a node fails, it moves. Most of our clusters use it for the database: TimescaleDB as a Postgres cluster across the three nodes, one primary and two replicas.

The things that talk to it, like your HMIs, don't talk to one node. They talk to one address, and the cluster keeps that address on a node that's healthy. If that node dies, the address is supposed to move within seconds. So we tested it: we hard-reset one node at a time, and on one of them, the address went dark for four minutes. The other nodes marked that node as failed within six seconds, but none of them took the address over until it finished rebooting.

The cause was one label. The control-plane nodes were marked to be excluded from external load balancers, so the one worker was the only node that ever announced the address. One patch to remove the label, and the same test came back in about six seconds.

If that bug had shipped, the likely thing is nobody would ever have caught it. The worker reboots for patches, the gap happens, it comes back four minutes later, and everybody keeps working. In a regulated industry, that could be a four-minute gap in the data, and a fine. There's way too much configuration in these clusters for us not to run a battery of tests before we ship them.

Power

A UPS keeps the cluster up through a short outage, and it also tells the servers they're on battery, over a network card or a USB cable. Software on each node watches for that and starts a timer, so a short blip doesn't shut anything down. If the power is still out when the timer runs out, each node shuts down in order: the virtual machines cleanly, and then itself. That feature is called a graceful shutdown. Without it, a server can die in the middle of writing data, and whatever is in flight can be lost or corrupted, even on clustered storage. Last, the UPS turns its own output off, so when the power comes back, everything boots back up without anyone touching it.

The other option is drives with power-loss protection, which have capacitors that let the drive finish writing its cache to flash when the power goes. With a good UPS and graceful shutdown, you'll very rarely need them for that, but they're also faster: clustered storage waits for every write to be safely on disk, and a drive with PLP can confirm that much sooner, so in a cluster they're usually worth it anyway.

We test the worst case too. Cut the power to the whole site, and when it comes back, every node powers itself back on, and the cluster is fully healthy in a little over five minutes.

What ships

When we ship a cluster, you get the test report. It shows how we tested it, what we found, what we mitigated, and that the system passed. On our last Windows cluster, that was 79 automated tests, and nothing failed. And if you're also entering into support, we keep bringing the system back to that baseline.

If you need on-premises compute and you want it fully tested and validated before it arrives on site, for industrial automation or anything else, that's what we do: joyautomation.com/services.

Subscribe to the weekly newsletter to keep up to date: joyautomation.com/newsletter