The video version is about six and a half minutes: watch it here.

Speaking as someone who maintains SCADA systems: our HMI software gives us good at-a-glance insight into our processes and our controls hardware (if we did it right 😅), but it rarely gives us any visibility into the computer and network hardware it runs on.

At Joy Automation we design and configure server platforms for industrial automation, from single nodes to multi-node, fault tolerant clusters with software defined storage. In September we built a three-node cluster, and I wanted real observability for its servers and network. I did install Prometheus + Grafana, but it was also a great opportunity to develop SNMP and Redfish drivers for Nautilus, and an alarm system built on templates and YAML. So I pursued that in parallel.

Why 3D

I went with 3D (three.js via Threlte, a cleverly named Svelte library for it), because what better way to know what's going on than to see the information as overlays directly on the physical equipment. The rack shows the three servers and three switches, with the cables drawn as blue pipes between the ports they're plugged into. You can zoom and pan around it to see it from every angle.

When you click on a server it slides out on its rails like it would in real life, keeping its cable connections intact. We read which drives, memory, CPU, network cards, fans and power supplies are actually installed from the BMC over Redfish, and render them dynamically. Then there are the overlays:

• X-ray: see through the chassis to the components inside

• Identify: what every internal component is and where it's located

• Heat: each component coloured by its temperature, so you can see how close it is to its rating

• Exploded: the components pulled apart so they're easier to get a good look at

• Free: the empty drive bays and memory slots, and how loaded each power supply is against its rating

The mesh view shows the system as a 3D network diagram: the switches in a ring, the servers connected to each switch, and labels showing which port each one is plugged into.

We also collect VLAN information over SNMP, so there's a VLAN view that shows each VLAN as a floor of a building with every device mapped out. On this cluster, 20 is host management, 21 carries the VM traffic, 22 is out of band management for the BMCs, and 251 is the ring's control VLAN, which only the switches are on.

It's just a Nautilus project

The whole monitoring system is a Nautilus project. The two drivers gather the data: SNMP for the switches and Redfish for the servers. The logic is Structured Text, written in VS Code with the Nautilus extension. It does aggregates, like how many devices are online or how many ports are active, and algorithms for harder problems like a broadcast storm or a flapping port.

The alarms are rules. A rule matches a tag type and a member, so SwitchPort Down covers every port on every switch, with a fifteen second on-delay. Ten rules cover the whole rack.

Developing after it shipped

We wanted to keep iterating on it after the cluster left, so before it shipped I recorded 68 hours of everything we monitored. From that I built a simulation: containers on my desktop that act as the SNMP and Redfish servers, publishing realistic values over time. That way I could keep developing while the servers were en route and being installed. Everything in the screenshots is that simulation.

Then I made it so I could force values over the recorded ones, to mimic problems, see what they look like and iterate on the interface:

• Pull the cable on switch three, port 25. The link drops when the SNMP driver reads it and the cable turns orange, and twenty seconds later the alarm symbol shows up next to the port. Plug it back in and it clears, but it stays in the list until it's acknowledged, just like a plant alarm.

• Take a whole switch down. It stops answering SNMP and goes dark. Instead of an alarm per port, which would be 28 alarms, the lower alarms are conditional, so you only get the other two switches saying they lost their neighbor, and the ring failure.

• Flap a port. It never stays down long enough to alarm as link down, so instead we look at how much it has changed state over time. A link down alarm with a debounce would miss it.

It's responsive too, so it works pretty well on a phone: one finger to orbit, pinch to zoom, two fingers to pan.

Joy Automation designs, delivers and supports server platforms for industrial automation systems, from a new build to long-term support of the one you have. If you're in need, get in touch.

Nautilus docs: nautilus.joyautomation.com · Repo: github.com/joyautomation/nautilus

Subscribe to the weekly newsletter to keep up to date: joyautomation.com/newsletter