It was time to get a handle on BACnet traffic at Penn State.
BACnet is a communications protocol for building automation and control (BAC) systems such as heating, ventilating and air conditioning (HVAC), lighting, access control and fire detection. Penn State standardized on BACnet because of its openness.
“Any device, any manufacturer – as long as they talk BACnet, we can integrate them,” says Tom Walker, system design specialist in the facility automation services group at Penn State. “It’s a really neat protocol, but you have to know the quirks that come with deploying it, especially at scale.”
One quirk is that BACnet is prone to broadcast storms. And with hundreds of BACnet systems running across multiple campuses, openly traversing the network, Penn State was worried about degrading performance in other parts of the network and exposing potential security vulnerabilities.
“I took over this infrastructure about four years ago, and I came into a flat, Layer 2 network spread across the whole main campus,” Walker says. He and his team decided to segment the BACnet traffic from the university’s shared infrastructure to improve security and manageability.
Penn State's facility automation services group handles functions including automation engineering, network administration and monitoring the university’s digitally connected buildings.
Building-automation systems allow the team to remotely control HVAC systems, for example, so classrooms and offices are appropriately heated and cooled for students and staff. Automation controls also help ensure the safe operations of critical research laboratories.
“We have glacial ice – it is irreplaceable,” Walker says. “So we have monitoring on the freezers to make sure those don’t defrost themselves.” There’s also an atomic clock in the basement of one Penn State building, where the temperature is controlled down to one-tenth of a degree.
Facilities automation includes IoT
The facility automation services group covers a lot of area – Penn State has 32 million square feet of buildings, spread across dozens of state-wide campuses and 22,000 acres of land.
Within 640-plus buildings, there are scores of systems, devices and sensors that monitor conditions. And the data they collect continues to grow.
“We used to be just building automation. But over the years, we’ve started incorporating more components. We now manage the networks for all the different utilities – waste water, water treatment, steam plants, electrical distribution, even chill water distribution,” Walker says.
“It’s been an evolution of our group. We’re taking on more stuff within the building –everything that keeps the building running. And we’re bringing that data back through our network infrastructure and into the data center, and then either jumping it up to the cloud or passing it over to other analytic systems to analyze the data.”
The facility automation group started tracking elevator use, for example. “We found one elevator that did 1,900 trips in one day. It was insight we never had before, and it explains why that elevator is constantly breaking down,” Walker says.
It sounds a lot like IoT, but to the facilities automation team, it’s just business as usual. “This IoT business – it’s a buzzword,” Walker says. “We’ve been doing IoT forever. That’s what building automation is. It’s taking controls in the building and bringing them back through a network, so we can remotely manage that building.”
Microsegmentation a better fit than VLANs, firewalls or ACLs
A key driver of Penn State’s decision to upgrade its building automation systems was the need to secure communications in the field. The BACnet infrastructure was a web of direct wireless- and cellular-network connections. Access control was a concern.
The university’s ongoing $3.28 billion capital spending plan – focused on the renewal of existing facilities and systems– has meant near constant construction for the last few years and a steady stream of contractors coming in and out of telecom rooms, tapping into building automation systems and installing rogue access switches and wireless access points. “These contractors would bring in their own switches, plug them into our flat Layer 2 network. Or they would add their own access points and wireless routers,” Walker says. “It was very complicated to manage.”
It was also a potential attack vector, which the building automation team aimed to eliminate. “That was the main purpose: to try to figure out the best way to clean up the network and secure it by the same token,” Walker says.
The solution was microsegmentation, which allowed Penn State to reduce its network attack surface and centralize control of hundreds of BACnet systems – without disrupting the legacy network during the upgrade.
Generally speaking, microsegmentation allows organizations to isolate workloads from one another and secure them individually. It enables a more granular partitioning of traffic than traditional network security techniques such as firewalls, virtual local area networks (VLAN) and access control lists (ACL).
Organizations can tailor security settings to different types of traffic, for example, creating policies that limit network and application flows between workloads to those that are explicitly permitted. And if a device or workload moves, the security policies and attributes move with it. The goal is to decrease the network attack surface: By applying segmentation rules down to the workload or application, organizations can reduce the risk of an attacker moving from one compromised workload or application to another.
For Penn State, part of the appeal was the ease of deployment and management, which meant the facilities team could manage the network re-architecture on its own. “We looked at creating separate VLANs or [private VLANS] within the buildings, doing MAC filtering, doing access control lists, or doing building-level firewalls,” Walker says. “But when we started looking at scalability, it gets crazy. For some of those options, I’d have to hire at least two more people just to manage the toolsets.”
The university instead chose microsegmentation technology from Tempered Networks to isolate and cloak its BACnet traffic. The vendor’s HIPswitch devices create a secure, private overlay network on top of the physical network, and only explicitly trusted systems or endpoints are allowed onto the overlay. The HIPswitches work in conjunction with Conductor, Tempered Networks’ centralized orchestration engine that creates, manages and monitors device configurations and security policies.
A proof-of-concept helped shape the Penn State deployment. The team originally considered deploying HIPswitch devices down to the individual controller level but decided to move up a layer – to the building level – which greatly simplified management, Walker says. “We’re able to create groups of controllers and groups of servers, and then tie them together very easily right through the trust relationships within the overlays.”
The individual building systems are locked down based on functionality, not by port. “If one of those controllers gets compromised, they only have access back to the data center, to one server. Whereas before, if somebody compromised a building, they could get access to the data center and all the applications that were exposed," Walker says. "Now we’re able to say: ‘the lighting controller can only talk to the lighting server. The elevator controller can only talk to the elevator server.’”
Tempered Networks’ technology “creates trust between those individual buildings and our data center.” It also cuts down on traffic between buildings, which was a problem before the re-architecture. “Every single building on the big, flat Layer 2 network heard every bit of the communication. Now only the building and the server are communicating.”
In case of BACnet broadcast storms or an intrusion, it’s easy to shut down traffic from a single building, Walker says. “Before, because it was a big, flat Layer 2 daisy-chain all the way out, I’d have to shut off maybe multiple ports. Or if I shut off one port, it might shut off half a dozen buildings. Now I can individually control traffic from each of the buildings.”
The technology also makes it easier to do adds, moves and changes; Penn State can provide secure contractor access to specific devices on the network via centralized orchestration.
“I wanted something that we could easily deploy and instantly secure the infrastructure down,” Walker says.
Early successes led Penn State to expand the project. Initially, the facilities team planned to deploy HIPswitches at University Park, the largest campus in the Penn State system. Now the university is extending the project to the rest of its campuses state-wide.
As Penn State continues to build out its campus properties, the flexibility of the HIPswitches is enabling the facilities team to connect to even the most remote sites. Just recently, Walker’s group deployed a HIPswitch with cellular connectivity in a building in the middle of a cornfield, where it would have taken a lot more time and money to establish a fiber connection.
That cellular capability is a feature Walker finds more useful than he initially expected. “We’re able to deploy in places that we weren’t able to get to before.”