Last week, one of those hidden risks that people don't often think about reared its ugly head. My company lost its building access control system to age and neglect. Suddenly, some of the automatic badge readers that unlock doors in our headquarters building stopped working, and people couldn't get past the doors they are used to walking through every day.
It all started with a PC that crashed. This very old and perfectly ordinary desktop PC was used by our facilities department to manage all of our building's door locks. It was definitely old school: a gray metal case containing a computer and its power supply, wired up to some peripherals and an old CRT monitor. It sat on the floor of a storage room, accessed only rarely by facilities staffers who had to add new employees' badges to the access control system or change areas of access for existing employees.
Just how old was it? I don't know. Very, very old. And dusty. And, now, dead.
Its fate was hardly surprising. The hard drive crashed. When that happened, our building's badge readers and door locks went haywire. They shouldn't have, but they did. According to the security consultant who was brought in to deal with the problem, a total failure like that is not supposed to happen. Modern building access systems store the access rules for each door in the badge reader itself, which has enough memory to store all the badge numbers allowed through. Modern door locks also have backup power supplies attached to them, so the doors will still work during a power outage. And if there is an extended power outage or other systemic failure, the doors will "fail open," so that people can get out of the building in an emergency. But our system was old, and poorly configured, so things didn't go that smoothly for us. Some of the doors did indeed continue to work, but not all.
But that's not the worst of it. That old and dusty PC, the one on which all our building access depended, was not backed up. And, because there was no high availability or redundancy built into the system, it could not be recovered. The consultant had to install a new computer, with updated software, and create an entirely new configuration for all the door access rules. Which, if you think about it, puts us right back where we started.
How did this happen? IT professionals know that critical systems such as building access belong in a data center, where they can be protected by reliable, conditioned power, cooled with modern air conditioning, and closely monitored by automated systems that will alert support staff when something starts to go wrong (like impending hard drive failure). And IT professionals usually provision additional systems for redundancy, in case one fails as this one did. Not to mention configuration backups so the system rules don't have to be rebuilt from scratch.
Yes, IT professionals know to do all of those things, but IT professionals had nothing to do with that rickety old computer in the storage closet. Our facilities department was responsible for building and maintaining the building access computer. But the facilities staff aren't conversant with the standard practices to protect critical systems, and they didn't think to consult with IT about it. So when the inevitable happened, we all experienced the consequences.
Lesson learned? In the short term, I'd say yes. Our facilities staff are somewhat better educated now about best practices for critical systems. But that hard-earned knowledge is almost certain to slowly be lost over time as normal staff turnover takes place. And even though they aren't technical, the facilities staff refuse to relinquish ownership and management of the building security system. They see it as their domain, despite its technical nature.
And as long as technologies continue to be territorially separated, they will be dependent upon the expertise of the few, instead of the many.
This week's journal is written by a real security manager, "J.F. Rice," whose name and employer have been disguised for obvious reasons. Contact him at jf.rice@engineer.com.
Join in
To join in the discussions about security, go to blogs.computerworld.com/security.
Read more about security in Computerworld's Security Topic Center.