Inside Symantec's Security Operations Center

The inside of the Symantec Security Operations Center looks like a scene out of the movie "War Games," and in many ways, the connection is fitting. The SOC, as it is known by Symantec employees, is in the business of detecting and analyzing network threats. And as malicious activity online gets increasingly more sophisticated, the war against cybercrime is definitely on.

The US site is one of four SOCs in the Symantec managed security services (MSS) system. Others are in England; Sydney, Australia; and India. All perform identical tasks for clients who pay Symantec for 24-7 monitoring, analysis and response to potential threats to their systems, according to Grant Geyer, vice president of Symantec MSS.

"Our clients are generally large-business customers that need bullet proof security," said Geyer. "A lot of these clients are responsible for huge energy systems, or they are large financial institutions that have a lot of assets at risk. They need real time access to incidents as well as to analysts they can work with on threats."

For the price they pay, these clients get immediate attention. The average hold time for a client calling an analyst at the SOC is 8.5 seconds, according to Geyer. And clients also get familiarity. Analysts are separated into teams and are assigned customers so clients know they will speak to the same group of people whenever they call.

Just getting into the room is a process. The SOC is secured by three different zones. Of Symantec's 17,000 employees worldwide, only 200 have access privileges to enter the SOC.

The first zone one must pass through is an average looking security point at a door with a badge reader and a biometrics scanner. But through that door is an area known as the "man trap," a large, circular waiting area with high walls that conjures up images of Dorothy and her crew waiting to be seen by the Wizard of Oz.

"I am the great and powerful Oz! Who are you?!" I expect the Wizard to boom from a place unseen. But quickly I am taken past security zone two and into a glassed in area with an impressive view of the SOC known as the "fishbowl" where we learn more about the SOC and how it works.

"We have experts looking at customer incidents and responding to them, in real time, to notify them about incidents they need to take care of at that moment," explains Geyer. "We receive over 2 billion security incidents on a daily basis."

Geyer points to a floor of employees.

Page Break

"The analysts are on the left. They are performing the monitoring and analysis," he said. "And on the right are the security engineers. They are responsible for fault configuration performance management of our services. That is, any firewall policy changes, any patching of systems, and any outages on a system that a client might need."

The system provides checks and balances, he noted. Analysts determine if there is a problem worth responding to but are unable to change anything. The engineers take action, if necessary.

The SOC is only one part of the managed security system. Symantec also has network of sensors deployed called "Deep Site." Users can download the agent and see a quick snapshot of current attack and threat trends. And there are response labs. In the labs, employees dissect malware to understand its methodology, how severe it is and then push it back out to customers in the form of products.

That dissection process includes 2 million decoy email accounts, or "honeypot networks," according to Geyer. They are decoy email accounts set up to gauge new kinds of spam. And there are also regional considerations that come into play because malware threats that affect some parts of the world are often unheard of in other countries.

"Vulnerability data is very different from malware which is very different from attack trends. And spam and phishing data are different. So, unless you have purposefully set up ways of getting slices of data, you miss the multidimensional aspect of security threats."

Looking for a needle in a needle stack

Of the 2 billion security logs analyzed by the SOCs each day, there are many incidents that look very bad but that are benign, said Geyer. In fact, about 3,300 are incidents that merit further investigation. But there are many which look benign, that are very bad. About 100 per day end up being severe incidents that need action, which is why Geyer likens the process to looking for a needle in a needle stack.

"If you were to automate this, judging by number of logs we analyze, you would miss most of the problems. It takes an expert to analyze it to see if there is something malicious going on."

We see this process at work by visiting the desk of Analysis Supervisor Tracy Williams, who is reviewing logs and making decisions about what needs further attention.

Page Break

Everything done from his view is source-IP correlated. In other words, he is reviewing which websites are talking to his clients systems and determining if there is malicious activity. Customers register all of their net blocks, so analysts have a sense where traffic is going to or coming from.

Williams points to one incident he is keeping an eye on.

"This is part of our bot-net command and control detection," he explained. "We only see one signature trigger, and only 13 logs. But there is one distinct destination IP address and it's only going across one device. There isn't a lot of data. But based on our work with our Deep Site partners, we know this is an IP address that is doing something malicious. They provide us with a list of suspicious IP addresses."

Whether the incident is a worm infection or other problem still needs to be determined. But Williams said it is ranked as critical level, and the client will be called immediately.

"The idea is if it is malicious, we want to get it to them in real time so they can start remediation immediately. We will wake you up in middle of night and say: "You might want to take a look at this now."

All of this information is stored in the third security zone, the locked server room. The data from the SOCs, as well as the Deep Site network and the security response labs is used to compile a bi-annual report on the internet threat landscape, which is evolving daily, said Geyer.

"If you looked back five years ago, you see on average about 6,000 to 9,000 new variants of malware in each report. But in the past 18 months, the increase is just staggering. It really just shows us how easy it is to write it, and also that there is true financial gain to it. Malware is proving to be good business model for people in the underground economy."