When it comes to the physical security of a data center, the last line of defense is the cabinet. Yet frequently, the security of that cabinet is an afterthought.

Many large data centers are defended with significant security, but once inside some, the security seems to lessen the closer you get to the actual IT infrastructure. Though it is tough to gain admittance to the data center, once someone with nefarious intent is inside, with a reason to be there and a good story, it can be relatively easy to access the data center floor.

Transcript

Kevin O’Neill, Data Center Spotlight: This is Kevin O’Neill of Data Center Spotlight. Our Topic today is cabinet level data center security. Particularly data center security the closer you get to the IT infrastructure itself, to the data center cabinets and data center racks. We have with us today a cyber security research consultant, Mr. Greg Riggs, who before forging out on his own as a consultant was responsible for the security of Microsoft critical assets and data centers. Greg, thank you for joining us today.

Greg Riggs, OSHI Technologies: Thank you Kevin, it is awesome to be here.

Data Center Spotlight: You have an extensive data center security background in the major enterprise world. Tell us a little bit more about your background.

Greg Riggs: I’ve worked as an IT engineer since the mid 90’s although the bulk of that time was at Microsoft managing several Microsoft Research & Data Center facilities. I became the senior Security Operations Manager for the TWC Security division.

If you’re not familiar with TWC or TWC Security specifically, they are basically the global face of Microsoft’s commitment to integrating security design into all phases of product development. My responsibility for that team was for the engineering and operational security infrastructure that ran security research, incubation, incident response, forensics, and most things network security. I’m leaving a few things out, but among hardening of the company’s most sensitive assets you get a pretty good idea.

Data Center Spotlight: It sounds like you had your hands full with a pretty wide range of duties involving security. Did developing that expertise in all those areas of security influence your decision to leave Microsoft and head out on your own?

Greg Riggs: Even though I had one of the coolest jobs in IT, I’ve always had a bit of an entrepreneurial spirit, once I had found my passion for cyber security technology, I decided that after 15 years it was time to start my own firm, OSHI Technologies, based in the Seattle area.

Data Center Spotlight:  The exterior of that data center can have cement barriers, gates, guards, guns; it can have a very strong external security presence almost like a military or airport level of security. Maybe you have to enter a mantrap to get into the building.   But in many instances, it seems once you’re inside the security protocols seem to ease. So the protection of the outer rings, serving as a deterrent, that protection is pretty strong. But once you get inside things can ease considerably.

Greg Riggs: It is interesting you bring that up and talk about those characteristics specifically. Anytime the topic of datacenter security has come up in the past, I find it very helpful to point out that we tend to assume the Hollywood definition of a datacenter as a big, warehouse sized industrial complex hosting hundreds of server racks. While those facilities certainly do exist, it’s important to acknowledge that many organizations host their IT infrastructure onsite, in office buildings, or even in a series of distributed server closets anywhere they can get enough power & cooling… granted these are not “datacenters” in the conventional sense, but very often when people use the term ‘data center’, they intend to convey “the facility where my critical assets and IT infrastructure live”. Does that make sense?

Data Center Spotlight: Yeah, it does make sense, I think I can see what you’re getting at, but for others listening in can you explain why this an important distinction when discussing Datacenter Security?

Greg Riggs: A conventional datacenter generally doesn’t have physical proximity to its customers who are often the owners or administrators of the machines. There will be IT staff working full time in the data center, but they aren’t necessarily the server owners and sometimes they only provide smart-hands support for maintaining hardware. When the ‘customers’ don’t have that convenient access to the physical servers this often creates a problem when resolving critical situations promptly. As a result, many organizations choose to keep their critical assets on-site when they can. If not for convenience, then for modest-sized organizations because it is far more cost effective that way.

Data Center Spotlight: Are there different layers of security required if the servers are hosted in a conventional datacenter vs. being hosted on site?

Greg Riggs: The importance and critical nature of those assets to the organization stays the same regardless of whether they are hosted in a conventional datacenter or locally, although this is a heavily debated argument within IT circles, and I’ve been in those conversations many times over, that’s probably a topic best for another time. My point is that the term ‘data center’, it does mean something different to everyone, so when we’re talking about data center security”, we’re talking about the security of any organizations critical IT infrastructure, which can change the scope of audience for the conversation but ultimately the answer is, no, the security requirements are the same, whether they’re being hosted on site or in a conventional data center.

Data Center Spotlight: I’ve seen good security and poor security in places that just have a data closet, I’ve seen good security and poor security in very large data centers. What have been your experiences across the different definitions of “data center” we’ve both been in?

For someone who isn’t an employee of the site, it’s a scary fact that very often; the hardest part of gaining access to a facility is getting past the front desk. Many organizations are pretty good about requiring escorts for people tagged as “visitors”, but there are a lot of non-employee visitors who may have legitimate reasons for having unmonitored access throughout a building. Vendors such as building facility maintenance techs, you have site security officers, and housekeeping is another good example, and those are just three examples of individuals who nearly always have master keys and are unattended access to the facility after-hours. It can be pretty easy to get into a facility if you’ve got enough planning and enough reason to do so.

Data Center Spotlight: Depending on the protocols of the data center they can sometimes actually have access you to the actual physical servers, can’t they?

Greg Riggs: Oh, absolutely. It’s pretty shocking, but very often the answer is yes. If it’s not enabled for the vendor company as a whole, there are often specific individuals who have been entrusted with access. But keep in mind, the people that own and manage the physical security controls, typically cardkey readers for the room, they have the capability to give themselves or others access at their discretion. All it takes is a bit of crafty social engineering. But ultimately yeah, it is not difficult to get access to the physical servers.

Data Center Spotlight: There are also issues with external visitors. Like someone who is delivering food to the space. I’ve personally seen sandwich delivery people just waved through a data center that was supposed to be high security. I’m sure they were a familiar face but sometimes the protocols ease up just based on people’s familiarity and it sounds like you’re talking about people that can be classified more as an ‘insider threat’, maybe?

Greg Riggs: There is a terrible stigma behind mitigating the insider threat, but yes. As the technology continues to improve and methods are employed to close the gap of protecting the network from external attacks over the internet, the path of least resistance becomes physical access. And as you’ve mentioned… it doesn’t take much to socially engineer your way around a building or into a building regardless of whether or not you work there and patterns catch up with us, so somebody who has been given regular access to deliver food to a data center becomes a vulnerability that can be exploited.

Data Center Spotlight: Why is the insider threat or even just the whole concept of social engineering, people letting their guard down with a familiar face, why is that such a difficult issue to tackle?

Greg Riggs: From personal experience, I can tell you that it’s very easy to spoil a positive working atmosphere or company culture with many layers of policy, process, and access control. Ultimately the primary message of those layers, if you’re not in a supportive culture is, “you’re not worthy” or “we don’t trust you”. Very often, any time the word “Security” comes up in the workplace it is taken as a negative… it implies process that will make my job harder, that I have learn a new way of doing things which will likely change in a few months. You just can’t do your job without worrying about things that will probably never affect me anyway.” That creates a pretty negative culture unless you’ve been in an environment that provides a lot of education, a lot of conversation, and a lot of support for why people need to care.

Data Center Spotlight: I’ve had a lot of people who are responsible for data center security apologetic that I have to go through a security process to get into a data center. Tying this back to data center security and what I like to call the most inner ring where the sensitive servers and IT infrastructure live….with all the people who have the potential to gain un-escorted access to a datacenter, or even a server closet in the an office building or a smaller business… I’m assuming that most servers require a username and password to log on, so short of pulling drives out of a machine… what kinds of hazards are we talking about if someone I don’t know can physically get to my servers? And again, I think I know the answer to this and I think it is going to make people uncomfortable.

Greg Riggs: Well, it is generally easiest to classify the risk into two main categories. The first category is careless negligence, where safeguards were not put in place to prevent a disaster which could’ve been easily avoided. I’ve personally seen janitorial staff with vacuum-cleaner backpacks walk into a hot aisle, which is the narrow space between two server rack rows, and try to turn around in that hot aisle, and when they did, the backpack smashed into a whole bunch of fiber optic connections on a rack whose door happened to be left open. I’ve also seen IT engineers access the console of mission critical servers just so he could access files from a virus infected USB key, only because it was conveniently close to where he was working. In both cases, the individuals had legitimate access to the server room but there were few protections beyond that threshold.

The second category is the popular one, the one of malicious intent. A person that has access to the server room where the virtual ‘crown jewels’ of an organization are being kept. Such a person can have many motivations. The big ones that are classified are Theft, Destruction, Disruption, and Confidentiality. There is too much ground to cover with all these scenarios in one conversation, but one example is that even if a server requires credentials to log on at the console, it’s still connected to a network… and this network is the weakest link. From outside the facility, on the other side of the firewall, things may be well-hardened, although the conventional methods depend entirely too much on perimeter security even at the network layer. Malicious intent with physical access to the local network where the servers live is a playground of unlimited potential, and a motivated attacker will generally make a change so subtle that most major organizations who have been required to report a breach incident to the Department of Homeland Security, find that the breach occurred on average between 2 and 6 years prior to being discovered. Which is kind of scary.

Data Center Spotlight:  Yeah, it is. Let’s take this in a compliance direction. Are current compliance standards and certifications addressing the physical access of IT infrastructure properly, or is this a case where what is needed to achieve compliance is trailing the actual threat and trailing some of the incidents that are actually happening out there?

Greg Riggs: Compliance standards have definitely come a long way, and they do seem to be effective serving as a ‘barrier to the basics’, but don’t seem to be worrisome for threat actors operating in the wild. This is a really big can of worms and we could, and probably should, have a segment all to its own discussing compliance at length, but I will say this much at least… if the IT Managers, the Budget Controllers, and the Auditors were to be communicating more effectively about the risks, and the costs, and the available countermeasures, many organizations would be far better off than they are today. But there is a disconnect between those three groups, who have a very set opinion and a very set degree of resources. I don’t think as an industry, for the organizations that need to recognize and follow data center compliance standards, we’re not where we need to be, or where we should be.

Data Center Spotlight: How did you guard against this threat when you were at Microsoft?

Greg Riggs: In the role that I had, I was able to see a lot of the incidents that were taking place and there were more opportunities to see what was happening.   As I paid attention to some of these incidences that were being reported by organizations from almost every corner of the world, I started searching for solutions that would allow me to secure physical assets at the server rack itself within my own environments initially and then I evangelized that solution to the other groups that I worked with within Microsoft. Some of my infrastructure was hosted in large multi-tenant facilities, the conventional data center along-side many other departments. Some of the infrastructure was at remote sites where my people weren’t able to visit more than a couple times a month. I needed the ability to demonstrate to my stakeholders exactly when and by-whom a physical asset was accessed, because of the unique nature of how we were hosting everything. So it was a bit challenging.

Data Center Spotlight: Greg, I know a lot of people rely upon card-key access control, for instance in a server closet or in a smaller data center, or actually even large data centers. Why can’t you just depend on card-key access control for your logs?

Greg Riggs: The reality is that even server closets are multi-tenant facilities. There are telco engineers, network engineers, very often electrical & facilities gear need to access those spaces, and so you complicate that complicated by the fact that it is actually pretty rare for a person who owns or manages an asset in a data center to have ownership over the access control system for the building, and the readers to those spaces, there under the control of the facility management system. Even if I had the ability, and I did, to add and remove people to a reader to a room, other people could do this as well. The owners of the facility management system have the ability to this and the logs for those systems are not often very robust and they’re not entirely dependable, to be perfectly honest. Unless you can own and manage those systems you don’t know for a fact that they’re being managed within your scope of purview or scope of control. Because of this Became a believer that facility access control should be entirely separate from what I call ASSET access control, which is the physical assets themselves. They shouldn’t necessarily be in the same system if you want to have an accountable degree that the logs haven’t been tampered with and you can speak to exactly who accessed those assets an any particular time.

Data Center Spotlight: I guess because of the need to see who may have been able to remove a drive or insert a USB device, I would imagine that it is not enough to just pull a report for who has logged into the server, correct?

Greg Riggs: You’d be surprised at the number of things which can be done to subvert the integrity of a server, physically, and this applies to network devices, or storage appliances or backup hardware, that don’t necessarily have the kind of logging that you would find on a server. Having a system in place which can help you quickly rule out the probability that a physical asset was compromised, in any kind of investigation, regardless of whether it is a server, or a network device, or any other kind of sensitive asset you run your infrastructure from. The ability to be able to quickly rule out if physical access is something I need to be looking at and do that can save valuable time in an investigation.

Data Center Spotlight: It sounds like there is a need for a security system for the rack. How did you explore that world. I know it is a bit of a burgeoning market for products that can protect the IT infrastructure at the rack or cabinet level.   What product did you settle on? What met your requirements?

Greg Riggs: There were a few options out there. There were some do-it-yourself options to get you kind of halfway where you wanted to be. And there were a number of options out there that were all intended to integrate with the building access control system but we knew right away that that ring of security needed to be managed out-of-band and so we wanted to integrate its logging and reporting into our IT monitoring infrastructure, which is separate from how the building folks managed the facilities. Ultimately we came across a product called E-Line by Dirak, and we contacted the company and worked with them. This was back in, I think 2008 or 2009. The product had just kind of hit the market and a lot of people were learning about it. I worked with the team over there. We did a trial and I was actually really surprised at the number of requirements the solution met and then from there we had a number of conversations that allowed us to start deploying in multiple facilities. We were pretty happy with it.

In fact, six years later I’m still using the product to secure the half-rack of infrastructure for my consulting firm. I consider rack-level security for critical infrastructure to be a very effective tool in responsibly managing an organizations sensitive infrastructure.

Data Center Spotlight: It seems like that’s a burgeoning market. It seems like they’re competing in a market where there’s a lot of interest. I keep hearing more people talking about access control at the cabinet or rack level. It is a market that really seems to be a growing. Are you suggesting to your consulting clients to implement an access control system such as E-Line by Dirak?

Greg Riggs: We have entered an age where it’s pretty clear that our age-old method of securing valuables with a lock & key aren’t enough. Physical keys can be copied, lost, assigned to people that never return them, and then making sure people don’t abuse the access they’re entrusted with can be very tiresome. I would say that RFID access control systems are the standard in the workplace at the building level. I’d be surprised if we don’t see it at the residential level in the next ten years or so. The cost of such a technology today is a fraction of what it was 10 years ago, which has really helped a lot of people be able to take advantage of it. When my clients take the time to consider the peace of mind and additional capabilities, that they get, such as two-factor authentication, that a system like this enables, most of them have been very receptive to include it in their overall security strategy. So yeah, I do recommend this and I would recommend this to any organization that has sensitive assets and they want to carefully manage and track who are accessing those assets.

A system like this would be perfect even in a space where, it doesn’t have to be a data center, in a space where there are a lot of people, and not everyone needs to access everything all the time. So you have the ability to provide granular access control requirements, set times of day, days of week, make sure you have requirements if two people need to access the rack at the same time, to set precautions. So this type of capability for us to secure our valuables and sensitive assets is definitely the way of the future.

Data Center Spotlight: There are a lot of companies that are expanding their network outward in various locations and a lot of times they   have something very small that needs to be protected but needs to be protected nonetheless. So I understand what you’re saying.   We’re talking about data centers and probably large data centers like Microsoft but it seems like this sort of solution has applications at a much smaller, more far-flung level.  What’s next in security access? We’re out of time but I’d like to have you on again. What’s something that would be of interest for us to talk about moving forward?.

Greg Riggs: I could think of a few fun topics. One of my favorite topics to talk about and to really get people thinking is to dive into social engineering, and really understand what that means, and how it works and how people are able to leverage social engineering which ties into situations today where workplaces are spending day and night trying to combat, which is credential theft. Social engineering ultimately leads to the ability to steal credentials and once you’ve got credentials, the world is your playground.

Beyond that another good topic would be to talk about compliance trends & challenges and figuring out what companies and organizations do compliance requirements apply to. For the rest of them, the rest of the enterprise public, who should consider compliance requirements, even if they don’t have an actual requirement set by the federal government to follow.

Data Center Spotlight: Greg, how can someone get in touch with you if they want to discuss these issues a little further.

Greg Riggs: I’d be happy to talk to folks who want to dive into this. They easiest way to get in touch with me is via my website OSHItech.com

Data Center Spotlight: And the solution you suggest is ELinebyDirak.us if people want to check that out. They can reach out to you directly. I imagine you do the LinkedIn thing as well.

Greg Riggs: Yes, you can find me at Gregory Riggs on LinkedIn.

Data Center Spotlight: Greg, so appreciate your time today. I know we went a little longer than we anticipated. You were very generous with your time and knowledge and it is appreciated.

Greg Riggs: It’s all good. There’s a lot to talk about here. Thank you. I look forward to next time.

Data Center Spotlight: Thanks, Greg.

Greg Riggs: Thank you.