Incident Response Essentials

Tom Field
August 24, 2009 | COMMENTS

The Heartland data breach and July's denial of service (DDoS) attacks against government agencies are among the biggest information security incidents of the year. And they've pushed incident response into the spotlight. Peter Allor is a member of the Forum for Incident Response and Security Teams (FIRST) Steering Committee, a forum for security and incident information exchange between teams international. He also is the program manager for cyber incident and vulnerability handling for IBM, where he is responsible for guiding the company's overall security initiatives and participation in enterprise and government implementation strategies.

In this interview he discusses key incident response issues facing organizations today, what we've learned from the Heartland and government DDoS incidents, and how to prepare for a successful career in incident response.

TOM FIELD: Tell us about FIRST and your role with the organization.

ALLOR: First of all, FIRST, or the Forum for Incident Response Security Teams, is a global non-profit organization and we network together incident response teams throughout the academic, corporate, enterprise, vendor and government teams who deal with incidents on a very regular basis, usually on a very daily basis. So it is a way to be able to reach out and learn from others throughout the globe about what incidents they are handling or how to better handle a particular type of incident.

My role within FIRST now is I am currently one of the ten elected Steering Committee members who also act as the board of directors for www.FIRST.org, which is our legal umbrella. My job then is to listen to the membership and then that program tends to carry throughout the year and it is only on a two-year term basis.

FIELD: What are the key incident response issues that organizations really need to be paying attention to today?

ALLOR: There are so many and they are so varied so it really depends on what kind of organization you are and what kind of data that you are trying to watch for and protect. That will drive a lot of how you approach your mission. After that you are looking how to defend your website, how to better support your business while securing it, and then it goes to your products or services that you are providing and how you secure those. And then of course, all relations you have as an organization that is strong, its vendors, and its suppliers, to its constituent stakeholders and to its customers.

FIELD: There have been a couple of incidents that have really got the interest of our audience over the course of the year and one of them is the Heartland data breach that we learned about in January, and then there was the July 4th denial of service attacks against mainly government organizations. Given these two events, what, if anything, have we learned from how we responded to those incidents?

ALLOR: Well there are several things that you can draw from each of those incidents. In the July distributor denial of service attack, or the DDOS as we like to refer to them, there are things you can do in advance. The biggest part of incident response is how you preempt something that has happened. The key part there as you are trying to go through and if you will, make sure that your website is prepared for that type of eventuality. I'm not trying to say that it may or may not happen; I am saying it is an eventuality that you are going to have to deal with.

So you work with your upstream providers. You make sure of what kind of resources are there and how do you load balance and how do you throttle everything. And of course then you are looking for how do you absorb that pain with resiliency for that entire effort. So there are a lot of things that could be learned from the July episode.

From a data breach perspective, you are looking at your ecosystem that you are watching over that is a constantly moving and dynamic issue. Systems that were protected today could be discovered to be vulnerable tomorrow and how are you going to mitigate that issue while you look at how you are going to upgrade your machines and protect the environments. What kind of security technologies, processes are you going to employ and what is the real risk. I think the part that most responders forget to ask is, “How do I support the business unit, the people who have to do the operations that they are counting on to your information technology resources?” You have to talk a lot with those folks so it can make an informed risk decision.

FIELD: A key issue is at some point you have got to bring the customer into play. When should the customer come into play at incident response and when and how should organizations be engaging the customer; at what stage?

ALLOR: The key part here is you have to be able to recognize when you have had an issue, and then it goes down to how do you move through and alert not only the customer that you have had an issue, but at what point do you involve outside organizations to either continue the forensics so you have a very broad scope independent audible way of looking at it, or is your organizations turning to law enforcement because of the nature of the breach? Each of those impacts what you say and when you say it to the customer.

So for instance, if you turn to law enforcement and they can actually investigate further and go after the parties that are perpetrating the attack, you may be delayed a little bit in how you inform the customer. In our eyes we always try to inform the customer as soon as possible. It is their right to know.

FIELD: Let's talk about careers in incident response. This is a discipline that really has become, I think a lot more evolved in recent years and I think you would agree. What does it take today to be an incident response professional in terms of academic background, certifications, real life experience; what do you need to succeed on the job?

ALLOR: We look for folks who actually have been schooled in IT disciplines. You don't have to be universal in your knowledge; you have to be conversant in what the issues are and how they impact. The key part of that is understanding to the point that you can actually use the most important skill you have, communicating to the business owner what the problem is in plain, simple English. There is a tendency in IT, especially in incident response, to jump into the acronym suit and as soon as we do that the eyes roll and the understanding dissipates and you have no informed decisions.

So the key thing that I look for in my incident responders and I think people within our organization is can you communicate to me the essence of the problem, what it means to me and what my options are, do you have ways we can mitigate and what is that going to cost me. It is not just a matter of cost in that it is going to take X number of dollars to remediate right now, it is a matter of what will it cost me in the process. When can I do this? Do I have to do it immediately and therefore stop operations, or can I get through to a natural quiet period, whether it is the night, the weekend, at the end of a quarter or things of that nature.

So you have to really be in tune to that so understanding business is probably the next important part right behind communications. Notice I put the technical skills almost third then. That is kind of different from what most people look at in incident response because they are looking for the most technically adept, which we want in incident handlers, but when you start moving up in the career field the look that tends to rise to the surface is understanding business and communication skills.

FIELD: But what it speaks to is, although you need the technology skills, you are also in service of the business, which we tend to forget.

ALLOR: Absolutely. Service to the business is probably the best way to sum that. If you are not in service to the business then you are not providing value and that is really where a tendency within the security field is we are always seen as the "no people" you can't do something and we are not providing the service of how can you best do it and mitigate the risk to the next step.

FIELD: Given the attention that has been given to cyber security this year from the President on down, careers in information security are looking very sexy to people. What advice would you give to somebody that is looking to start or restart a career and they are looking at incident response?

ALLOR: Well understand that incident response is very, very high on a lot of people's wish list and again, it comes back to do you have some technical skills, do you understand at least in a general sense how the networking of an organization works from the outside as well as inside and how the data is stored and how it moves. If you could understand that, then you can apply anything to where different applications, offering systems and all of that [indiscernible] follow the same general format. So again, I look at it as a very broad education that doesn't necessarily always have to have depth.

If you are going to be an incident handler on a particular part you need depth, but if you are working in more management you will need a general sense. After that it is really do you understand business? I am not saying you have to have a business degree, you have to understand spreadsheets and you know, how everything goes through a Perc Chart, but if you don't understand that is how business people think and what they are looking for to demonstrate a value, that is the real part that is lost there. So I look for people that have those types of skills.

FIELD: And I guess more specifically you need to know your business.

ALLOR: Yes. Well the other part too is you have to have a community of network people outside your organization. Most people don't think about this part.

For instance, on my day job I work for IBM, but I also work with folks from HP or Microsoft or Cisco or Juniper and so it is a networking of people that when you have an issue and you haven't seen something like this before, you reach out. This is where FIRST comes in and it is a very important networking group because globally you already know who the team is. They are vetted to you, you have the secure means to pass sensitive information back and forth, we re-encrypt all of our sensitive information, and there is a way to get everyone's PGP Key to make sure it is valid. So that networking part becomes very important because you realize quickly that you are a generalist, and the specifics you can get from others. Knowing how to get that specific and to get it quickly is the key part to flexing with a rapidly changing dynamic and then applying a good fix quickly. That is the important part. How do you go for a quick fix and the reality is you are looking for a good fix quickly.

Tom Field is the editorial director for CUInfoSecurity. Reprinted with permission.