What is an incident response?


In the event that the security of a system has been compromised, an incident response is required . It is the responsibility of the security team to respond quickly and effectively to problems.


Definition of incident response

An incident response is an accelerated reaction to a problem. With regard to information security, an example would be the actions of the security team against a hacker who has penetrated a firewall and is currently humping network traffic. The incident is the breach of security. The answer depends on how the security team reacts, what actions they take to reduce damage, and when they restore resources, all while trying to ensure data integrity.


Think about your organization and how almost every aspect of the business depends on technology and computer systems. If there is a problem, imagine the potential results. In addition to the very obvious downtime and data theft, there could be data corruption, identity theft (from online personal records), compromising advertising, or even devastating financial results as customers and associates can react negatively. to the news of compromised systems.


Research into past internal and external security breaches shows that companies can often go out of business as a result of a breach. A breach can result in unavailable resources and data stolen or corrupted. But those problems that cannot be calculated financially, such as bad publicity, cannot be ignored. To get an idea of ​​how important an efficient incident response is, an organization must calculate the true cost of a violation as well as the financial effects of negative publicity, both in the short and long term.

What is an incident responseCreate an Incident Response Plan

It is important to formulate an incident response plan , support it throughout the organization, and test it regularly. A good incident response plan can not only minimize the effects of a violation but also reduce negative publicity.

From the perspective of the security team, it does not matter if a breach or breach occurs (as such events are an eventual part of when doing business using an untrustworthy method such as the Internet), but rather when it occurs. Do not think of a system as weak or vulnerable, it is important to realize that given enough time and resources, someone will breach even the safest and most secure system or network. Just check out the Security Focus website at for a detailed description of recent security breaches, from the most frequent attacks on corporate web pages to attacks on DNS name servers. in 2002.

The positive aspect of understanding the inevitability of a systems breach is that it allows the security team to develop a course of action to minimize potential damage. Combining a course of action with experience enables the team to respond to adverse conditions in a formal and timely manner.

The incident response plan can be divided into four phases:

  • Immediate action to stop or minimize the incident
  • Investigation of the incident
  • Restoration of affected resources
  • Report the incident to the appropriate channels

An incident response must be decisive and executed quickly. Because there is very little room for error, it is critical that emergency drills are performed and response times are measured. In this way, it is possible to develop a methodology that encourages speed and precision, minimizing the impact of unavailability of resources and the potential damage caused by the compromised system.

An incident response plan has a number of requirements, including:

  • A team of local experts (a Computer Emergency Response Team )
  • A reviewed and approved legal strategy
  • Company financial support
  • Executive support from senior management
  • A feasible and proven action plan
  • Physical resources, such as redundant storage, standby systems, and backup services

The Computer Emergency Response Team (CERT)The Computer Emergency Response Team (CERT)

The Computer Emergency Response Team ( CERT ) is a group of local experts who are prepared to act quickly in the event of a computer catastrophe. Finding the essential competencies for a CERT can be challenging. The concept of suitable personnel goes far beyond technical expertise and includes logistics such as location, availability, and the desire to put the organization beyond personal life when an emergency arises. An emergency is never planned, it can happen at any time, and all CERT members must be willing to accept the responsibility required of them to respond to an emergency at any time.

CERT teams typically include network and system administrators as well as information security experts. System administrators will provide knowledge and experience of system resources, including data backup, backup hardware available to use, and more. Network administrators will provide their knowledge of network protocols and the ability to dynamically redirect network traffic. Information security personnel are helpful in closely monitoring security issues as well as conducting port-mortem scans of compromised systems.

It may not always be possible, but there should be redundant personnel within a CERT. If in-depth knowledge is not required in certain key areas, then cross-training should be implemented whenever possible. Keep in mind that if only one person holds the key to data security and integrity, then the entire company will be helpless if the person is absent.

Legal considerations

Other important aspects to consider in an incident response are the legal ramifications. Security plans should be developed with members of the legal advisory team or some form of general consulting. In the same way that each company should have its own corporate security policy, each company has its own way of handling incidents from a legal perspective. Local, state, or federal regulations are beyond the scope of this document, but are mentioned because the methodology for conducting post-mortem analysis will be dictated, at least in part, by the legal consultancy. General consulting can alert technical staff to the legal ramifications of a violation; the dangers of a customer’s personal information leaking, medical or financial records; and the importance of restoring service in mission critical environments such as hospitals and banks.

Implementation of an Incident Response Plan

Once an action plan has been created, it must be accepted and actively implemented. Any aspect of the plan that is questioned during active implementation is likely to result in poor response time and downtime in the event of a violation. This is where practical exercises are invaluable. Implementation of the plan should be agreed between all related parties and safely executed, unless attention is drawn to something before the plan is put into production.

If a violation is detected while the CERT is present for quick reaction, the potential responses may vary. The team can decide to take out network connections, disconnect affected systems, repair the violation, and then quickly reconnect without further complication. The team can also observe the perpetrators and track their actions. The team can even redirect perpetrators to a honey pot – a system or segment of the network intentionally containing false data – used in order to safely and seamlessly track the raid to production resources.

Incident response should be accompanied by information gathering whenever possible. Running processes, network connections, files, directories and much more should be actively audited in real time. It can be very useful to have a snapshot of production resources when tracking malicious services or processes. CERT members and internal experts will be excellent resources for tracking such anomalies in a system. Sysadmins know which processes should and shouldn’t appear when the top or ps command is run . Network administrators are aware of what normal network traffic looks like when running snort or even tcpdump. These team members should know their systems and be able to notice an anomaly faster than anyone else who is not familiar with the infrastructure.

Investigation of an incident

Investigating a systems breach is like investigating a crime scene. Detectives gather the evidence, write down any strange clues, and take an inventory of the loss and damage. An analysis of a compromised system can be carried out while the attack is in progress or post-mortem (after the attack).

While it is unwise to trust any log file from an attacked system, there are other forensic utilities to assist you in your analysis. The purpose and features of these utilities vary, but they typically create image copies of the media, correlate events and processes, display low-level system information, and recover deleted data whenever possible.

It is also a good idea to record all investigation actions performed on a compromised system, using the script command , as shown in the following example:

script -q <file-name>

Replace <file-name> with the name of the file for the script log . Always save the log file somewhere other than the hard drive of the attacked system – a floppy drive or CD-ROM works fine for these cases.

By recording your actions, an audit trail is created that can be useful if the attacker is caught.

Gathering an image of the evidence

Creating a bit image of the mean is a feasible first step. It is a requirement if forensic data work is being carried out. It is recommended to make two copies: one for analysis and investigation and another to be stored with the original evidence for any legal procedure.

You can use the dd command that is part of the coreutils package in Red Hat Enterprise Linux to create a monolithic image of an exploited system as evidence in an investigation or to compare with trusted copies. Suppose there is only one hard drive on a system that you want to image. Connect that drive as a slave to your system and then use the dd command to create the image file, as shown below:

dd if = / dev / hdd bs = 1k conv = noerror, sync of = / home / evidence / image1

This command creates a single file called image1 using a 1k block size for speed. The conv = noerror, sync options force dd to continue reading and downloading data even if bad sectors are found on the suspect drive. Now if it is possible to study the resulting image file or even try to recover deleted files.

Information gathering after the violation

The topic of digital forensics is quite broad in itself, however the tools are most of the time architecture specific and cannot be applied in a general way. However, the issues of incident response, analysis, and recovery are important. With the right knowledge and experience, Red Hat Enterprise Linux can be an excellent platform for this type of analysis, as it includes many utilities for breach response and restore.

The Table 10-1 details some commands for file auditing and management. It also lists some examples that can be used to identify files and their attributes (such as permissions and access dates) so that you can gather additional evidence for analysis items. These tools, when combined with intrusion detection systems, firewalls, hardened services, and other security measures, can help you reduce potential damage when an attack occurs.

Command Function Example
dd Creates a bit image copy (or disk download ) of files and partitions. Combined with an md5sums check of each image, administrators can compare an image of the partition before the breach with an image of the already breached system to check if the sums match. dd if = / bin / ls of = ls.dd | md5sum ls.dd> ls-sum.txt
grep Finds useful text information within files and directories as well as reveals permissions, script changes, file attributes, and more. It is commonly used as a boxed command with another command such as ls , ps, or ifconfig . ps auxw | grep / bin
strings Prints the printable character strings to a file. It is widely used for auditing executable files such as mail commands to unknown addresses or logging to non-standard log files. strings / bin / ps | grep 'mail'
file Determine file characteristics based on format, encoding, linking libraries (if any), and file type (binary, text, etc). It is very useful for determining if an executable file such as / bin / ls has been modified using static libraries, which are a sure sign that an executable has been replaced with another installed by a malicious user. file / bin / ls
find Search directories for particular files. It is a useful tool to check the directory structure by keywords, date and time of access, permissions, etc. This can be of great help to administrators performing general audits of particular file or directory systems. find -atime +12 -name * log * -perm u + rw
stat Displays various information about a file, including the last time it was accessed, permissions, UID and GID bit settings, etc. It is very useful to verify when was the last time that a violated system executable was modified or used. stat / bin / netstat
md5sum Computes the 128-bit checksum using the md5 hashing algorithm. You can use the command to create a text file that lists all the crucial executable files that are often modified or replaced during a security attack. It redirects the sums to a file to create a simple database of checksums, and then copies the file to read-only media such as a CD-ROM. md5sum / usr / bin / gdm >> md5sum.txt

Resource restoration and recovery

While incident response is in progress, the CERT team should be investigating as well as working on system and data recovery. Unfortunately, it is the nature of the violation that dictates the course of recovery. Having redundant systems or offline data backups is invaluable during these times.

To recover systems, the response team must bring any down systems or applications, such as authentication servers, database servers, and any other production resources, up and running.

It is recommended to have production backup hardware ready to go, such as extra hard drives, hot spare servers, and the like. Ready-made systems should have all production software loaded and ready for immediate use. Perhaps only the most recent and important data will need to be imported. These ready-made systems should be kept isolated from the rest of the network. If an attack occurs and the backup systems are part of the network, then the purpose of having a backup system is defeated.

Recovering a system can be a tedious process. In many cases there are two courses of action from which to choose. Administrators can perform a clean reinstallation of the operating system on each affected system followed by restoring all data and applications. Alternatively, administrators can patch the vulnerability system and put it back into production.

System reinstallation

Performing a clean reinstall ensures that the affected system will be clean of any Trojans, back doors, or malicious processes. Reinstalling also ensures that any data (if it has been restored from a trusted backup source) is clean of any malicious modifications. The downside to a total system reinstall is the time it takes to rebuild the systems from scratch. However, if a hot backup system is available where the only action to take is to download the latest data, then the downtime is greatly reduced.

Patch the system

Patching the affected system is a more dangerous course of action and should be taken with great caution. The risk with repairing a system rather than performing a reinstallation is determining whether the system has been sufficiently cleaned of holes, Trojans, and corrupted data. Most rootkits (programs or packages that a hacker uses to gain root access to your system), Trojan commands, and shell environments are designed to hide malicious activities from audits. If the output is to repair the system, only reliable binaries should be used (for example, from a read-only CD-ROM).

Incident notification

The last part of the incident response plan is to report the incident. The security team should take notes while the response is occurring and appropriately report to organizations such as local and federal authorities, or multi-vendor software vulnerability portals, such as the Common Vulnerabilities and Exposures (CVE) website. at . Depending on the type of legal advice your organization employs, a port-mortem analysis will be needed. Even though it is not a functional requirement for an attack analysis, a post-mortem analysis can be an invaluable aid in learning how a cracker thinks and how their systems are structured to prevent future attacks.

Leave a Comment