Saturday 20 September 2014

Vulnerability Assessment on SCADA Networks - A Guide

Recently, I wrote an article on how to perform a Vulnerability Assessment on SCADA networks which was published in Hakin9.org. This time I took the opportunity to describe the methodology and processes in detail.

Tools Used:
1) Ping Utility
2) Nmap
3) Wireshark
4) Nessus
5) Metasploit

The tools used above are mentioned by several credible websites that deal with SCADA systems and infrastructure which include:

1) SCADA HACKER:  
http://scadahacker.com/tools.html
2) Idaho National Laboratory:
http://www.inl.gov/scada/publications/d/cyber_assessment_methods_for_scada_security.pdf
3)Tenable Security: http://www.tenable.com/sites/drupal.dmz.tenablesecurity.com/files/uploads/documents/whitepapers/SCADA%20Network%20Security%20Monitoring.pdf
4) Digital Bond:
http://www.digitalbond.com/tools/the-rack/nessus/
5) US Department of Energy: http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/Introduction_to_SCADA_Security_for_Managers_and_Operators.pdf

Compliance - The Need for Vulnerability Assessment
NERC CIP 005-3 specifically mentioned the following:
Cyber Vulnerability Assessment — The Responsible Entity shall perform a cyber vulnerability assessment of the electronic access points to the Electronic Security Perimeter(s) at least annually. 
The vulnerability assessment shall include, at a minimum, the following:
R4.1. A document identifying the vulnerability assessment process;
R4.2. A review to verify that only ports and services required for operations at these access points are enabled;
R4.3. The discovery of all access points to the Electronic Security Perimeter;
R4.4. A review of controls for default accounts, passwords, and network management community strings;
R4.5. Documentation of the results of the assessment, the action plan to remediate or mitigate vulnerabilities identified in the assessment, and the execution status of that action plan.

NERC CIP 007-3 specifically mentioned the following:
Cyber Vulnerability Assessment — The Responsible Entity shall perform a cyber vulnerability
assessment of all Cyber Assets within the Electronic Security Perimeter at least annually. The
vulnerability assessment shall include, at a minimum, the following:
R8.1. A document identifying the vulnerability assessment process;
R8.2. A review to verify that only ports and services required for operation of the Cyber
Assets within the Electronic Security Perimeter are enabled;
R8.3. A review of controls for default accounts; and,
R8.4. Documentation of the results of the assessment, the action plan to remediate or
mitigate vulnerabilities identified in the assessment, and the execution status of that
action plan.

NERC CIP 005-3: http://www.nerc.com/files/cip-005-3.pdf
NERC CIP 007-3: http://www.nerc.com/files/cip-007-3.pdf


The Methodology
From the experience i gathered doing assessments on SCADA networks and systems, i came out with the below methodology. 


Information Gathering stage
The information gathering stage is the first and the most important stage in this methodology. Failing to perform and collect the necessary information can be a problem for the later stage. In this stage, we need to collect the following information:

1) The Network Devices information such as the IP addresses for Routers, Switches, Firewalls, IPS/IDS, Honeypots, Printers and any other devices that is connected to the SCADA network.

2) The Computer Systems such as the IP addresses, the Hostnames, the type of Operating Systems, the Services running on the systems and Hardware specification information

Challenge experienced
As SCADA has been around since the 60s, one of the problems i faced was the fact that there were little or no documentation of what the system owners have. And when they did have it, it was vague such as only the hostnames but no IP addresses tied to it. And since servers and workstations are all connected on to a single network, it was hard to determine which is which. 

How I did it
Being provided only the IP ranges, i had to use NMAP to scan the IP ranges. Take note that at this stage only runs the NMAP Ping Sweep switch and not the service/OS scans since we do not know what systems are there and what are the consequences running a service/OS scan. 

Once i determined the IP addresses found from the Ping Sweep scan, i checked with the system owners to see if they can determine which are workstations, servers, network devices, etc. In my case, the system owners could not identify and determine which is which hence i had to perform an OS scan to determine the OS of the devices in the network.

Once i managed to collect the OS information, the next step i did was to put it in a spreadsheet and categorize it according to its IP | Hostname | OS. With these, i safely determined which were servers, workstations and network devices.

Take Note
Always, always remember to save all these information in a spreadsheet. 

Grouping stage
For this stage, this is where my excel spreadsheet came into play. After determining the IP | Hostname | OS, i placed all the same OS into a tab or a column. For example, i placed all the Windows XP systems into one column, the Windows NT 4.0 into another column, Windows Server 2000, 2003, Cisco Switches, etc...all into its own individual columns. 

Once separated, i showed the system owners to determine which are the Active Servers and which are the Passive ones. This is needed later when it comes to the Scanning phase. After successfully determining these, i start to use Nessus to scan for open services beginning with a handful of workstations first, followed by the rest of the workstations, then the Passive servers and then the Active servers. Remember, at this stage, the Vulnerability Scans are not started yet as i only performed the scans for open ports and services.

Once again, remember to save all these information.

Policy and Plugins stage
Now that we have determined the IP | Hostname | OS | Open Ports | Services, its time to sit down with the system owners and find out more about the systems/servers. Take a couple of minutes to go through the workstation and look at the Add/Remove programs to determine what software/applications are installed. This is needed in order to customize the Nessus policy and plugins settings. An example is to check if there's a database installed. If there is, what brand? version? This is to eliminate the unwanted plugins and retain only the necessary ones.

Take Note
Never Ever use the Nessus Default Settings when performing a VA scan. By default, it will use huge bandwidth and unlimited amount of TCP connections to the host at a time and this can potentially cause Denial of Service issues such as System Reboot, Blue Screen and Application Hang.

The Nessus Scan Default Settings. Note the 'Unlimited' connections. This needs to be throttled. In my case, the following settings was used:
Max Hosts per Scan: 5
Max Simultaneous TCP Sessions Per Host: 15
Max Simultaneous TCP Sessions Per Scan: 5

*The above settings were used for Windows XP and Windows 7 OS. Anything below such as Win NT 4.0, the following settings was used:
Max Simultaneous TCP Sessions Per Host: 5
Max Simultaneous TCP Sessions Per Scan: 1


Now that the Default Performance Settings is edited, i also need to change the Plugins. Again, by default, all the Plugins are selected. This may cause an old system like Win NT 4.0 to crash as it could not handle the load from the scanner. Hence, in the Plugin section, select only the necessary. For example, if its a Windows OS, uncheck the Linux associated plugins, and if the Windows system is using a MS SQL database, uncheck the Oracle and MySQL plugins. 



Also, create individual customized scanning policy for individual group of devices. If its a Windows XP workstations, place them into a policy which i called 'SCADA-WinXP', followed by 'SCADA-WinNT', 'SCADA-Win2K-Passive' and 'SCADA-CiscoSwitches'..you get the point.

Take Note
The lesser and more specific plugins you select, the smoother and less intrusive the scans will be. 

The Scanning stage
Now that we have customized the policies, it is time to scan. Always scan the backup or passive systems first to see what is the outcome. A good way is to scan individually (provided you have the time). What is important is to ensure that little or no downtime is achieved during the scan. This is where the Ping utility and Wireshark comes in. By pinging what you scan, you can monitor whether or not the system is alive and running. And in an unfortunate case, if a system goes down, you can quickly pause the scan and use wireshark to analyze the issue. 


Take Note
Certain systems when i scanned, no matter how much i throttled the settings will still hang the application. I realized that this happened on Win NT 4 and below systems running on old and unsupported hardware. In my case, it was running on a 5GB hard disk with a 64-256MB of RAM. So what i did at that time was to determine how many Win NT 4 and below systems were available. I then informed the system owners that these systems will definitely hang during scanning. Once acknowledged, scan one system at a time. When the scan is completed, physically check the system if the application hang, if it is, reboot, login and ensure that the system is operational before moving on to the next system.

The Validation stage
When the vulnerability scannings are done, its time to validate the findings. My recommendation is to validate those that can be validated WITHOUT exploitation. This is very important as we do not know the consequences on exploiting the vulnerability on a Live & Production systems. Hence, although we could not validate our findings, it is still our ethical duty to report them. Certain critical vulnerabilities like MS08-067, we can use Metasploit to run a 'check' utility to determine whether or not the vulnerability really exist. However certain vulnerabilities like MS09-050, is something we cannot validate without exploitation and exploiting MS09-050 can succeed yet some experienced that sometimes, the system rebooted, hence be extra careful when validating a vulnerability through exploitation.


Manual Assessment
So far i have shown was how to use automated tools like Nmap and Nessus to perform the assessment. However, during the course of this, certain times, downtime occurred due to the fact that ancient hardware and unsupported operating systems could not handle the scans. Hence, there are ways to perform the information gathering manually.

Below is the document by NIST that provides suggestions on how you can get it done manually. I recommend that this actions to be taken only on environment where the systems and hardware are more than 10 years old. 




Risks Risks Risks!!!!
So what are the risks involved when performing vulnerability assessments and/or penetration testing? As SCADA systems are sensitive, sometimes, unexplained risks could happen. Take for example the below 'Unintentional Internal Security Consequences' taken from the NIST 800-82 document.


Hence, to prevent such accidents to happen, always ensure to gather information as much as you can before deciding what approach to use (manual or automatic). 

TIPS
Digital Bond has a project that greatly helps to perform a Nessus scan to comply against NERC CIP 007-3. You can refer to the following links below:
1) https://www.digitalbond.com/tools/bandolier/downloads/
2) https://www.digitalbond.com/wp-content/uploads/2012/04/cip007r8v1.2.zip

You can read more about it on how it assist on complying to NERC CIP 007-3.
1) http://www.digitalbond.com/tools/bandolier/nerc-cip-scan-policies/

Take Note
However, do review the plugins selected by this customized policy. This scanning policy to comply specifically to NERC CIP 007-3 is good but i need to mention that this is a baseline policy to check what is necessary to comply to NERC CIP 007-3 but it did not include the additional plugins to check for other applications and OS vulnerabilities. You may need to select additional plugins to fulfill your VA requirements and using Digital Bond's NERC CIP 007 Nessus policy is a good baseline to start with.

No comments:

Post a Comment