Vulnerability Assessment on SCADA Networks - A Guide

Recently, I wrote an article on how to perform a Vulnerability Assessment on SCADA networks which was published in Hakin9.org. This time I took the opportunity to describe the methodology and processes in detail.

Tools Used:
1) Ping Utility
2) Nmap
3) Wireshark
4) Nessus
5) Metasploit

The tools used above are mentioned by several credible websites that deal with SCADA systems and infrastructure which include:

1) SCADA HACKER:  
http://scadahacker.com/tools.html
2) Idaho National Laboratory:
http://www.inl.gov/scada/publications/d/cyber_assessment_methods_for_scada_security.pdf
3)Tenable Security: http://www.tenable.com/sites/drupal.dmz.tenablesecurity.com/files/uploads/documents/whitepapers/SCADA%20Network%20Security%20Monitoring.pdf
4) Digital Bond:
http://www.digitalbond.com/tools/the-rack/nessus/
5) US Department of Energy: http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/Introduction_to_SCADA_Security_for_Managers_and_Operators.pdf

Compliance - The Need for Vulnerability Assessment
NERC CIP 005-3 specifically mentioned the following:
Cyber Vulnerability Assessment — The Responsible Entity shall perform a cyber vulnerability assessment of the electronic access points to the Electronic Security Perimeter(s) at least annually. 
The vulnerability assessment shall include, at a minimum, the following:
R4.1. A document identifying the vulnerability assessment process;
R4.2. A review to verify that only ports and services required for operations at these access points are enabled;
R4.3. The discovery of all access points to the Electronic Security Perimeter;
R4.4. A review of controls for default accounts, passwords, and network management community strings;
R4.5. Documentation of the results of the assessment, the action plan to remediate or mitigate vulnerabilities identified in the assessment, and the execution status of that action plan.

NERC CIP 007-3 specifically mentioned the following:
Cyber Vulnerability Assessment — The Responsible Entity shall perform a cyber vulnerability
assessment of all Cyber Assets within the Electronic Security Perimeter at least annually. The
vulnerability assessment shall include, at a minimum, the following:
R8.1. A document identifying the vulnerability assessment process;
R8.2. A review to verify that only ports and services required for operation of the Cyber
Assets within the Electronic Security Perimeter are enabled;
R8.3. A review of controls for default accounts; and,
R8.4. Documentation of the results of the assessment, the action plan to remediate or
mitigate vulnerabilities identified in the assessment, and the execution status of that
action plan.

NERC CIP 005-3: http://www.nerc.com/files/cip-005-3.pdf
NERC CIP 007-3: http://www.nerc.com/files/cip-007-3.pdf

The Methodology
From the experience i gathered doing assessments on SCADA networks and systems, i came out with the below methodology. 

Information Gathering stage

The information gathering stage is the first and the most important stage in this methodology. Failing to perform and collect the necessary information can be a problem for the later stage. In this stage, we need to collect the following information:

1) The Network Devices information such as the IP addresses for Routers, Switches, Firewalls, IPS/IDS, Honeypots, Printers and any other devices that is connected to the SCADA network.

2) The Computer Systems such as the IP addresses, the Hostnames, the type of Operating Systems, the Services running on the systems and Hardware specification information

Challenge experienced

As SCADA has been around since the 60s, one of the problems i faced was the fact that there were little or no documentation of what the system owners have. And when they did have it, it was vague such as only the hostnames but no IP addresses tied to it. And since servers and workstations are all connected on to a single network, it was hard to determine which is which. 

How I did it

Being provided only the IP ranges, i had to use NMAP to scan the IP ranges. Take note that at this stage only runs the NMAP Ping Sweep switch and not the service/OS scans since we do not know what systems are there and what are the consequences running a service/OS scan. 

Once i determined the IP addresses found from the Ping Sweep scan, i checked with the system owners to see if they can determine which are workstations, servers, network devices, etc. In my case, the system owners could not identify and determine which is which hence i had to perform an OS scan to determine the OS of the devices in the network.

Once i managed to collect the OS information, the next step i did was to put it in a spreadsheet and categorize it according to its IP | Hostname | OS. With these, i safely determined which were servers, workstations and network devices.

Take Note

Always, always remember to save all these information in a spreadsheet. 

Grouping stage

For this stage, this is where my excel spreadsheet came into play. After determining the IP | Hostname | OS, i placed all the same OS into a tab or a column. For example, i placed all the Windows XP systems into one column, the Windows NT 4.0 into another column, Windows Server 2000, 2003, Cisco Switches, etc...all into its own individual columns. 

Once separated, i showed the system owners to determine which are the Active Servers and which are the Passive ones. This is needed later when it comes to the Scanning phase. After successfully determining these, i start to use Nessus to scan for open services beginning with a handful of workstations first, followed by the rest of the workstations, then the Passive servers and then the Active servers. Remember, at this stage, the Vulnerability Scans are not started yet as i only performed the scans for open ports and services.

Once again, remember to save all these information.

Policy and Plugins stage
Now that we have determined the IP | Hostname | OS | Open Ports | Services, its time to sit down with the system owners and find out more about the systems/servers. Take a couple of minutes to go through the workstation and look at the Add/Remove programs to determine what software/applications are installed. This is needed in order to customize the Nessus policy and plugins settings. An example is to check if there's a database installed. If there is, what brand? version? This is to eliminate the unwanted plugins and retain only the necessary ones.

Take Note
Never Ever use the Nessus Default Settings when performing a VA scan. By default, it will use huge bandwidth and unlimited amount of TCP connections to the host at a time and this can potentially cause Denial of Service issues such as System Reboot, Blue Screen and Application Hang.

The Nessus Scan Default Settings. Note the 'Unlimited' connections. This needs to be throttled. In my case, the following settings was used:
Max Hosts per Scan: 5
Max Simultaneous TCP Sessions Per Host: 15
Max Simultaneous TCP Sessions Per Scan: 5

*The above settings were used for Windows XP and Windows 7 OS. Anything below such as Win NT 4.0, the following settings was used:
Max Simultaneous TCP Sessions Per Host: 5
Max Simultaneous TCP Sessions Per Scan: 1

Now that the Default Performance Settings is edited, i also need to change the Plugins. Again, by default, all the Plugins are selected. This may cause an old system like Win NT 4.0 to crash as it could not handle the load from the scanner. Hence, in the Plugin section, select only the necessary. For example, if its a Windows OS, uncheck the Linux associated plugins, and if the Windows system is using a MS SQL database, uncheck the Oracle and MySQL plugins. 

Also, create individual customized scanning policy for individual group of devices. If its a Windows XP workstations, place them into a policy which i called 'SCADA-WinXP', followed by 'SCADA-WinNT', 'SCADA-Win2K-Passive' and 'SCADA-CiscoSwitches'..you get the point.

Take Note

The lesser and more specific plugins you select, the smoother and less intrusive the scans will be. 

The Scanning stage

Now that we have customized the policies, it is time to scan. Always scan the backup or passive systems first to see what is the outcome. A good way is to scan individually (provided you have the time). What is important is to ensure that little or no downtime is achieved during the scan. This is where the Ping utility and Wireshark comes in. By pinging what you scan, you can monitor whether or not the system is alive and running. And in an unfortunate case, if a system goes down, you can quickly pause the scan and use wireshark to analyze the issue. 

Take Note

Certain systems when i scanned, no matter how much i throttled the settings will still hang the application. I realized that this happened on Win NT 4 and below systems running on old and unsupported hardware. In my case, it was running on a 5GB hard disk with a 64-256MB of RAM. So what i did at that time was to determine how many Win NT 4 and below systems were available. I then informed the system owners that these systems will definitely hang during scanning. Once acknowledged, scan one system at a time. When the scan is completed, physically check the system if the application hang, if it is, reboot, login and ensure that the system is operational before moving on to the next system.

The Validation stage

When the vulnerability scannings are done, its time to validate the findings. My recommendation is to validate those that can be validated WITHOUT exploitation. This is very important as we do not know the consequences on exploiting the vulnerability on a Live & Production systems. Hence, although we could not validate our findings, it is still our ethical duty to report them. Certain critical vulnerabilities like MS08-067, we can use Metasploit to run a 'check' utility to determine whether or not the vulnerability really exist. However certain vulnerabilities like MS09-050, is something we cannot validate without exploitation and exploiting MS09-050 can succeed yet some experienced that sometimes, the system rebooted, hence be extra careful when validating a vulnerability through exploitation.

Manual Assessment

So far i have shown was how to use automated tools like Nmap and Nessus to perform the assessment. However, during the course of this, certain times, downtime occurred due to the fact that ancient hardware and unsupported operating systems could not handle the scans. Hence, there are ways to perform the information gathering manually.

Below is the document by NIST that provides suggestions on how you can get it done manually. I recommend that this actions to be taken only on environment where the systems and hardware are more than 10 years old. 

Link: http://csrc.nist.gov/publications/nistpubs/800-82/SP800-82-final.pdf

Risks Risks Risks!!!!

So what are the risks involved when performing vulnerability assessments and/or penetration testing? As SCADA systems are sensitive, sometimes, unexplained risks could happen. Take for example the below 'Unintentional Internal Security Consequences' taken from the NIST 800-82 document.

Hence, to prevent such accidents to happen, always ensure to gather information as much as you can before deciding what approach to use (manual or automatic). 

TIPS
Digital Bond has a project that greatly helps to perform a Nessus scan to comply against NERC CIP 007-3. You can refer to the following links below:
1) https://www.digitalbond.com/tools/bandolier/downloads/
2) https://www.digitalbond.com/wp-content/uploads/2012/04/cip007r8v1.2.zip

You can read more about it on how it assist on complying to NERC CIP 007-3.
1) http://www.digitalbond.com/tools/bandolier/nerc-cip-scan-policies/

Take Note
However, do review the plugins selected by this customized policy. This scanning policy to comply specifically to NERC CIP 007-3 is good but i need to mention that this is a baseline policy to check what is necessary to comply to NERC CIP 007-3 but it did not include the additional plugins to check for other applications and OS vulnerabilities. You may need to select additional plugins to fulfill your VA requirements and using Digital Bond's NERC CIP 007 Nessus policy is a good baseline to start with.

First Published Article in Hakin9

Recently, i was selected to be involved to perform an assessment on a SCADA environment. It was an amazing experience getting to see the SCADA systems, the monitoring and the control systems that control the powerplants and power grids. Although there were many challenges faced during the assessment, it allowed me to develop my own methodology for performing a Vulnerability Assessment on SCADA networks.

I was more than happy to share the basic requirements and techniques on how to properly perform a VA on SCADA networks/systems to Hakin9. Unfortunately, you need to subscribe to Hakin9 before you can download a copy.

Link: https://hakin9.org/advanced-exploitation-with-metasploit/

Cyber Security in Singapore - Opinions

Recently i was invited by a local radio station to give a talk about Cyber Security in Singapore but due to company and legal reason, i had to decline the opportunity. Nonetheless, these were the questions i was supposed to answer during the talk show.

1) Cyber security in Singapore - has the recent hacking episodes exposed a "weakness" in Singapore's cyber security?

I wouldnt call it a weakness but an eye opener as to what else could be done by potential skillful hackers. In one of the hacking movies back in the 80s called 'Wargames', David Lightman, the hacker stated that 'I dont believe that any systems is totally secure' when someone told him that it was impossible to gain access to the systems. Taking that quote, i believed that there is no way to say that a system, a server or a website is totally 100% secure. There will always bound to have a potential issue, potential backdoor, security misconfigurations, missing or outdated patches that can be taken advantage and exploited. Before the much talked hackings of government sites lately, back in 2011, 17 of our govt sites were defaced by a hacker group called Brazil Hack Team and fortunately, that was all they were able to do. The so called hacking of the Istana and PMO website were not really a hack. It was a client side exploitation of a vulnerability called XSS or Cross Site Scripting which do not affect the server side and still maintain the confidentiality, integrity and availability of the PMO's and Istana's website/server. In other words, nothing was leaked or compromised.

2) As an Ethical Hacker and Security Consultant, what do you think are the challenges in cyber security here, and worldwide?

One of the challenges that we faced not just in Singapore but also in other countries is investments in cyber security. Singapore, similarly like USA and Israel, we invested billions in physical military warfare but not much in the technology and manpower in cyber military. In my opinion, we should also invest not just the F-16s jets but also in technology and skills that could potentially bring down an F-16 jet by using a laptop. When i went to a security conference in Amsterdam, a hacker managed to show how he can potentially hack the control systems of an airplane. If we think that that is farfetched, in 2011, hackers from China managed to hack and control a NASA satellite for approximately 11minutes. Needless to say, when it comes to hacking, nothing is impossible.

Another thing is skillset. Before 2007, local instituitions, polytechnics and Universities do not have courses that involves Ethical hacking. These ethical hacking courses were mostly seen in private instituitions. In the US, schools are established for future and potential hackers. Hacker schools, hacking academy are created so that students are trained from young. In India for example, students are exposed to security at such a young age and you have people like Ankit Fadia, an Indian hacker who published a book on Ethical hacking at the age of 16. However, i am glad that the government understand the gravity of the importance of cyber security and since mid 2007 onwards, ethical hacking modules, courses are introduced in majority of the local institutions. The graduates from these faculties will be the ones who will safeguard our network and infrastructure.

The third thing is Security education and conferences. In Singapore there are not many security conferences that are open to public. There is one that holds annually here called Syscan and i believed that such a conference will benefit the security community here in Singapore. There are also other conferences such as GovWare but such government sponsored conferences are not open publicly and can be expensive at times. If we look at countries such as US, in Europe and even in Malaysia, there are a number of conferences held every year and are affordable and open to the public. Singapore must learn from such countries and organize more conferences open to public that can educate the public in security awareness and the importance of the roles they play in the organizations. Remember that security is a shared responsibility. 

3) Are companies here prepared to deal with cyber challenges? Why or why not?

As long as the company invests in cyber security, i believed that those companies are more or less prepared for potential cyber challenges. Whenever there's a hacking incident, security officers and management will question 3 important things: whether the Confidentiality, the Integrity and the Availability of the information got compromised. Therefore, even if the website got defaced at least the information or data are not compromised, stolen or leaked.

 

4) What have been your experiences in ethically "hacking" company sites? What more can be done?

One of the most important things before ethically hacking company sites or servers is to ensure we agreed on the rules of engagement, the DOs and the DONTs. Trust is a very important matter. Just imagine if we are able to compromise a credit card database,  this is where the word ethical comes into hacking. Such major findings will be alerted to the stakeholders and we will assist them through recommendations on how to remediate such findings. Security managers in organizations must also understand the difference between performing a vulnerability assessments and a penetration testing assessment. Both may sound similar but totally different when applied. 

Companies can additionally invest on security services that perform vulnerability assessments, risks analysis on a periodic basis instead of doing it just because they have to abide by their policies and audit requirements.