X-Force Red is an autonomous team of veteran hackers within IBM Security that is hired to break into organizations and uncover risky vulnerabilities that criminal attackers may use for personal gain. Our team recently unveiled new statistics collected from its penetration testing engagements. One statistic that stood out, although not surprisingly, was that out of 1,176 phishing emails sent to employees within five organizations from October 2017 to November 2018, 198 people clicked on the malicious link inside the email and 196 people submitted valid credentials.

While those numbers do not appear significantly high, they still show that criminals had 196 unique opportunities to move around inside a target organization and access sensitive data. And considering one set of valid credentials is all it might take for a criminal to launch an attack, 196 of them is a gold mine.

These security mistakes are the types of vulnerabilities that can be identified by penetration testers. On the other hand, vulnerability assessments, which typically require an automated scanning tool, are designed to identify known system vulnerabilities. However, despite those differences, some vendors, cybersecurity professionals, marketing teams and others often use the terms “penetration testing” and “vulnerability assessment” interchangeably, mixing two completely different security engagements.

It’s a misconception that should be corrected so that security professionals understand exactly what they are buying and receiving and how that investment will help solve the challenge at hand. If they are unwittingly misled into buying the wrong solution for their environment, a critical unknown vulnerability exposing a high-value asset could be missed.

A Q&A With X-Force Red Penetration Testing Consultant Seth Glasgow

Seth Glasgow, an X-Force Red penetration testing consultant, has participated in many conversations with clients and security professionals where he has had to clarify the difference between vulnerability assessments and penetration testing. I chatted with Seth about the misconception, including how it came to be and what the difference is between penetration testing and vulnerability assessments.

Question: Seth, thank you for chatting with me about this topic. Can you provide more details about how some in the industry use penetration testing and vulnerability assessments interchangeably?

Glasgow: Sure, Abby. Some vendors, security professionals and others in the industry believe penetration testing is a substitute for vulnerability scanning, or vice versa. Basically, they say they don’t need both; they need one or the other. Sometimes, the two names alone cause confusion. Some may say “vulnerability testing” or “penetration scanning.” Others may say they offer penetration testing, but it’s really just an automated scan that can find known vulnerabilities. It does not involve actual manual testing.

To cover all your bases, it’s best to use a combination of manual penetration testing and vulnerability assessments. I like to compare it to clubs in a golf bag. Not every club is needed for every shot, but to play the whole game, you need all of them.

I like that analogy. How do you think this mixing of the two terms came to be? Was it marketing-related where marketers used the same language to describe the different solutions?

Glasgow: There are a few reasons, none of which began with marketing. One is related to compliance. Some mandates lump penetration testing and vulnerability assessments into one requirement, which muddies the water. At a technical level, the conversations are like a game of telephone. Information is repeated in the wrong context, and before you know it, a vendor is offering to sell a low-cost “penetration test,” but it’s really an automated scan. Also, in the past, the two terms could have been used interchangeably based on the threat and vulnerability landscape at the time. Whereas today, the two are very different and solve different problems.

Can you provide an example of how the evolution of the industry has caused significant differentiation between the two?

Glasgow: Sure, I have a couple examples. In the past, before the cloud became popular, most companies worked with physical servers. A vulnerability assessment, which involved scanning servers before they went into production, was often all that was needed to find critical vulnerabilities and make sure they were patched. After all, the servers were managed locally, making it somewhat easier to control the security around them (such as who can access them). Today, an increasing number of companies are migrating to the cloud, which has a large variety of other security implications. At a minimum, this means more server configurations need to be set up, and there can be less control and visibility into who’s accessing which data from which network. In this new security environment, penetration testing is essential in identifying configuration and access control vulnerabilities and can link those vulnerabilities together to show how an attacker could leverage them to compromise a cloud environment.

Another example is with the Payment Card Industry Data Security Standard (PCI DSS). Companies could comply with older versions of the standard by just doing a vulnerability assessment and possibly a light penetration test. However, in the PCI DSS version 3.2, the requirements specify companies implement a penetration testing methodology (see requirement 11.3) and say companies must “validate segmentation,” which can only be done by performing a manual penetration test.

So, what is the difference between the two? Can you break it down for us?

Glasgow: Whereas vulnerability scanning is 10 miles wide and one mile deep, penetration testing is 10 miles deep and one mile wide. Vulnerability assessments involve automated scanning, which cast a wide net across the entire network. Scanning evaluates every in-scope system to identify known vulnerabilities. Vulnerability assessments review systems for patching and security configuration items that represent security risk. They also include confirmation that the vulnerabilities are real and not false positives; however, they do not include exploitation of the vulnerability. Frequent assessments are important because they enable companies to understand what their attack surface looks like on a regular basis. The vulnerability landscape is constantly evolving as new discoveries are made and patches are released. I could scan a system today and have a clean bill of health, but I could scan that same system next month and find critical vulnerabilities.

Penetration testing is a manual exercise that focuses on identifying and exploiting vulnerabilities within the in-scope networks and applications. It can assess all facets of the security of a company, including networks, applications, hardware, devices and human interactions. The facets to test are decided prior to the engagement. Testing involves hackers actively exploiting vulnerabilities, emulating how a criminal would leverage and link vulnerabilities together to move laterally and/or deeper into the network to access the crown jewels. As testers, we are less concerned about vulnerabilities we cannot exploit, or those that don’t lead to anywhere valuable.

For example, let’s say you have a webpage that hosts an online brochure and has minimal user engagement. A vulnerability assessment will treat that page the same as if it were a webpage with a high level of user engagement. A penetration test would not focus on that page because the testers know it wouldn’t lead them to a highly valuable place. They may be able to use information from the brochure to move elsewhere within the network; however, they would focus on other components that would give them the most access.

Think of it this way: A vulnerability assessment identifies if the office doors in a building are unlocked. A penetration test identifies what criminals would do once they are inside the office.

Figure 1: Top differentiators between vulnerability assessments and penetration testing (source: X-Force Red)

I have one final question: If I am a cybersecurity leader looking for penetration testing services, which red flags should I look for that may indicate a vendor is actually offering a vulnerability assessment but says it’s a penetration test?

Glasgow: Be wary of the timeline. A good penetration test doesn’t adhere to a strict timeline, but it should take at least a week’s worth of work. And that’s on the low end. If a vendor is saying they can perform a test with a much quicker turnaround, that’s a sign they are probably going to use an automated scanning tool and quickly send you a report of all the findings. Also, ask about the deliverable. What kind of information will be in the findings report? If it’s a spreadsheet with scan results, that’s a sign it’s a vulnerability assessment. A penetration testing report typically includes the findings, a detailed narrative of what the testers did and remediation recommendations.

The report should also include the types of testing performed to help ensure security professionals know where remediation emphasis should be placed to make a network more difficult for hackers to gain access, maintain access and exfiltrate data.

Download the free white paper: Penetration Testing — Protect Critical Assets Using an Attacker’s Mindset