“You won’t know you have a problem unless you go and look.”

Neil Wyler, who is known as ‘Grifter’ in the hacker community, made that statement as a precursor to an unforgettable story. An organization hired Grifter to perform active threat hunting. In a nutshell, active threat hunting entails looking for an attacker inside an organization’s environment.

Engagement is a critical first step to any security program. After all, if you set up detection and prevention tools while attackers are already lurking inside, they will blend into the behavioral baseline. The tools will be configured with the attackers’ footprint already embedded into the environment, which makes it more difficult to detect them.

Threat Hunting Engagement

On the first day of the threat hunting engagement, the client showed Grifter detailed documentation. It listed every server, database and other asset connected to the network, in addition to protocols being used, how traffic flowed in and out, the egress and ingress points, how the network was segmented and recent changes to the environment. The level of detail was a rarity for most organizations. It also saved Grifter a day’s work.

He began hunting.

Within minutes he spotted something unusual. Data was leaving the environment, and it looked like personally identifiable information (PII). It included names, addresses, social security numbers, tax identification codes and other highly sensitive information. All of it was unencrypted. Grifter looked at the source of exfiltration, or, in other words, how the PII was leaving the environment.

“Should data ever go out that way?” he asked.

“No, it shouldn’t. That data shouldn’t go anywhere,” the client replied nervously.

Grifter discovered the data was being exfiltrated from a web server that was not included in the inventory documentation. When he mentioned it to the client, it sparked a memory. Nearly a year ago, the company had spun up a test server, which was never decommissioned. For months, the server remained publicly accessible on the internet. To the security team, it didn’t exist. They had forgotten about it. Also within that time frame, the Apache Struts exploit was released. The client had patched its known vulnerable systems, but because the test server was unknown it was overlooked.

Grifter tracked down the destination of the PII and discovered the data was going to a nation-state. For four months, 10 records were taken every two to 10 seconds. The attack flew under the radar. The attackers weren’t noisy. They didn’t exfiltrate a chunk of records at one time daily. To the security team, it looked like normal web traffic, although the traffic wasn’t coming from a ‘normal’ location. The server sat in the research and development department, an unusual place to transmit PII. That’s how Grifter knew something was strange. With some quick math, Grifter concluded the attackers must have slowly stolen millions of records.

Grifter and the client switched from threat hunting to incident response mode.

Top Takeaways

Grifter discussed his top takeaways from the experience. First, the best threat hunters are human. No matter how many tools or blinking boxes sit in your environment, they can’t provide the deep dive that a human hunter can provide. With this incident, a tool may have detected the server, and data leaving it, but it would most likely baseline the activity as normal because the web traffic looked normal.

If the security team set up a rule for the tool, it may have raised a red flag, although many organizations don’t set up rules. And if they do, the main question applied to the rule is, ‘Is that normal or not normal activity for us?’ (In this case, the traffic appeared to be normal.) The client didn’t have rules for the overlooked server despite having the best asset inventory Grifter had ever seen. It only takes one forgotten asset to cause a breach. To find active threats, humans must sit down and look through traffic and endpoint telemetry. Humans must create alerts and mitigations to ensure tools are doing what they are designed to do. Tools can help hunters whittle down data, but humans are still needed.

Grifter also pointed out that for nation-states, data is valuable, no matter the type. For example, let’s say a dating website was breached. If you are not on the website, you may not care. Then, let’s say a government agency is breached. If you don’t work for that agency, you may not care. Finally, let’s say a tax filing company was breached. That may impact you, although you might be offered a free credit score check, and the breach fades into history.

None of those breaches may affect you, yet for nation-state attackers, all of them are valuable. Attackers can correlate the stolen data and use it as a weapon. For example, they can identify a person who subscribed to the dating website and then cross-check the information to see if that person worked for the government agency and retrieve their tax information. They can then use that information to target the person for spear-phishing campaigns, spy recruitment or other nefarious purposes.

The bottom line is: even if a data breach didn’t impact you, it may in the future. This is why all organizations should be hunting for threats with humans.

If you are interested in hearing more stories from Grifter, including how he discovered another nation-state attacker exfiltrating data from a global enterprise, join the upcoming webinar, “Storytime with Grifter: How Russia Exfiltrated Data from a Global Company” on February 3, 2022.