By Sila Ozeren Hacioglu, a Security Research Engineer at Picus Security.

This is a common story in cybersecurity. A new automated pentesting tool is implemented, and the initial results are impressive. The dashboard highlights crucial vulnerabilities, unexpected lateral movement options, and a previously unknown issue with an old account.

The Red Team feels empowered, and the CISO thinks they’ve successfully automated aspects of security that previously relied on human effort.

However, the initial excitement fades.

Typically, after four or five uses, new discoveries become scarce. The tool starts reporting the same old problems, and the dashboard’s value diminishes. This isn’t just a temporary slowdown; it’s the Validation Gap – the increasing difference between what organizations actually verify and what they document as verified.

If your automated pentesting tool seems to be underperforming its promises, you’re experiencing a change in the market. The industry is realizing that automated pentesting, while a valuable feature, becomes a risky approach when used by itself.

The pattern of high initial results followed by a sharp decline after the fourth run is not uncommon.

Security professionals call it the Proof-of-Concept (PoC) Cliff: the fast decrease in new vulnerability discoveries once the tool has explored its set boundaries. This is not a consequence of settings misconfiguration.

By design, automated pentesting solutions provide maximum benefit during their first use. After a few iterations, the accessible vulnerabilities within their parameters are identified. But this doesn’t guarantee a secure environment. It simply means the tool’s capabilities have been exhausted while other undetected vulnerabilities could remain.

This limitation is inherent in the tool when it is applied to a fixed system. It’s an architectural restriction, not an operational concern.

This is where Breach and Attack Simulation (BAS) offers a sharp contrast.

BAS operates independently of the relationships between attacks; it carries out numerous standalone simulations. Each method is tested independently. A blocked data leak attempt through DNS won’t prevent an exfiltration test over HTTPS afterwards. A failed lateral movement method won’t prevent other potential attack actions from being tested.

One checks which paths exist. The other checks if there is suitable protection.

To better grasp the “why” behind the PoC Cliff, it’s essential to address a growing source of confusion in the industry. While Breach and Attack Simulation (BAS) and automated pentesting both share the aim of assessment, they employ different techniques to address different questions.

Consider BAS as a set of separate tests. It consistently and safely replicates adversarial tactics, malware, lateral movement, and data exfiltration, to verify if your specific protection measures (firewalls, WAF, EDR, SIEM) are effective.

Its core goal is to assess if your protections block or flag known malicious activities. Each assessment is an autonomous evaluation of your defense capabilities.

Automated Penetration Testing, on the other hand, is targeted. It adopts a more strategic technique by chaining vulnerabilities and misconfigurations together to replicate real-world attacker methods. It excels at uncovering specific attack routes, such as Kerberoasting in Active Directory or elevating privileges to gain Domain Admin access.

While frequently referred to as “assessment methods,” the two techniques differ in objective and results. One shows the robustness of defenses, while the other shows how far an attacker could potentially progress.

Lately, some vendors have suggested that automated pentesting can replace BAS. In theory, this may seem appealing.

But in practice, this isn’t an improvement; it is a reduction in coverage presented as simplification.

As previously discussed, automated pentesting and BAS tools address distinct questions. Securing a modern organization needs both answers:

• BAS asks: “Are my firewalls, EDRs, WAFs, and SIEMs working as intended across the entire MITRE ATT&CK framework?” It focuses on the effectiveness of your security tools.

• Automated Pentesting asks: “Can an attacker move from Point A to Point B by exploiting existing vulnerabilities?” It focuses on the success of specific attack routes.

If you replace BAS assessments with automated pentesting, you neglect the assessment of your prevention and detection mechanisms.

You might be aware that an attacker can’t access your database through one specific route, but you get no insight into whether your EDR would be aware of an alternative attack technique.

While marketing promises “complete” protection, the reality of most automated pentesting is superficial in assessing applications and network infrastructure paths.

As shown above, two areas receive no protection from automated pentesting. Four areas have limited coverage, at best. None of the areas have complete protection. Thus, 0 out of 6 were fully assessed. That creates a significant validation gap in the areas modern breaches are actually targeted at:

This intersecting layer unifies each of these silos. Combining documented vulnerabilities and the working performance of live security solutions extracts ambiguity by transforming the 60% of vulnerabilities that were ranked as high or critical down to ~10% of what could actually be exploited. That produces a defensible action list that reduces false urgency by 80%.

Understanding this gap is a first step. Remedying it requires you to hold your security vendors to a higher standard. To cut through the marketing hype and discover what a tool can actually accomplish, everything comes down to the three diagnostic questions listed below.

Ask these during every vendor meeting, contract renewal, and budget discussion. They are effective because they’re based on operational components, not subjective feelings. Any tool that can completely and accurately answer each of them deserves examination. Any tool which can’t displays where your weak points exist.

Accepting, “we’ve chosen against assessing this sector” Versus “we did not realize we were not assessing this sector” creates a significant difference between risk management and exposure.

Your exploitable components aren’t concerned about the vendor producing the technology.

They’re concerned about security protection. If your automatic assessment deployment is bypassing dangerous vulnerabilities, it’s time to realign your strategic direction.

Our most recent guide, The Validation Gap: What Automated Pentesting Alone Cannot See, offers a valuable structure to evaluate your vulnerability coverage. The framework is designed to pinpoint coverage concerns and develop a consolidated strategy.

Start with evaluating the state of all six points of access. Quantify your level of understanding. Deciding what your next point of action is relies on knowing the exact limitation with existing resources.

Article by Picus Security.