ISASecure vulnerability identification testing: how the pass threshold scales with security level

Of the three streams in an ISASecure CSA evaluation, vulnerability identification testing (VIT-C) is the one where the laboratory does the testing itself. It is also the one that prompts the most anxious question from suppliers: what do we have to fix to pass? The answer depends on the capability security level you are certifying to, and the dependency is simpler than most people expect.

This article covers the pass threshold, what "addressed" means, what the requirements and the scanner policy actually cover, how scope is defined, and how to rehearse the scan before the lab runs it.

A scan, not an attack

VIT looks for known, published vulnerabilities on the component using a vulnerability scanner. It does not fuzz protocols, flood interfaces or attempt exploitation; those are supplier activities under the development lifecycle, and the testing article in this series explains where each of them lives. Because the scan is a lookup against a vulnerability feed rather than a creative attack, its results are predictable, and a supplier who understands the acceptance rule can know the outcome before the lab does.

The threshold is a ratchet

The pass criterion is stated in the scanner's severity bands, with one rule per security level.

At SL 1, only critical findings must be addressed. SL 2 adds the high band. SL 3 adds medium. SL 4 adds low. Informational findings never count against a pass at any level. Each step up adds exactly one band and removes nothing.

Two things follow. First, the criterion is expressed in bands, not in numbers. There is no score cut-off to compute; a finding is in or out of scope for the verdict according to the severity the scanner assigns it. Second, the scan itself does not change with the level. The same policy, interfaces and operating modes are exercised whether you are certifying to SL 1 or SL 4; only the filter applied afterwards moves. A supplier stepping from SL 2 to SL 3 does not face a harder scan. It faces the same scan with medium findings now on the wrong side of the line.

"Addressed" is not "fixed"

The word is deliberately broad. A finding inside the threshold is addressed when it has been corrected, or when the supplier has documented why it is not relevant to the component as deployed. Both routes are valid and both are recorded in the report.

That has a consequence people miss: a VIT pass never requires zero findings. Findings below the threshold are reported but do not block. Findings inside the threshold can be closed with a rationale rather than a patch, provided the rationale holds up: a signature that matches a version string but not the code actually shipped, a fix backported without a version change, a condition that cannot be reached over the interface in question. The lab evaluates each rationale on its merits; it does not accept them on assertion. Findings that are neither corrected nor argued away stay open, and the VIT stream cannot close while any finding is open.

What the 21 requirements are about

The VIT specification carries 21 requirements. Five are component-specific and sixteen are shared with the system-level scheme, which uses the same scan. Only one of the 21 is the pass criterion. Ten are about reporting and reproducibility: what gets recorded, and enough about the tool, policy and configuration that the result could be reproduced later. The remainder cover the tool itself, its configuration, execution, the currency of the vulnerability feed, and a fallback path for components the standard scanner cannot support.

The shape is the message: the scheme cares as much about a scan that can be reproduced and audited as about the verdict. A finding you dispute can be traced to a specific check.

The scanner policy: 36 departures from the defaults

The scheme names a specific commercial scanner, Tenable Nessus, and defines the required policy as 36 settings that differ from that tool's defaults: 30 policy settings, grouped by the scanner's own policy areas, plus 6 credential settings. Assessment checks, including web application tests, and discovery carry the most settings; reporting, advanced options and credentials make up the rest.

The direction of the departures matters more than the count. A scanner's defaults are tuned for a live network, where a probe that knocks over a device is a serious problem. The certification policy is tuned for a lab bench, where the device is expected to be pushed. Checks a default policy would hold back for safety are switched on, and the tool's own suppression of probable false alarms is switched off, because the lab wants to judge those itself. A supplier who rehearses with an out-of-the-box policy is scanning more gently than the lab will.

Scope: interface, mode, version

The unit of scanning is the accessible network interface, and each one is scanned on its own. If the component has operating modes in which an essential function is active, the scan is repeated in each of them, because a service exposed in one mode may be absent in another. The whole exercise is tied to the component version under evaluation; a new version means a new scan.

A component with no accessible network interface is not scanned. That is a legitimate outcome, but it is recorded in the report together with the reason, not left silent. These three dimensions define the scan set, and the supplier's own interface list is what populates them; an interface missing from that list is missing from the scan.

The known-good state

The lab does not scan a locked-down test unit, and it does not scan a factory-fresh one. It scans the component in the state a customer would reasonably deploy it: hardened according to the supplier's own published hardening guidance, with administrative credentials supplied so the scanner can log in, and with services running and ports open unless that guidance says to turn them off.

This makes the hardening guide a test input. A guide that says to disable a service removes it from the bench and therefore from the findings; a guide that is silent leaves the service on. Suppliers who treat hardening guidance as an afterthought get scanned in a more permissive configuration than they intended.

Who does what

One row deserves emphasis. Raw scanner output is never handed to the supplier. The lab triages first, removes what it can show to be a false positive, and discloses the remaining findings with their severity. As an ISASecure certification body, we release findings only after that triage, which is why the list you receive is shorter than the one the scanner produced.

How to rehearse

Because the scan is a known quantity, it can be rehearsed. A useful dry run has five parts.

1. Build the bench the way the lab will. Configure a unit exactly as your hardening guide says, and nothing more. Note every place the guide is silent.
2. Scan credentialed, per interface, per mode. Give the scanner the administrative access the lab will have. Scan each accessible interface on its own, in every operating mode where an essential function runs.
3. Scan thoroughly, not safely. Turn off the protective defaults. If your product cannot survive an unsafe scan, learn that before the lab does, even though survival is not itself a VIT criterion.
4. Sort by the bands your level covers. For SL 2, everything critical and high is the work list; medium and below is information.
5. Write the disposition now. For each finding on the work list, decide: correct it, or write the non-relevance rationale. Rationales prepared in advance and tied to the hardening guide are the ones that survive review.

A supplier who arrives with an interface list, a hardening guide, an essential-functions list, credentials and a per-finding disposition has removed most of the uncertainty from the one test the lab performs. The rest of the CSA series is on the Insights index.