Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

How do we…

How do we know our scanning covers everything it should?

7 decision pointsThe pack behind this answer: Vulnerability & Exposure Management

Add to board memo

The short answer

Recommendation
Measure coverage against your own list of in-scope systems, not the scanner's report. Count the share scanned within their required frequency, give every gap a reason and a scan-by date, and review the scanner set-up at least every six months.
Reasoning
The scanner counts only the systems it was pointed at, so a system nobody added is invisible to it and to every figure built on it. A failed authenticated login produces a scan that looks clean, and stale definitions, silent engines, cut-short scans and old exclusions lower real coverage without changing the figure. Every other vulnerability figure is only as good as coverage, so a fall in open findings alongside a fall in coverage is not an improvement.
Considerations
  • The scanning frequencies, the 95% coverage and login targets, the 5-working-day limits and the 6-month review are the defaults in the CISO Times vulnerability pack. Your own standard may set different ones.
  • Systems you cannot scan at all, such as a cloud software service with no access to the systems underneath, stay on the list marked out of scanning scope, with the reason and the supplier assurance that covers them.
  • Show coverage by criticality as well as overall, so a good total cannot hide unscanned Critical systems.
  • Where the scanner review finds failed checks that affect the figures, report the figures with those checks named as a limitation, approved by the standard owner.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
StartWe need to show that scanning covers what it shouldStepKeep one list of every system in scopeDecisionCan you scan this system yourself?OutcomeKeep it on the list and cover it by supplier assuranceStepTest the list against other sourcesDecisionIs each listed system in a scan target or covered by an agent?StepAdd it to scanning, or record why not and a scan-by dateDecisionWas it scanned within its required frequency?DecisionDid the scan log in, or the agent report in?StepFix the credential or agent, and count the system as a gapDecisionIs every internet-facing system also scanned from outside?StepAdd the missing systems to the external scanStepCalculate coverage against the list, by criticalityDecisionHas the scanner set-up passed a review in the last 6 months?StepReview the scanner and fix the checks that affect the figuresDecisionIs coverage at or above target for every criticality?OutcomeReport the shortfall with its reasons and scan-by datesOutcomeCoverage is shown; repeat the checks every period

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 18 steps.

  1. Start

    We need to show that scanning covers what it should

    The scanner's own report cannot answer this. It counts the systems it was pointed at, so a system nobody added is invisible to it and to every figure built on it. Coverage has to be measured against a list of what you are responsible for, kept separately from the scanner.

    In the pack

  2. Step

    Keep one list of every system in scope

    One row per server, network or security device, cloud account, business application and website. End-user devices can be one row per managed group. Give each an owner, a criticality rating (Critical, High or Standard) and whether it can be reached from the internet. Include systems a service provider runs for you, and name your own internal owner for them.

    In the pack

    See also

  3. Decision

    Can you scan this system yourself?

    Answer No only when you cannot scan the system at all, for example a cloud software service with no access to the systems underneath. Systems a service provider runs for you stay in scope where the contract gives you, or requires the provider to provide, vulnerability information. If you are not sure, keep it in scope until you have shown it cannot be scanned.

    In the pack

  4. Outcome

    Keep it on the list and cover it by supplier assurance

    Do not delete it. Mark it out of scanning scope, record why, and say what covers it instead: contracts, certificates, reports or questionnaires from the supplier. It stays visible on the list, so the gap is a decision anyone can see.

    In the pack

    See also

  5. Step

    Test the list against other sources

    A list only you maintain can drift. Compare it with cloud accounts and subscriptions, the directory, network discovery, and every internal network range. For internet-facing systems, compare the external target list with public DNS records, public address allocations and cloud services with public addresses. The pack's defaults: add new systems to scanning before, or within 5 working days of, going live, and reconcile the external list every 90 days.

    In the pack

  6. Decision

    Is each listed system in a scan target or covered by an agent?

    Reconcile the list with the scanner's target groups and working agents, system by system. Also check that the criticality and internet-facing tags held in the scanner match the list, because priorities depend on both. If you cannot tell for a system, treat it as not covered.

    In the pack

  7. Step

    Add it to scanning, or record why not and a scan-by date

    Every in-scope system that is not scanned needs a reason and a date by which it will be. The same applies to every exclusion of a system, address range or check: record the reason, who approved it and when it will be reviewed. An exclusion with no review date quietly becomes permanent.

    In the pack

  8. Decision

    Was it scanned within its required frequency?

    The pack's defaults: weekly for internet-facing systems (external scan) and for Critical and High systems; monthly for Standard servers and network devices; monthly at least for laptops and desktops, or continuously by agent; continuous or weekly for cloud infrastructure; web applications monthly if internet-facing and quarterly if internal, and before major releases. A scan that did not complete against the system does not count. If you don't know, answer no.

    In the pack

  9. Decision

    Did the scan log in, or the agent report in?

    Servers and end-user devices should be scanned with credentials or an installed agent wherever the technology allows. A scan that fails to log in misses most missing patches and looks clean. The pack's defaults: at least 95% of authenticated scans log in, and failures are fixed within 5 working days. If the login result is not known, answer no.

    In the pack

  10. Step

    Fix the credential or agent, and count the system as a gap

    Until it is fixed, that system's results are incomplete, and its open findings are under-counted. Look for the common causes: a changed service-account password, a locked account, or an agent that has gone silent. The pack's default for agents is a check-in within the last 7 days.

    In the pack

  11. Decision

    Is every internet-facing system also scanned from outside?

    An internal scan does not show what an attacker on the internet can reach. Each internet-facing system on the list needs an external scan as well, across all network ports. If you are not sure the external target list is complete, answer no.

    In the pack

  12. Step

    Add the missing systems to the external scan

    These are the systems an outside attacker can reach and you cannot see. Add them to the external target list, and reconcile it again against public DNS records and address allocations. Where payment card data is handled, check the external scanning that standard expects as well.

    In the pack

  13. Step

    Calculate coverage against the list, by criticality

    Coverage is the number of in-scope systems scanned within their required frequency, divided by the number of in-scope systems. The pack's default target is 95%. Show it for Critical, High and Standard systems too, so a good overall figure cannot hide unscanned Critical systems. A fall in open findings alongside a fall in coverage is not an improvement.

    In the pack

  14. Decision

    Has the scanner set-up passed a review in the last 6 months?

    Coverage can look complete while the scanner is not working. Stale vulnerability definitions report a clean result for flaws they do not yet know. An engine that fails silently, a scan cut short by a time window, or a suppression that outlived its reason all hide exposure. The pack's default is a checklist review at least every 6 months. If there was none, answer no.

    In the pack

  15. Step

    Review the scanner and fix the checks that affect the figures

    Work through the checklist. Some checks make coverage, deadline or overdue figures wrong when they fail. Fix those first. Until they are fixed, report the figures with the failed checks named as a limitation, approved by the standard owner.

    In the pack

  16. Decision

    Is coverage at or above target for every criticality?

    Compare the figure with your standard's target, overall and for each criticality. If any part of the calculation is uncertain, for example the list has not been reconciled this period, answer no.

    In the pack

  17. Outcome

    Report the shortfall with its reasons and scan-by dates

    Report coverage in the monthly figures with the unscanned systems, their reasons and the dates by which they will be scanned. Say what decision is needed to close the gap, such as access from a provider or a scanning account. Do not soften it: every other figure in the report is only as good as coverage.

    In the pack

    See also

  18. Outcome

    Coverage is shown; repeat the checks every period

    You can show coverage from your own list, with reasons for every gap and a scanner you have checked. Repeat the reconciliation each scanning period and the scanner review at least every 6 months. Keep each period's coverage figures, because a single period tells you less than the trend.

    In the pack

Practical checklist

  1. Keep one list of every in-scope system with an owner, a criticality rating and whether it faces the internet, including systems a provider runs for you.
  2. Reconcile the list with cloud accounts, the directory, network discovery and internal ranges, and the external target list with public DNS records and address allocations.
  3. Check that each listed system is in a scan target or has a working agent, with matching criticality and internet-facing tags.
  4. Record a reason and a scan-by date for every unscanned system, and a reason, approver and review date for every exclusion.
  5. Treat failed logins and silent agents as gaps, and fix credential failures within the pack's default of 5 working days.
  6. Add every internet-facing system to the external scan.
  7. Calculate coverage overall and by criticality, and compare it with your target.
  8. Review the scanner set-up against a checklist at least every 6 months, and fix first the checks that affect the figures.

Evidence to have ready

  • The scope list with owners, criticality, internet-facing flags and out-of-scope reasons.
  • Each period's coverage figure, overall and by criticality, with the unscanned systems, their reasons and scan-by dates.
  • The exclusions list with the reason, the approver and the review date for each.
  • Authenticated login results, and the dates credential failures were fixed.
  • The latest scanner configuration review and the standard owner's sign-off.
  • Records of the reconciliations against other sources, including the external target list.

Assumptions

  • You run at least one scanner or agent, and you have some form of asset inventory, even an incomplete one.
  • The frequencies, targets and time limits shown are the defaults in the CISO Times vulnerability pack. Your own standard may set different ones. The checks themselves matter more than the numbers.

Not covered here