Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

If…, then how…

If a control test finds an exception, has the control failed?

6 decision pointsThe pack behind this answer: Security Control Assurance

Add to board memo

The short answer

Recommendation
Not always. One exception in a manual control whose population is at least twice the sample lets you test the same number again; if none of those fails, the control is Effective and the exception is still named. Any other exception makes the control Not effective.
Reasoning
One failure in a small sample may be chance; a second is not. That is why the method allows one extension, of the same size, chosen the same way (rule CA-07). An automated control gets no extension, because a setting either works or it does not, and a fully tested population has nothing left to add. A failure another control later caught is still this control's exception: the other control counts when a deficiency is rated, not when this control is concluded (rule CA-08).
Considerations
  • Here an exception means a sampled failure. It has nothing to do with an approved exception to a policy.
  • The sizes and the one extension are the pack's defaults. Your method may make them stricter.
  • This differs on purpose from a maturity assessment, which rates a whole practice with its own checks.
  • A pattern of single exceptions over several tests is worth a conversation with the owner about the design, even when each test ends Effective.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
StartA sampled occurrence failed one of the control's attributesStepRecord what failed, by how much, and the owner's reasonDecisionDid another control catch and correct it later?StepKeep it as this control's exception; note what caught itDecisionIs it an automated control?DecisionWas every occurrence in the period tested?DecisionIs it the only exception in the sample?DecisionIs the population at least twice the sample?StepAdd the same number again, chosen the same wayDecisionDid any added occurrence fail?OutcomeEffective: name the exception and give it to the ownerStepConclude Not effective; the reviewer confirms itOutcomeLog a deficiency; rate it with any tested compensating control

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 13 steps.

  1. Start

    A sampled occurrence failed one of the control's attributes

    An exception is a sampled occurrence that fails any attribute: an account left open past the leaving date, a fix after its deadline, a month in which a monthly control left no record. Here the word means a sampled failure. It has nothing to do with an approved exception to a policy.

    In the pack

  2. Step

    Record what failed, by how much, and the owner's reason

    Write down the days late, the account left open or the approval missing, and the reason the owner gives. The reason explains the exception; it does not remove it. Never drop or replace the occurrence that failed: it stays in the sample and on the workpaper, whatever the result.

    In the pack

  3. Decision

    Did another control catch and correct it later?

    Controls back each other up: a leaver's account missed on the leaving date may be found by the monthly account reconciliation. That does not make the leaver control work. The account was open, unnoticed, in between, and the leaver control is the one under test.

    In the pack

  4. Step

    Keep it as this control's exception; note what caught it

    The other control counts when a deficiency is rated, not when this control is concluded. This differs on purpose from a maturity assessment, which rates a whole practice with its own checks, so a failure the practice caught itself shows it working. A control test rates one control; the catch is evidence for the other control.

    In the pack

    See also

  5. Decision

    Is it an automated control?

    An automated control is tested once for each configuration while change control over the system was tested effective for the same period; otherwise it is sampled. The extension is for a manual control only, and an automated control has none: one failure means it does not work as designed. If you are unsure, take the cautious branch.

    In the pack

  6. Decision

    Was every occurrence in the period tested?

    When the population is no larger than the sample, every occurrence has been tested. There is nothing left to add, so the exception stands.

    In the pack

  7. Decision

    Is it the only exception in the sample?

    One failure in a small sample may be chance. A second is not. Count the exceptions in the first sample before deciding anything.

    In the pack

  8. Decision

    Is the population at least twice the sample?

    An extension is allowed only when enough untested occurrences remain to add the same number again. A monthly control that has run three times, with two sampled, has too few left. A quarterly control with four occurrences and two sampled has just enough.

    In the pack

  9. Step

    Add the same number again, chosen the same way

    Choose new occurrences from those not yet tested, spread across the period by the same rule as the first sample, and test every attribute of each. The pack allows one extension, of the same size. In its example, one late leaver in a sample of five led to five more leavers being tested.

    In the pack

  10. Decision

    Did any added occurrence fail?

    An exception among the added occurrences ends the test. Do not extend a second time.

    In the pack

  11. Outcome

    Effective: name the exception and give it to the owner

    The control has not failed. Write the result as counts, such as 9 of 10 sampled passed, and name the exception in the conclusion and the report. It is not a deficiency. A pattern of single exceptions over several tests is still worth a conversation with the owner about the design.

    In the pack

  12. Step

    Conclude Not effective; the reviewer confirms it

    The control has failed for this testing period. Write the result as counts, never as a share. Someone other than the tester reviews the workpaper, by default within 10 working days, and the conclusion is final only when they sign. That day the deficiency is confirmed and its deadlines start.

    In the pack

  13. Outcome

    Log a deficiency; rate it with any tested compensating control

    Log it in your own tracker and tell the risk owner at once. A failed key control is High unless a compensating control was tested effective for the same period, which makes it Medium; a control that is not key is Medium. Either is High if the failure leaves a risk outside appetite, or an obligation recorded as not met, without its control. A control never tested cannot compensate. A record that falls short of the evidence standard alone is Low.

    In the pack

    See also

Practical checklist

  1. Record what failed, by how much, and the owner's reason, and keep the occurrence in the sample.
  2. Note any control that caught it later, as evidence for that control.
  3. Conclude Not effective at once if the control is automated or every occurrence was tested.
  4. Count the exceptions in the first sample, and check the population is at least twice the sample.
  5. Add the same number again, chosen the same way, from occurrences not yet tested.
  6. Conclude Effective only if none of the added occurrences fails.
  7. Have someone other than the tester review and sign the conclusion.

Evidence to have ready

  • The workpaper showing the exception, the owner's reason and the occurrence kept in the sample.
  • The extension's occurrences, how they were chosen and each result.
  • The conclusion written as counts, with the exception named, and the reviewer's signature.
  • For a Not effective result, the deficiency logged and rated, with any tested compensating control named.

Assumptions

  • The tester took the population from the system of record, checked it was complete, and chose the sample, spread across the testing period.
  • Each control has written attributes, and the exception failed one of them, not a step the statement never asked for.
  • Your testing method uses the pack's rule for exceptions. Its sizes and its one extension are defaults your method may make stricter.

Not covered here