Independent thinking. Informed defence.

CISO Times

Intelligence for the people behind the defence.

The CISO Decision Brief

How do we…

How many samples do we need when testing a control?

5 decision pointsThe pack behind this answer: Security Control Assurance

Add to board memo

The short answer

Recommendation
Count how many times the control ran in the testing period, then read the sample size from the population band and the control's risk rating. Test every occurrence when the population is no larger than the sample, and raise the size, never lower it, where a failure would be severe.
Reasoning
Fixing the size from the population and the rating alone means two testers reach the same size for the same control, and nobody picks a number to suit the result (rule CA-05). One table serves calendar and event-driven controls, because it counts occurrences however they arise. The sizes give reasonable confidence that a control operated. They are not statistical samples and do not measure an error rate.
Considerations
  • The sizes are the pack's defaults: 2, 5, 20 or 25 for a Standard or Lower control and 3, 10, 30 or 40 for a Higher one, for populations of 3 to 12, 13 to 52, 53 to 250 and 251 or more.
  • An automated control is tested once for each configuration, but only while change control over its system was tested effective for the same period. Otherwise it is sampled as a manual control (rule CA-09).
  • A month or quarter with no record is an exception, not a smaller population.
  • A Higher rating means a key control on which a risk outside appetite or an unmet obligation rests, or one that failed its last test until a retest passes.

The decision, step by step

Select any step for the reasoning behind it and where the pack covers it.

  • Start
  • Step
  • Decision
  • Outcome
StartYou are about to choose a sample for a control testDecisionIs it automated, with change control tested effective?OutcomeTest each configuration once; no sample of occurrencesStepCount every occurrence in the testing periodDecisionDid any occurrence fall due in the period?OutcomeNo sample: conclude Not tested, with the reasonDecisionIs the control rated Higher risk?StepRead the Higher column: 3, 10, 30 or 40StepRead the Standard column: 2, 5, 20 or 25DecisionWould a failure of this control be severe?StepRaise the size, and record why on the workpaperDecisionIs the population no larger than the sample?OutcomeTest every occurrence in the periodOutcomeChoose that many yourself, spread across the period

Keyboard: Tab to a step, arrow keys move along the flow, Enter pins its explanation, + and − zoom, 0 fits the diagram, Esc closes. The text version below has the same content.

Where a step quotes a number, it is the default in the CISO Times pack. Your own standard may set a different one.

Read as text

The same flow written out: each step, its reasoning and where the pack covers it. 14 steps.

  1. Start

    You are about to choose a sample for a control test

    The sample size follows from two things only: how many times the control ran in the testing period, and its risk rating. Fixing them first means two testers reach the same size for the same control, and nobody picks a number to suit the result.

    In the pack

  2. Decision

    Is it automated, with change control tested effective?

    A system that performs the control without a person behaves the same way each time unless someone changes it. That holds only if change control over the system was itself tested effective for the same period. If it was not, or you do not know, the control is sampled like a manual one.

    In the pack

  3. Outcome

    Test each configuration once; no sample of occurrences

    Inspect the setting as it stands, then re-perform it once for each configuration: an input it should act on and one it should let through. Obtain the change record for the setting over the period. There is no extension: one failed test case makes the control Not effective.

    In the pack

  4. Step

    Count every occurrence in the testing period

    Count from the list you took and checked, not from the owner's estimate. For a monthly or quarterly control, the population is every month or quarter in the period: a period with no record is an exception, not a smaller population. A monthly control tested over 12 months has a population of 12, as does an event that happened 12 times.

    In the pack

  5. Decision

    Did any occurrence fall due in the period?

    Not tested applies only when nothing fell due, such as a quarterly review whose first quarter has not ended, or a control that started after the period. A review that fell due and was not held stays in the population as an exception; it does not make the population smaller. If you cannot tell, treat it as due.

    In the pack

  6. Outcome

    No sample: conclude Not tested, with the reason

    Record why nothing fell due and when the control will first run. Not tested is not a pass: it counts as untested in every report until a test is concluded.

    In the pack

    See also

  7. Decision

    Is the control rated Higher risk?

    Higher means a key control on which a risk outside appetite, or an obligation recorded as not met, rests; or a control that failed its last test, until a retest passes. A control the tester merely thinks is important stays Standard. A Lower control is one that is not key, and uses the Standard column.

    In the pack

  8. Step

    Read the Higher column: 3, 10, 30 or 40

    By population: 1 to 2, all; 3 to 12, 3; 13 to 52, 10; 53 to 250, 30; 251 or more, 40. A higher rating means a larger sample as well as more frequent tests, because a failure here would matter most. These are the pack's defaults.

    In the pack

  9. Step

    Read the Standard column: 2, 5, 20 or 25

    By population: 1 to 2, all; 3 to 12, 2; 13 to 52, 5; 53 to 250, 20; 251 or more, 25. Standard and Lower controls use the same column. For example, 38 leavers over six months gives a sample of 5. These are the pack's defaults.

    In the pack

  10. Decision

    Would a failure of this control be severe?

    The sizes give reasonable confidence that a control operated. They are not statistical samples and do not measure an error rate. The pack allows you to raise them, for example for a control whose failure would be severe. It never allows lowering them for a test.

    In the pack

  11. Step

    Raise the size, and record why on the workpaper

    A larger sample makes a missed failure less likely. Record the size you used and why, so a reviewer can see it was set before the results. Never lower a size to save time.

    In the pack

  12. Decision

    Is the population no larger than the sample?

    Where the table says all, or the population is smaller than the size, every occurrence is tested. A control that ran once or twice in the period is always tested in full.

    In the pack

  13. Outcome

    Test every occurrence in the period

    With the whole population tested, there is nothing left to add if one fails. Any exception then makes the control Not effective.

    In the pack

    See also

  14. Outcome

    Choose that many yourself, spread across the period

    Sort the population by date, divide it into as many parts as the sample, and pick one from each part at random. Record how you chose. Never let the owner choose, suggest or remove one. If one sampled occurrence fails in a manual control whose population is at least twice the sample, the method allows one extension of the same number.

    In the pack

    See also

Practical checklist

  1. Check whether the control is automated and change control over its system was tested effective.
  2. Count every occurrence in the period from the list you took and checked.
  3. Conclude Not tested, with the reason, only if nothing fell due.
  4. Read the size from the Higher or Standard column for that population.
  5. Raise the size where a failure would be severe, and record why on the workpaper.
  6. Test every occurrence where the population is no larger than the sample.
  7. Otherwise choose the sample yourself, one from each part of the period at random.

Evidence to have ready

  • The population count, its source and the second-source check.
  • The risk rating, the size read from the table and the reason for any increase.
  • The rule used to choose the sample, written before the results were seen.

Assumptions

  • Each control already has a risk rating of Higher, Standard or Lower in your inventory, and a key control is never Lower.
  • The population was taken by the tester from the system of record and agreed to a second source before anyone counts it.
  • Your testing method uses the pack's sample table. These sizes are its defaults, which your method may raise.

Not covered here