How do we test whether a security control actually works between audits?

Start

A control is due for a test between audits

A test checks, from records the tester chooses, that a control ran as designed every time it should have, throughout a stated period. A policy or an owner's assurance says what should happen; a test shows what did. It is not an internal audit, a certification audit, a maturity assessment or a statement of compliance, though its results can feed each of them.

• Then: Is it written as who does what, how often, and its record?

Decision

Is it written as who does what, how often, and its record?

A control is tested against its written statement, not against what the owner says usually happens. "Access is reviewed regularly" cannot be tested. "The system owner and information security review privileged access every three months and record a decision for every account" can. The attributes the tester checks on each occurrence come from the statement.

• Yes: Does the tester own or perform none of this control?
• No: Record a design deficiency; test operation only after the fix

Decision

Does the tester own or perform none of this control?

No one tests a control they own or carry out, manage the people who carry it out, or wrote the procedure for. An owner's own check is useful to them, but it is a self-check: it does not count as a test, and the control stays untested in every report. If you are unsure, take the cautious branch.

• Yes: Set the testing period and confirm the risk rating
• No, or not known: Assign a tester and a reviewer who are independent of it

Step

Assign a tester and a reviewer who are independent of it

The pack suggests, in order, a colleague from another function trained on the method, a tester from a peer team, internal audit, or a contracted tester. The reviewer must be someone other than the tester. Never let a self-check stand in for the test because no one else was free.

• Then: Set the testing period and confirm the risk rating

Step

Set the testing period and confirm the risk rating

The period runs from the end of the last tested period, or from when the control started, to the end of the period being tested, and is never longer than the control's test cycle. The rating, Higher, Standard or Lower, sets that cycle and the sample size. Copy the statement, owner, performer and attributes onto a new workpaper from the inventory.

• Then: Would the design keep the risk in check if run as written?

Decision

Would the design keep the risk in check if run as written?

Choose one recent occurrence and follow it with the performer from trigger to record, inspecting each record. Ask whether it reaches every system and person in scope, and whether its timing is tight enough. A leaver process that misses systems with local accounts fails here, however faithfully it runs. The pack's default is a design check at least every 12 months and whenever the control changes.

• Yes: Did the control run at least once in the testing period?
• No, or not known: Record a design deficiency; test operation only after the fix

Outcome

Record a design deficiency; test operation only after the fix

Conclude the design Deficient and record the operation as Not tested, with the reason. Testing the operation of a control that could not work wastes the test and gives a false result. Log the deficiency in your own tracker, rate it, and agree a fix with the owner. Then check the design again before the operating test.

Decision

Did the control run at least once in the testing period?

A control with no occurrence in the period has nothing to sample. Find out why: "no leavers in the period" is not the same as "the review that should have happened did not". For a monthly or quarterly control, count every month or quarter in the period: one with no record is an exception, not a smaller population.

• Yes: Take every occurrence yourself from the system of record
• No: Should it have run in the period?

Decision

Should it have run in the period?

A quarterly review whose first quarter has not ended, or a control that started after the period, had no reason to run. A review that fell due and was not held did. If you cannot tell, treat it as a control that should have run.

• No: Conclude Not tested, with the reason and the first due date
• Yes, or not known: Treat the missed run as one failed occurrence

Outcome

Conclude Not tested, with the reason and the first due date

Nothing ran, so nothing can be concluded. Record why and when the control will first run. Not tested is not a pass: it counts as untested in every report until a test is concluded, so put the test back in your plan.

Step

Treat the missed run as one failed occurrence

If the control should have run and did not, that is a failure of the control, not an empty population. Test it as one occurrence that failed, and conclude Not effective.

• Then: Write the result in counts; an independent reviewer signs it

Step

Take every occurrence yourself from the system of record

The system of record is where the trigger is recorded apart from the control: the HR system's leaving dates, not the list of accounts removed; the scanner's detections, not the list of fixes. Extract it yourself or watch it run. Record the system, the report, when it ran, the filter and the count. A list the owner prepared and you cannot check is not a population.

• Then: Does its count agree with a second source?

Decision

Does its count agree with a second source?

Compare the count with a source that records the same events independently, such as payroll for HR leavers. A list the owner filtered can leave out the occurrences that failed. Explain each difference and add any missing occurrence. If you cannot show the list is complete, take the cautious branch.

• Yes, or each difference explained: Choose the sample yourself, spread across the period
• No, or not known: Do not sample; take it to the assurance lead

Outcome

Do not sample; take it to the assurance lead

Do not sample from a list you cannot check. Look for another source first. If none exists, the person who runs control testing decides whether the control has a design deficiency, because it leaves no record that can be tested. A population found to be incomplete is itself an exception.

Step

Choose the sample yourself, spread across the period

The size comes from the pack's default table, by population and risk rating; you may raise it but never lower it. Where the population is no larger than the sample, test every occurrence. Write down the rule for choosing before you look, such as one from each part of the period picked at random. Never take the first, the latest, or the ones the owner points to, and do not tell the owner which you chose until you ask for the records.

• Then: Check every attribute by inspection, observation or re-performance

Step

Check every attribute by inspection, observation or re-performance

Inspect the records the control left, watch it run, or do it again from the same inputs. Asking someone only explains what you saw; an answer is never the evidence. Where a record cannot be found, the attribute fails. Do not let anyone create or correct a record during the test.

• Then: Did any sampled occurrence fail an attribute?

Decision

Did any sampled occurrence fail an attribute?

An occurrence that fails any attribute is an exception, even if another control later caught and corrected it. Write down what failed, by how much, and the reason the owner gives. The reason explains the exception; it does not remove it. This differs on purpose from a maturity assessment, which rates a whole practice with its own checks.

• No: Write the result in counts; an independent reviewer signs it
• Yes: Apply the exception rule before concluding

Step

Apply the exception rule before concluding

One exception in a manual control whose population is at least twice the sample lets you add the same number again, chosen the same way; Effective only if none of them fails. Any other exception, including any in an automated control or a fully tested population, makes the control Not effective. Never drop or replace the occurrence that failed.

• Then: Write the result in counts; an independent reviewer signs it

Step

Write the result in counts; an independent reviewer signs it

The workpaper holds the population and its source, the sample, the attributes, each result, the exceptions and the proposed conclusion, written as counts and never as a share. Someone other than the tester reviews it, by default within 10 working days of the last test step. The conclusion is final only when they sign.

• Then: Is the signed conclusion Effective?

Decision

Is the signed conclusion Effective?

A test ends Effective, Not effective or Not tested. Effective means the control ran as designed on the sample for that period. It does not mean you meet any law or standard.

• Yes: File it and set the next test date from the rating
• No: Log a deficiency, rate it and tell the risk owner

Outcome

File it and set the next test date from the rating

Name any exception the test recorded, and give it to the owner. File the workpaper with its evidence where the owner cannot change it. The next test is due within the control's cycle from the end of this testing period: by default 6, 12 or 24 months for Higher, Standard or Lower.

Outcome

Log a deficiency, rate it and tell the risk owner

A Not effective result is a deficiency, confirmed on the day the reviewer signs. Log it in your own tracker, rate it High, Medium or Low, and tell the owner of every risk that relies on the control at once. Agree a fix with the control owner, rate the control Higher until a retest passes, and close it only on that retest.