Control Testing Methodology

Purpose

This methodology is how [[Organisation Name]] tests whether its security controls work. A control that is written down is a promise. A test checks, from records the tester chooses, that the promise was kept: that the control ran, as designed, every time it should have, throughout a stated period.

It exists because a policy, a procedure or an owner's assurance says what should happen and nothing about what did. Auditors, customers, regulators and the board increasingly ask for the second. The method turns that question into steps two testers would follow the same way and reach the same result: the same sample size, the same rules for an exception, the same severity for a failure, and the same deadline to fix it.

It sets the rules CA-01 to CA-20, the sample table, the risk ratings that set how often each control is tested, and the severities and deadlines for a deficiency. Every other document in the Security Control Assurance pack quotes them by number: the Control Inventory & Ownership Register lists the 30 controls with their owners, the Unified Control Framework Mapping Workbook lists the requirement sets you may rely on them for, the Control Test Execution Procedure runs one test, the Control Test Workpaper Template records it, the Control Evidence Standards Checklist says what acceptable evidence looks like, the Control Deficiency & Remediation Tracker follows each failure to its retest, the Annual Control Testing Plan schedules the year, and the Control Assurance Dashboard reports it. The Security Control Assurance Pack — Guide shows how they fit together.

Legal note

This document helps you test whether your security controls work. It is not legal advice, and a test result is not a statement that you comply with NIS2, DORA, the GDPR or any standard. Where a law, a certification or a contract asks for an audit or an independent review, confirm with counsel, your certification body or the other party what it must include.

Scope

The method applies to every control in the Control Inventory & Ownership Register, whoever owns it, and to every test made under the Annual Control Testing Plan: the first test of a control, its routine tests, and the retest of a deficiency. It covers manual controls, such as a review or an approval, and automated ones, such as a system setting.

It tests controls one at a time. It does not rate a whole security practice (the Security Maturity & Internal Assessment pack, P19, does that), audit the management system (the ISO 27001 Implementation & Certification Readiness pack, P08, does that), or test systems for weaknesses, as a penetration test does. It checks records of controls that already exist; it adds no new control.

Who is involved

Eight roles take part. Name who holds each before the first test. One person may hold more than one, except where the independence rule stops it (CA-10).

What a control is and what makes it testable

A control is something done routinely to keep a risk in check, written as who does what, how often, and the record it leaves (CA-01). A control is key when its failure alone would let a risk on the register happen or go unnoticed, or when the organisation names it as its evidence for an obligation. A key control is never rated Lower risk.

A control is tested against its written statement: who does what, how often, and the record it leaves. A control with no written statement, or one whose statement could not keep its risk in check, has a design deficiency, and its operation is not tested until the design is fixed (CA-01).

A statement that names all four parts can be tested. "Access is reviewed regularly" cannot: it does not say who, how often, or what record shows it happened. "The system owner and information security review privileged access every three months and record a decision for every account" can. The tester tests the control against the statement. What the owner says usually happens is background only.

Each control in the inventory lists its attributes: one thing the tester checks on every sampled occurrence, such as "removed by the end of the last working day" or "approved before the change". Attributes come from the statement. For example, leavers' access removed (CT-16) has these: "every account disabled by the end of the last working day in the HR record" and "removal recorded in the provisioning log". A sampled occurrence passes only when every attribute does.

Design before operation

Design is checked before operation: a walkthrough of one occurrence from start to record, at least every [[12]] months and whenever the control changes. An operating test is made only on a design found adequate (CA-02).

The tester takes one recent occurrence and follows it through every step, looking at the record each step leaves. The question is whether the control, run as written, would prevent or detect the risk. A leaver process that removes accounts from the directory but not from systems with local accounts would fail here, however faithfully it runs.

An operating test then checks, on a sample, that the control ran as designed throughout the testing period. A deficient design is recorded, the owner fixes it, and the design is checked again before the operation is tested. Testing the operation of a control that could not work wastes the test and gives a false result.

A design ends Adequate or Deficient. An operating test ends Effective, Not effective or Not tested. Every design is checked at least every [[12]] months, including for a Lower risk control whose operation is tested less often.

How evidence is gathered

Asking someone is never enough. Every test inspects records, observes the control being done, or re-performs it; an interview only explains what the tester saw (CA-03).

Most tests inspect records. Observation suits a control that leaves little record, such as a review meeting; it shows the control can run on the day watched, and nothing about other days, so it is combined with inspection where any record exists. Re-performance is the strongest evidence and the slowest; use it where a control is a calculation or a comparison, such as a reconciliation.

Evidence is acceptable when it meets six tests: relevant (EQ-01), dated (EQ-02), attributable (EQ-03), complete (EQ-04), obtained by the tester (EQ-05) and kept (EQ-06). The Control Evidence Standards Checklist sets out each test, and what acceptable evidence looks like for each kind of control.

Population and sample

Taking the population

The tester takes the population from the system of record, checks it is complete against a second source, and chooses the sample, spread across the testing period. The control owner never chooses the sample or supplies a list the tester cannot check (CA-04).

For a leaver control, the population is every leaver in the period, from the HR system, checked against payroll. For a vulnerability control it is every finding due in the period, from the remediation tracker, checked against the scanner's own list. The second source is what makes the population trustworthy: a list the owner filtered can leave out the occurrences that failed.

If there is no second source to check the population against, do not sample. Take it to the control assurance lead, who decides how its completeness can be shown before the test continues (CA-04). A population found to be incomplete is itself an exception: the control's record does not show every occurrence.

How many to test

The sample size comes from the population band and the control's risk rating. When the population is no larger than the sample, every occurrence is tested (CA-05).

The sample comes from this table. Read the row for the population and the column for the control's risk rating. A Lower risk control uses the Standard column. Where the size is "All", or the population is smaller than the size, every occurrence is tested.

The sizes are the pack's defaults, in line with common practice for testing recurring controls. They are not statistical samples, and they do not measure an error rate. They give reasonable confidence that a control operated. You may raise them, for example for a control whose failure would be severe. Do not lower them.

One table for periodic and event-driven controls

Some controls run on a calendar: a monthly reconciliation, a quarterly review. Others run when something happens: a leaver, a change, a new vulnerability. The table serves both, because it counts occurrences in the testing period, however they arise. A monthly control tested over 12 months has a population of 12, and so does an event that happened 12 times. Both get the same sample, and a tester never needs a second table or a judgement about which kind of control it is.

Sizes for common controls, read from the table:

Choosing the sample

Spread the sample across the period: for a sample of five over six months, take roughly one from each month, chosen by a rule you write down before looking at the occurrences (such as every seventh leaver, starting from a number you pick at random). Never take the first five, the most recent five, or five the owner points to. Record the rule on the workpaper so a reviewer can see the sample was not chosen to pass.

The testing period

The testing period runs from the end of the last tested period, or from when the control started, to the end of the period being tested, and is never longer than the control's test cycle. A control that has not yet run in the period is Not tested, with the reason (CA-06).

Periods follow on from each other, so that over time every month of a control's life falls in some test. A Higher risk control, tested every six months, never has a period longer than six months; a Lower risk control, tested every 24 months, may have a period of up to 24.

A control that started during the period is tested from the day it started. A control that has not yet run in the period, such as a quarterly review whose first quarter has not ended, is concluded Not tested, with the reason and the date it will first run. Not tested is not a pass. It counts as untested in every report until a test is concluded.

EXAMPLE. The distributor's pilot covered 1 Mar 2026 to 31 Aug 2026. Its next test of each control starts its period on 1 Sep 2026. CT-20 Privileged access reviewed started on 21 Sep 2026, after the period ended, so it was concluded Not tested: no review has yet fallen due; the first is due by 21 Dec 2026.

Concluding a test

No exception: Effective. One exception in a manual control whose population is at least twice the sample: the tester adds the same number again, new occurrences chosen the same way, and concludes Effective only if none of them fails, recording the exception. Otherwise, and for any exception in an automated control (even one sampled as a manual control under CA-09) or in a whole population, Not effective (CA-07).

An exception is a sampled occurrence that fails any attribute (CA-07, CA-08). The word means a sampled failure here. It has nothing to do with an exception to a policy approved under the Security Exception, Waiver & Segregation of Duties pack (P02), which this document calls a deferral.

The rule allows one extension, of the same size, after a single exception in a manual control, because one failure in a small sample may be chance. A second failure, in the first sample or in the extension, is not chance. The extension is chosen the same way as the first sample, from occurrences not already tested. It is allowed only when the population is at least twice the sample, so that enough untested occurrences remain. When the whole population was tested, there is nothing to extend to, and any exception makes the control Not effective.

An exception on an Effective test is still recorded on the workpaper, named in the report, and given to the owner. It is not a deficiency. A pattern of single exceptions over several tests is still worth a conversation with the owner about the design.

Worked cases

Each case is run through the method's rule when this document is built. A workpaper or tool that applies the rule should give the same answers.

A failure another control caught

A sampled occurrence that fails an attribute is an exception, even if another control later caught and corrected it. The other control counts when the deficiency is rated (CA-12), not when this control is concluded. This differs from P19's MA-05 on purpose: P19 rates a whole practice with its checks, P20 one control (CA-08).

Controls often back each other up. A leaver whose account is missed at the leaving date may be found a few weeks later by the monthly account reconciliation. The temptation is to say the leaver control worked, because the account was removed in the end. It did not: the account was open, unnoticed, between the leaving date and the reconciliation, and the leaver control is the one being tested.

So the exception stays with the control that failed. The control that caught it matters when the failure is rated: if the leaver control is concluded Not effective and the reconciliation has been tested Effective for the same period, the deficiency is Medium rather than High (CA-12). A reconciliation that has not been tested cannot count, however well it is believed to work.

Why this differs from the maturity assessment

P19's MA-05 says that a sampled record that fails means a maturity level is not met "unless the practice itself caught and corrected the failure and recorded it". That is right for P19, which rates a whole practice, including its own checks. P20 tests one control. Here the reconciliation is a second control, with its own owner, its own record and its own test. The difference is deliberate: the same event can show that access management as a practice is working and that one control within it failed once.

EXAMPLE. In the pilot, leaver L-2026-031 had their last working day on 15 May 2026. ERP account left enabled. Found and disabled by the May reconciliation (CT-18) on 3 Jun 2026. That was an exception of leavers' access removed (CT-16), the only one in the sample of 5. The population of 38 was more than twice the sample, so the tester added 5 more leavers chosen the same way, and none failed. CT-16 was concluded Effective, with the exception recorded and monthly account reconciliation (CT-18) noted as having caught it. CT-18 itself had not been tested, so it could not have counted as a compensating control had the test failed; its first test is planned for Dec 2026.

Automated controls

An automated control is tested once for each configuration, and that test holds only while change control over the system was itself tested effective for the same period. Otherwise it is tested as a manual control (CA-09).

An automated control is one a system performs without a person, such as an account that is locked after repeated failed sign-ins, or an approval step a system will not let anyone skip. Run a thousand times, it behaves the same way each time, unless someone changes it. So it is tested once for each configuration, by inspecting the setting and re-performing it with a test case, with no sample of occurrences.

That single test holds only while nobody could have changed the setting unnoticed. If change control over the system was tested effective for the same period, the setting tested is the setting that ran. If it was not, the tester cannot tell, and the control is tested as a manual control, with a sample of its occurrences from the table.

An automated control has no extension, even when it is sampled as a manual control. One test case or sampled occurrence that fails means the setting does not work as designed, and the control is Not effective (CA-07). None of the 30 controls in the appendix is automated; when you add one, name the system, the setting, and the change control it relies on.

Independence and workpaper review

Who may test

No one tests a control they own or perform (P08 IS-07, P19 MA-14). A check by the owner is recorded as a self-check: it does not count as a test, and the control stays untested (CA-10).

This is P08's IS-07, "Internal auditors must not audit their own work", applied to a control test, as P19's MA-14 applies it to a maturity assessment. The performer, who carries out the control each time it runs, may not test it either.

A small organisation will find this hard: the person who knows the controls best usually runs them. In order, try a colleague from another function, such as quality, finance or HR, trained on this method; a tester from a peer team, each testing the other's controls; internal audit; or a contracted tester. Record which you used. A check the owner makes on their own control is useful to them and is recorded as a self-check, but the control stays untested in every report.

The control assurance lead owns and performs no control in scope, so that every workpaper has someone independent to review it. Where the lead tests, someone who did not test reviews, such as [[internal audit]].

The workpaper and its review

Every test is recorded on a workpaper with the population and its source, the sample, the attributes, the result for each, the exceptions and the conclusion. Someone other than the tester reviews it within [[10]] working days of the last test step; the conclusion is final only when they sign (CA-11).

The Control Test Workpaper Template is the form. The reviewer checks that the population is complete, that the sample was chosen by the rule recorded, that each result is backed by evidence on file, and that the conclusion follows from the results under CA-07. A reviewer who disagrees sends the workpaper back; the tester does more work or changes the conclusion, and both sign the final version.

The review date matters: a deficiency is confirmed on the day the reviewer signs, and its deadlines run from that day (CA-13). A workpaper unsigned [[10]] working days after the last test step is overdue, and the dashboard counts it as untested.

Deficiencies, severity, deadlines and retest

Rating a deficiency

Every design deficiency and every Not effective result is a deficiency. The control assurance lead rates each one when the reviewer signs the workpaper, using this table (CA-12).

A compensating control is another control that keeps the same risk in check when this one fails. It counts only once it has been tested and found effective (CA-12). Name it on the deficiency, with the workpaper that tested it.

Each case is run through the method's rule when this document is built:

Deadlines

The owner agrees a remediation plan within [[10]] working days and fixes the deficiency within [[90]] days for High, [[180]] for Medium and [[365]] for Low, from the day the reviewer confirms it. The risk owner is told at once; a High deficiency goes to management in the next report. A deficiency that cannot be fixed in time is deferred only as a P02 exception (CA-13).

Deadlines are calendar days. The plan names what will change, who will change it, and by when. The risk owner is told the same day, because a risk that relied on the control may need reassessing. A deficiency that cannot be fixed by its due date is not left open past it: the owner requests a deferral under P02 before the date passes, and the approver accepts the risk left while it stays open.

Where the fix is work another pack already set, such as an action from the risk register or a maturity assessment, the tracker names that action instead of starting a second one, and that action must be due no later than the deficiency.

Retest and closure

A deficiency closes only when a retest passes. The retest waits until the fixed control has run at least [[2]] times, or once for an annual or automated control, and is made by someone other than its owner (CA-14).

The owner's statement that the control is fixed starts the retest; it never closes the deficiency. A retest uses the same sample table, over a period that starts after the fix, and a passed retest returns the control from Higher to its usual rating (CA-15). A retest that fails reopens the deficiency with the same due date, and the next report shows it.

Risk rating and how often each control is tested

Each control is rated Higher, Standard or Lower risk, and tested at least every 6, 12 or 24 months respectively. A control never tested is due within one cycle of the start of the first plan year, or of the day it started if that is later; one whose design was found deficient, within one cycle of the day its design is found adequate. A key control is never Lower, and a control that failed its last test is Higher until a retest passes (CA-15).

The rating is set in the inventory, with the reason for every Higher rating. A control is Higher only for a reason the organisation has recorded elsewhere: a risk outside appetite in the risk register, an obligation the responsible body has recorded as not met, or a failed test. A control the tester merely thinks is important is Standard. The rating changes when its reason changes, and the next test date moves with it.

The next operating test is due within the control's cycle from the end of its last testing period. A Higher rating means a larger sample as well as more frequent tests: both follow from the same judgement that a failure here would matter most.

Planning around audits and reviews

The sponsor approves the year's test plan before it starts, and any change to it is recorded. Tests that an audit, a supervisory review or a customer review will rely on are final at least [[6]] weeks before it starts (CA-16).

The Annual Control Testing Plan lists every test due in the year, by month, with its tester. Higher controls appear twice, Standard controls once, and Lower controls in one year of their two. The plan also lists the audits and reviews that will rely on test results: a certification or surveillance audit, a supervisory review, a customer's annual supplier review. Tests they rely on are scheduled so that the workpaper is reviewed and final at least [[6]] weeks before each starts. That leaves time to fix a deficiency, or at least to have its plan agreed, before the auditor asks.

Spread tests across the year. A plan that puts every test in the month before an audit tells the auditor that testing is done for the audit, and leaves no time to act on what the tests find. The assurance sponsor approves the plan before the year starts; any later change, such as a test moved because a tester is ill, is recorded with its new date, and the dashboard reports tests delivered against the plan.

Test once, rely on it many times

Test once, rely on it many times: one test supports every requirement the organisation relies on the control for, as long as the attributes tested include what each of those requirements needs. Which requirement a control is relied on for is the organisation's own record in the workbook; P20 does not say it (CA-17).

Organisations often rely on the same control for several requirement sets at once: a certification, a law, a customer contract. Testing it separately for each wastes effort and produces results that can disagree. If one requirement asks that the record show who approved, and another asks for the date, both are attributes of the same test.

Which requirement you rely on a control for is your decision, recorded in the "Requirements you rely on it for" column of the Unified Control Framework Mapping Workbook. This methodology does not say which control meets which requirement, and no P20 document does. The workbook lists the requirement sets in their own words; you fill in the column, and you check that the control's attributes cover what you entered.

What control testing is not

Control testing is not an internal audit, a certification audit, a P19 maturity assessment or a statement of compliance, and is never presented as one. Its results can feed each of them (CA-18).

A control test is evidence that one control operated in one period. Several other activities look at controls too, each with its own rules on scope, independence and competence. The table says what each is and how test results can feed it. It does not say that a test is, or meets, any of them.

The NIS2 Implementing Regulation (EU) 2024/2690 binds only the providers its Article 1 names, such as cloud computing, data centre and managed service providers. For other entities its Annex is a guide.

Reporting in counts

Results are reported as counts, such as "21 of 30 controls tested", with every deficiency named. There is no overall assurance score, no weighting, and no figure presented as a share of compliance with any framework (CA-19).

A share or a score hides which control failed, and a figure presented as compliance with a framework claims more than any test can show. Counts keep the detail: the reader sees how many were tested, how many worked, and the name of every one that did not. The Control Assurance Dashboard reports these six measures:

A pass rate for a single test is written as a count too: "5 of 5 sampled leavers removed on time". The assurance sponsor receives the dashboard each quarter, and every High deficiency goes to management in the next report (CA-13).

Keeping workpapers

Workpapers and the evidence they rely on are kept for [[3]] years, or longer where law or a contract asks, where the tester and reviewer can reach them and the control owner cannot change them (CA-20).

Keep each workpaper with the evidence it relies on: the population extract, the second source, the sampled records, and the reviewer's sign-off. File them where the tester and reviewer can reach them and the control owner cannot change them, such as a restricted folder or a document system with version history. A workpaper the owner could edit is no evidence that the control operated. An auditor may ask for any test in the last [[3]] years and expect to re-perform it from the file.

Workpapers, the inventory, the plan, the tracker and the dashboard are your own files. Keep them in your own systems.

Worked example: the distributor's pilot round

EXAMPLE. The organisation used throughout the pack is a wholesale distributor with about 900 staff, three warehouses and an online ordering service. It is an important entity under NIS2 in its Member State, and not a financial entity, so DORA's reporting duties do not apply. Before its first full year of testing it ran a pilot round, TR-2026-P, of seven tests.

EXAMPLE. The Chief Operating Officer is the sponsor. The Quality and Compliance Manager, who owns and performs no control in the inventory, has led the programme since 1 Sep 2026 and was the tester for all seven pilot tests; the Internal Audit Manager reviews the lead's workpapers. Fieldwork ran from 21 Sep 2026 to 25 Sep 2026, over a testing period of 1 Mar 2026 to 31 Aug 2026 (CA-06), and the reviewer signed every workpaper on 28 Sep 2026 (CA-11). The sponsor approved the year's plan, for 1 Oct 2026 to 30 Sep 2027, on 30 Sep 2026 (CA-16).

The seven tests

EXAMPLE. Each sample is read from the table by the control's rating when tested (CA-05); each conclusion follows CA-07.

• WP-2026-01, CT-16. L-2026-031's ERP account stayed enabled for 19 days after the last working day. The May reconciliation (CT-18) found and disabled it. One exception, so five more leavers were tested (CA-07): none failed. Effective, with the exception recorded and CT-18 noted as having caught it (CA-08).
• WP-2026-02, CT-13. VUL-2026-0301 and VUL-2026-0318 were fixed after their June deadlines, with no deferral requested (P17 PM-14). Two exceptions: Not effective. P19's assessment found the same misses (MP-18).
• WP-2026-03, CT-22. All 20 sampled joiners completed the induction within 10 working days of their start date.
• WP-2026-04, CT-04. Each of the five sampled reviews was made in its quarter by the owner the register names, with a decision recorded.
• WP-2026-05, CT-24. INC-2026-014, closed on 21 Aug 2026, had its review on 27 Aug 2026, four working days later, with an owner and due date for each action; R-04 was reassessed on 20 Aug 2026 (P05).
• WP-2026-06, CT-20. Design walked through on the Privileged Access Standard approved on 18 Sep 2026. Not tested: no review has yet fallen due; the first is due by 21 Dec 2026.
• WP-2026-07, CT-02. Design deficient: with no register of review dates, nothing shows when a review falls due, and there is no cryptography policy to review (CA-01). Operation not tested until the design is fixed.

EXAMPLE. In WP-2026-02, VUL-2026-0301 (ordering platform database server, P2, due 16 Jun 2026, fixed 29 Jun 2026) and VUL-2026-0318 (remote access gateway, P1, due 16 Jun 2026, fixed 22 Jun 2026) were fixed after their deadlines, and no deferral had been requested. Two exceptions allow no extension, so CT-13 was concluded Not effective. P19's maturity assessment had sampled the same practice and found the same June misses: the two methods agree, from different angles.

The deficiencies

EXAMPLE. The reviewer signed both workpapers on 28 Sep 2026, which confirmed two deficiencies. Both are High (CA-12). Plans were due on 12 Oct 2026 and fixes within [[90]] days (CA-13).

EXAMPLE. DF-02 is a design deficiency, so CT-02 had no operating test (CA-01, CA-02). Both fixes are work other packs had already set, so the tracker links to that work (P05 TA-03, P12 GAP-01 and P19 MF-01) instead of starting new actions. Each retest waits until the fixed control has run (CA-14).

EXAMPLE. One exception was recorded on an Effective test: L-2026-031 on WP-2026-01 (CT-16), ERP account disabled 19 days after the last working day, by the May reconciliation (CT-18). It is named in the report and is not a deficiency.

What the pilot changed in the plan

EXAMPLE. CT-13 Urgent vulnerabilities fixed in time failed its test, so it is Higher until a retest passes (CA-15). Its next test is due six months after its testing period ended, and the plan schedules the retest within that.

EXAMPLE. The inventory rates 3 controls Higher, 21 Standard and 6 Lower; 24 of the 30 are key. The year's plan holds 33 tests. CT-12, CT-14 and CT-26 are Lower and wait for the second year of their cycle. The distributor's largest customer holds its annual supplier security review on 14 Jun 2027, and relies on 5 controls; their tests must be final by 3 May 2027 (CA-16), and the plan has them by then: CT-13 (Feb 2027), CT-16 (Mar 2027), CT-19 (Mar 2027), CT-22 (Mar 2027) and CT-24 (Mar 2027).

The results, in counts

EXAMPLE. As at 30 Sep 2026, reported to the board on 29 Oct 2026. There is no overall score (CA-19).

Common mistakes

The rules in full

The 20 rules of the method, as every P20 document quotes them.

Adapting this template

Framework references

These references show where this document supports an external framework. They indicate relevance only and do not reproduce the text of any standard. Editions referenced: ISO/IEC 27001:2022 incl. Amd 1:2024; NIST CSF 2.0; Directive (EU) 2022/2555 (NIS2) and Implementing Regulation (EU) 2024/2690; Regulation (EU) 2022/2554 (DORA) and Delegated Regulation (EU) 2024/1774; Regulation (EU) 2016/679 (GDPR).

ISO/IEC 27001 references name the clause or control this document relates to. They do not say that meeting it meets any other framework's requirement.

Definitions

Appendix: the thirty controls

Each control is a recurring rule a published CISO Times pack already sets, named by its source rule. The Control Inventory & Ownership Register holds the full list, with each control's statement, attributes, owner, key status, risk rating and last test. Restore testing is not in the set; to add it, quote the Cyber Resilience & Recovery pack's RR-14 or a rule of the Backup & Restore Assurance pack.