When does DORA require threat-led penetration testing, and how do we plan the other tests?

Start

Plan DORA resilience testing

DORA fixes few test frequencies: a yearly test of the systems behind critical or important functions, yearly tests of the continuity and crisis communication plans, and a TLPT every 3 years for entities an authority identifies. Everything else in the programme is yours to set within a risk-based plan. What you are decides which of these bind you, so start there.

• Then: Has your scope assessment shown that DORA binds you?

Decision

Has your scope assessment shown that DORA binds you?

DORA binds financial entities only. The scope assessment settles your entity type, any exclusion, the simplified framework, microenterprise status and whether a TLPT authority has identified you. If nobody has done it, or you cannot tell, do it before planning any test, because every later answer depends on it.

• Yes: Are you a microenterprise?
• No, or not known: Run the scope assessment first; a supplier follows its contract

Outcome

Run the scope assessment first; a supplier follows its contract

An organisation that is not a financial entity does not run this testing programme (DR-22). If you supply ICT services to a financial entity, check your contracts against the flow-down list. For a service supporting a critical or important function, the terms include taking part in the customer's TLPT (FD-05). If you cannot tell which you are, complete the scope assessment before you plan a test.

Decision

Are you a microenterprise?

A microenterprise has fewer than 10 staff and low turnover or balance sheet, and is not a trading venue, central counterparty, trade repository or central securities depository. DORA asks no TLPT of it and no yearly system test; it tests on a risk basis (TR-5). If you cannot tell, take the No branch, which carries the fuller duties.

• Yes: Follow your risk-based testing plan; no TLPT
• No, or not known: Are you under the simplified framework?

Step

Follow your risk-based testing plan; no TLPT

A microenterprise tests on a risk basis with a strategic testing plan and sets its own frequency (TR-5, RQ-35). It plans no TLPT (DR-16). Under the full framework it still has the plan tests of TR-2 and the exit plan review of TR-6, so carry on to the plan tests. The pack's yearly dates are still a good discipline, but your plan sets the frequency.

• Then: Add the plan tests: continuity, crisis communication, exit

Decision

Are you under the simplified framework?

The simplified framework is for the entities of Article 16(1), such as small and non-interconnected investment firms and exempted payment institutions. DORA asks no TLPT of them, so they never plan one (DR-16). They still test the systems behind critical or important functions yearly. If you cannot tell, take the No branch, which carries the fuller duties.

• Yes: Keep an ICT security testing plan; no TLPT
• No, or not known: Has a TLPT authority told you in writing to do TLPT?

Step

Keep an ICT security testing plan; no TLPT

An entity under the simplified framework keeps an ICT security testing plan and tests its backup and restore procedures at least yearly, or on every major change to the plan (TR-3, RQ-40). It plans no TLPT (DR-16). The yearly test of each system behind a critical or important function still applies, so list those systems next (TR-1).

• Then: List the systems behind critical or important functions

Decision

Has a TLPT authority told you in writing to do TLPT?

Being a regulated financial entity does not mean you do TLPT (PF-08). The Delegated Regulation names kinds of entity, such as systemically important banks and the largest payment institutions, that authorities require it of unless their assessment shows it is not justified. The pack plans TLPT only after the authority has told you in writing. If you cannot tell, ask your competent authority before relying on No.

• Yes: Agree a scope on live systems, with providers in it
• No, or not known: List the systems behind critical or important functions

Step

Agree a scope on live systems, with providers in it

A TLPT covers several or all critical or important functions and runs on the live production systems behind them. You identify every relevant underlying system, including those outsourced to ICT third-party providers, and the authority validates the scope. You take measures to make sure providers in scope take part, and you keep full responsibility for compliance. Provider contracts should say so (FD-05).

• Then: Choose testers that meet DORA's tester criteria

Step

Choose testers that meet DORA's tester criteria

Testers need the highest suitability and reputability, expertise in threat intelligence, penetration testing and red teaming, certification or a code of conduct, independent assurance on managing TLPT risks, and professional indemnity insurance. Internal testers need the authority's approval, the threat intelligence provider is external, and with internal testers you contract external ones every three tests.

• Then: Plan to the TLPT dates and close with the authority

Step

Plan to the TLPT dates and close with the authority

Send the initiation information to the test managers within 3 months of the authority's notice. Use at least 3 scenarios, no more than 1 of them not threat-led, and an active red team phase of at least 12 weeks. The red team report follows within 4 weeks of that phase ending. The blue team report and replay follow within 10 weeks; the TLPT sheet counts from the same end date, so check it against the test managers' notice. Summary and remediation plans are due within 8 weeks of the authority's notice that reports are adequate.

• Then: List the systems behind critical or important functions

Step

List the systems behind critical or important functions

Every ICT system and application behind a critical or important function is tested at least yearly (TR-1, RQ-33). List each with its date in service, which starts the clock for a system never tested. A system whose function is unassessed counts as behind one (DR-10, a working rule of the pack). Judgement, not a pack rule: a TLPT does not replace these tests, because the pack keeps DT-13 and TR-1 apart.

• Then: Choose the other test types and say why you left some out

Step

Choose the other test types and say why you left some out

DORA lists twelve kinds of test (DT-01 to DT-12) and fixes no frequency for them. Pick the ones your risk assessment supports, set an interval for each and record why any type is left out (RQ-34). The pack's defaults are every 3 months for vulnerability scans and 12 months for most others, with compatibility testing run on change. Penetration testing (DT-12) is not TLPT (DT-13).

• Then: Will someone independent of the builders and operators test it?

Decision

Will someone independent of the builders and operators test it?

A test counts towards the yearly test only if it was held, an independent party did it and it is dated no later than the date you report at. Independent means apart from the people who built or run the system, internal or external. An internal tester needs enough resources and no conflict of interest. If you cannot tell, take the No branch.

• Yes: Add the plan tests: continuity, crisis communication, exit
• No, or not known: Find an independent tester before you count the test

Outcome

Find an independent tester before you count the test

Use an external firm, or an internal person apart from the system's builders and operators who has the resources and no conflict of interest. Enter a test held by the system's own team in your test log with Independent set to No, and book a test that counts. Until one is held, the system has no counting test for the yearly rule.

Step

Add the plan tests: continuity, crisis communication, exit

Under the full framework, test the continuity, response and recovery plans at least yearly and after a substantive change to the systems behind critical or important functions (DT-14, TR-2), and test the crisis communication plans (DT-15). Under the simplified framework TR-3 replaces TR-2. DORA says only 'periodically' for exit plans, so the pack defaults to 12 months (DT-16, TR-6). The recovery test is P22 RR-14 and the restore test is P23 BR-13. Cite their records.

• Then: Are you counting a tabletop or drill as a test?

Decision

Are you counting a tabletop or drill as a test?

An exercise or tabletop under the incident exercise pack tests decisions and roles. P21 never counts one as a test of a system (DR-13). Only a recovery or restore test whose record meets its own pack's rules counts as the plan test DT-14. If you cannot tell what a planned activity is, take the Yes branch.

• No: Log every test and send each finding to your tracker
• Yes, or not known: An exercise is not a test; add a real one

Outcome

An exercise is not a test; add a real one

Keep the exercise for what it shows: decisions, roles and communication. Plan a test that exercises the system itself, or cite a recovery test (P22 RR-14) or restore test (P23 BR-13) whose record meets their rules, which P21 then counts as DT-14. Do not enter the exercise in the test log.

Step

Log every test and send each finding to your tracker

Log the date, type, tester, result and findings of every test, including those that did not count. Enter each finding in your own tracker with an owner, a priority and a due date that follows the priority (DR-06, DR-14). A person other than the one who fixed it confirms the weakness is fully addressed.

• Then: Is any system or plan test overdue or never done?

Decision

Is any system or plan test overdue or never done?

For each system the programme gives a state: Current, Overdue, or Never tested. Plan tests show the same against their due dates. Report counts with names, such as 4 of 5 systems tested in time, never a percentage (DR-04). If you cannot tell whether a test counted, take the Yes branch.

• Yes, or not known: Enter each as a gap and tell the management body
• No: Book the next tests and review the programme each year

Outcome

Enter each as a gap and tell the management body

An overdue or never-tested system, or a late plan test, is a gap: enter it with an owner, an effort and a due date. Operating needs every system Current. A binding readiness item rated below In place is a blocking gap, Priority 1, and goes to the management body at its next meeting (DR-05). A binding item In place but not operating is Priority 2. Report counts with names.

Outcome

Book the next tests and review the programme each year

Put each next due date in your calendar and keep the records behind each test. Review the programme at least every 12 months, and after a substantive change to the systems behind a critical or important function. Report the counts to the management body. If you did a TLPT, plan the next at least every 3 years unless the authority changes the frequency.