How do we define a security metric so the number means the same every month?

Start

You need a number that means the same every month

A change in the number should be a change in the world, never a change in how it was counted. That takes one written definition, the definition card, used by every report that shows the metric. The dashboard, the committee report and the board report then show the same number because they read the same card.

• Then: Can you name the decision it informs, and who takes it?

Decision

Can you name the decision it informs, and who takes it?

Write the question in the words the audience would use, and the decision as something a named person does: fund, reassess, escalate, change a plan, accept. 'Awareness' or 'visibility' is not a decision, because nobody would act differently. If you cannot write the decision, the metric is not needed.

• Yes: Does the process that runs the control already define this measure?
• No, or not sure: Do not collect it

Outcome

Do not collect it

A metric with no decision behind it is not collected, however interesting it is. Do not write the card. If a decision matters but no good number exists for it, report that decision in words rather than with a poor number.

Decision

Does the process that runs the control already define this measure?

Vulnerability, exception, risk, supplier, access, incident and exercise processes each own their headline measures. Search your catalogue and those processes first, and write a new definition only when none fits. A new name for an existing measure makes two reports seem to show two different things.

• Yes: Copy its name, definition and target word for word
• No, after searching: Write the formula and the scope, with a scope count

Step

Copy its name, definition and target word for word

The card takes the owning process's name, definition and target unchanged. It adds only what it needs to run the metric: type, audience, roles, the quality-check numbers, the board measure it feeds and, for a risk indicator, the linked risk. To change an imported measure, change it where it is owned first, then copy the new words to the card with the date. An imported definition reworded 'to make it clearer' is a common reason a card is sent back.

• Then: Write the formula and the scope, with a scope count

Step

Write the formula and the scope, with a scope count

A ratio names its numerator and denominator and is shown as 'n of N', because a percentage alone hides how few items there were. A count says what one item is and its window; a time says its start and end events and whether days are calendar or working days. Every value is as at the period end, and the card says whether higher or lower is better. The scope says what is in and out, with a count of the items in scope: 'all systems' gives the completeness check nothing to test.

• Then: Can a named source give the facts as at the period end?

Decision

Can a named source give the facts as at the period end?

Name the system, register or workbook and the report or export within it. 'The scanner' is not a source; 'the weekly authenticated scan report, all sites' is. Where two sources hold the same facts, choose one, and the other becomes the cross-check. If you do not know whether the source can give the figure as at the period end, treat the answer as no.

• Yes: Name the supplier, validator, owner and reporter as roles
• No, or not known: Record it as not yet measurable, and fix the source first

Outcome

Record it as not yet measurable, and fix the source first

A metric whose source cannot give the facts as at the period end is not yet measurable. Fix the source before the first value is reported, then finish the card. A period of 'current', or whatever date the extract happened to be run, is a common reason a card is sent back.

Step

Name the supplier, validator, owner and reporter as roles

One data supplier per input, one validator, one metric owner who interprets the number and answers for it, and one reporter per audience. Each is a person in a named role, never a team, a mailbox or a tool. The validator is never the supplier, and has a named deputy. Where a provider supplies an input, the supplier is the person in your organisation who receives it; a provider never validates its own figures.

• Then: Set the numbers each quality check will use

Step

Set the numbers each quality check will use

With the validator, write four numbers on the card: the scope count, the extract deadline, the plausible range, and the largest change from last period accepted without explanation. The pack's default extract deadline is 3 working days after the period end. Name any cross-check, such as another process's count of the same things. For a new metric, set a wide range and narrow it after three periods.

• Then: Does it warn that a named risk in the register may be moving?

Decision

Does it warn that a named risk in the register may be moving?

A performance indicator shows whether a control or process works, judged against a target. A risk indicator gives early warning on one risk in your risk register, judged against thresholds. A metric is one or the other, never judged against a target and thresholds at once. A risk indicator with no linked risk is a performance indicator, or is not needed.

• Yes: a risk indicator: Set the thresholds with the owner of the linked risk
• No: a performance indicator: Set the target, and the tolerance the audience accepts

Step

Set the target, and the tolerance the audience accepts

Use the owning process's target unchanged where there is one. The tolerance is the furthest short of target the audience accepts without taking a decision. The status is then one of four phrases: On target; Below target, within tolerance; Outside tolerance; No target set. A metric with no target reaches no audience until one is set.

• Then: Write how it can mislead, then approve the card

Step

Set the thresholds with the owner of the linked risk

Ask the risk owner at what value of the indicator they would score the risk's likelihood one step higher, and at what value the impact too. Amber is the first value that would put the risk outside appetite but within tolerance; red is the first that would put it outside tolerance. Write the reasoning on the card with who is told at amber and at red, and have the risk owner approve the thresholds. Thresholds chosen because they are round numbers are sent back.

• Then: Write how it can mislead, then approve the card

Step

Write how it can mislead, then approve the card

Name how the number can mislead, such as vanity, gaming, lagging, no target or lacking context, and what to read beside it. 'None' is never the answer. Fill every field, writing 'None' where one does not apply: a card with an empty field is not approved. The metric owner and the head of each audience approve it before the first value is reported, and set a review date within 12 months.

• Then: Later, has the definition, source, thresholds or scope changed?

Decision

Later, has the definition, source, thresholds or scope changed?

Record every change on the card first, with its date, before the next value is reported. Change the card before the number. Each kind of change is handled differently, because some earlier values can be restated and some cannot.

• No change: Report it on the same card every period
• Scope only: Update the scope count and explain the step
• Thresholds or target: Mark the break on the zone or status line
• Definition or source: Can earlier values be recalculated on the new basis?

Outcome

Report it on the same card every period

Compare each value with the last period's at the same cadence and say Better, Worse or Same, with the earlier value. A metric shown to two audiences uses the same value and the same card at both; only the detail and the commentary differ. At the yearly review, check every field is still true and set the next review date.

Outcome

Update the scope count and explain the step

A new site, a new critical supplier or an acquisition changes the scope, not the definition. Update the scope count on the card and explain the step in the owner's commentary. It is not a break in the series while the definition is unchanged.

Outcome

Mark the break on the zone or status line

When the risk appetite changes, or the owning process changes its target, the values stay comparable but the zones or statuses before and after do not. Record the change and its date on the card, and mark the break on the zone or status line. A risk indicator's thresholds are reviewed whenever the appetite statement changes.

Decision

Can earlier values be recalculated on the new basis?

For a new definition, restate if the source still holds what is needed. For a new source, run the old and new sources side by side for at least one period first. If you cannot tell whether the earlier values can be rebuilt, treat the answer as no and mark the break.

• Yes: Restate earlier values and show only the new series
• No, or not known: Mark the break, and draw no trend across it

Outcome

Restate earlier values and show only the new series

Recalculate the earlier values from the source on the new basis. Record on the card which periods were restated, and show only the restated series. A risk indicator whose counting window changes needs its thresholds set again.

Outcome

Mark the break, and draw no trend across it

Record the date and the reason on the card. On every chart and table, draw a line at the break and never join values across it. For the first period after it, say 'Not comparable' in place of a direction.