If a metric informs no decision, then do we keep reporting it?

Start

A metric is reported, but nobody acts on it

A metric exists for a decision. If nobody would decide anything differently when the number moves, it is not collected: interesting is not enough. At board level every extra number competes for the same few minutes of attention, and a programme where metrics are only ever added has no review at all.

• Then: Ask what anyone would do differently if it moved

Step

Ask what anyone would do differently if it moved

Write the decision as something a named person does: fund, reassess, escalate, change a plan, accept. 'Awareness' or 'visibility' is not a decision. Vanity metrics, such as attacks blocked, look impressive and always move the right way, but no decision changes because of them.

• Then: Did it change or confirm a decision in the last 12 months?

Decision

Did it change or confirm a decision in the last 12 months?

This is the first of six review questions, and it decides on its own. A metric that informed no decision in the pack's default window of 12 months is retired, whatever else it does well. If you are unsure, answer no: a no is a prompt to look, not a verdict.

• Yes: Does it reach the deciding audience, at that audience's cadence?
• No, or not sure: Is it, or does it feed, a measure the board agreed to see?

Decision

Does it reach the deciding audience, at that audience's cadence?

Every metric in use goes to the security and IT leadership team. It reaches the executive committee only if the committee decides on it, and the board only through the board measures the board agreed. The pack's default cadences are monthly for the leadership team and the executive committee, and quarterly for the board.

• Yes: Are the remaining review questions all answered yes?
• No, or not sure: Move it to the audience that decides on it

Outcome

Move it to the audience that decides on it

The metric informs a decision, so it is kept and fixed: it is shown to the wrong audience. Report it to the audience that takes the decision, at that audience's cadence, and change the audience and cadence on its definition. Close any other gaps the review questions find before its next report.

Decision

Are the remaining review questions all answered yes?

Does it have an approved definition, used unchanged by every report that shows it? Did its inputs pass the quality checks in each of the last three periods? Does it have a target, or thresholds tied to appetite? Does it have one owner and one supplier per input, other than the validator? Answer no if you are unsure.

• Yes: Keep it, and review it again within a year
• No, or not sure: Fix the gaps before its next report

Outcome

Keep it, and review it again within a year

Every answer is yes, so the metric stays as it is. Review it again against the same six questions within 12 months, and set that date on its definition.

Outcome

Fix the gaps before its next report

A metric that informs a decision but fails another question is kept and fixed. Close each gap before the next period. Moving to a catalogue metric that answers the same question, with its own definition, is one way to fix it: in the pack's example, 'access reviews completed' became 'removals confirmed on time'.

Decision

Is it, or does it feed, a measure the board agreed to see?

The board agreed its selection of measures, with their targets and tolerances. Depending on how you are governed, that is the board or a committee it delegated to, the management board in a two-tier structure, or executive management where there is no board. The definition says which board measure a metric feeds. Because that body agreed the selection, changing it is for that body, through its yearly review. If you do not know, treat it as a board measure.

• Yes, or not known: Propose dropping it at the board's yearly reporting review
• No: Stop collecting it at the next reporting period

Outcome

Propose dropping it at the board's yearly reporting review

Once a year the board's reporting is reviewed, and measures the board no longer asks about are dropped. Put the measure forward there, with the reason it informs no decision and any better measure for the same question.

Step

Stop collecting it at the next reporting period

Retire it: stop collecting and reporting it, mark its definition Retired with the date and the reason, and keep the definition. If anyone proposes the metric again, the kept definition shows why it was dropped.

• Then: Did it point at a real need that a better metric could serve?

Decision

Did it point at a real need that a better metric could serve?

Some dropped metrics were a poor answer to a real question. In the pack's example, 'backup jobs completed successfully' informed no decision, but whether critical systems can be restored does. Where that is so, a better metric for the same decision can replace the old one.

• Yes: Replace it with a new metric, with its own definition
• No: Dropped, with its definition and reason kept

Outcome

Replace it with a new metric, with its own definition

The replacement gets a new definition and a new identifier. It is never the old metric renamed, because a metric's identifier is how its history is found. Write its definition before its first value is reported.

Outcome

Dropped, with its definition and reason kept

The metric leaves every report, and its definition is kept with the date and the reason. Fewer, better metrics: the set should grow only as fast as it is pruned. The pack's guidance is that most organisations reviewing an existing set drop a third of what they report.