Security metrics and KPIs: making security measurable

"You can't manage what you can't measure" is especially true for IT security. Without metrics, no one knows whether the security program is effective, whether investments are justified, or whether risk is increasing or decreasing. This article provides a comprehensive KPI framework for various security areas.

Why Metrics Fail (and How to Do It Better)

Common mistakes with security metrics:

Mistake 1: Activity metrics instead of outcome metrics

• Bad: "We patched 150 vulnerabilities this month"
• Good: "Critical patch compliance rose from 73% to 91%"

Mistake 2: Too many metrics

• 50 KPIs - no one understands what's important
• Solution: 5-7 strategic KPIs + 15-20 operational metrics

Mistake 3: Metrics without a baseline and target

• Bad: "MTTD: 48 hours"
• Good: "MTTD: 48h (Baseline: 96h, Target: <24h, Trend: down)"

Mistake 4: Wrong audience

• Showing technical metrics to the board - eye-rolling
• Board metrics: Risk, Business Impact, EUR
• Ops metrics: MTTD, FP rate, incidents per week

Good metrics framework:

• Define the goal (what needs to improve?)
• Establish a baseline (measure the initial state)
• Set a target value (realistic and ambitious)
• Determine measurement method (how is it measured, by whom, how often?)
• Reporting cadence: daily/weekly/monthly/quarterly

Security Operations KPIs

Detection & Response Metrics:

MTTD (Mean Time to Detect)

• Definition: Time from the start of the attack to detection
• Formula: Sum of (Detection time - Start of attack) / Number of incidents
• Benchmark: IBM DBIR 2024: 204 days (!)
• SOC Target: < 24 hours for critical incidents
• Best-Class: < 1 hour
• Measurement: SIEM timestamp of incident detection vs. forensic analysis

MTTR (Mean Time to Respond/Contain)

• Definition: Time from detection to containment
• Formula: Sum of (Containment time - Detection time) / Number of incidents
• Benchmark: Industry: 73 days to containment (IBM DBIR)
• SOC Goal: < 4 hours for critical incidents
• Measurement: Ticket system: "Incident detected" -> "Containment confirmed"

MTTF (Mean Time to Fully Remediate)

• Definition: Time from detection to full remediation
• Formula: Sum of (Close time - Detection time) / Number of incidents
• Target: Critical < 24h, High < 7 days, Medium < 30 days

Alert Quality Score

• Definition: Percentage of true positives among all alerts
• Formula: True Positives / (True Positives + False Positives) x 100
• Target: > 70% (below 70%: Alert Fatigue!)
• Why: Too many FPs - analysts ignore alerts - blind

Incident Volume

• Definition: Number of incidents per period by severity (trend is more important than absolute number)
• Critical: < 5/month (Target)
• High: < 20/month
• Medium: Baseline x 1.2 (sharp increase = problem)

---

Table for SOC Reporting:

Vulnerability Management KPIs

Patch & Vulnerability Metrics:

Patch Compliance Rate

• Definition: % of systems with patch installed within SLA
• Formula: Patched systems / All affected systems x 100
• Target: Critical: >99%, High: >95%, Medium: >90%
• Critical SLA: Critical patches: 24h external, 72h internal
• Measurement: Vulnerability scanner post-scan comparison

Mean Time to Remediate (Vulnerability)

• Definition: Average time from discovery to patch
• Formula: Sum of (Patch date - Discovery date) / Patches
• Target: Critical: < 3 days, High: < 14 days, Medium: < 45 days

Vulnerability Discovery Rate

• Definition: Time from CVE publication to detection in the system
• Target: < 48 hours (for Critical CVEs)
• Measurement: NVD publication date vs. scanner result

Overdue Findings

• Definition: Number of findings outside SLA
• Target: 0 Critical/High outside SLA
• Escalation: Any violation - automatic CISO notification

Attack Surface Trend

• Definition: Trend of open findings over time
• Visualization: Burn-down chart by severity
• Rising: New vulnerabilities > fixes (alarming)
• Stable: Steady state (acceptable)
• Declining: Active reduction (optimal)

Scanner Coverage

• Definition: % of known assets that are scanned
• Formula: Scanned IPs / Total IPs x 100
• Target: >95%
• Risk at <90%: Blind spots in the vulnerability program

---

Dashboard Example (Monthly):

Security Awareness Metrics

Phishing Simulation KPIs:

Click-through rate

• Definition: % of recipients who click on a phishing link
• Industry average: 17-25% (without training)
• Target after training: < 5%
• Best-in-class: < 2%
• Warning sign: > 15% after 6 months of training

Report Rate

• Definition: % of recipients who correctly report phishing
• Target: > 30% (report first, then click)
• Why it matters: Reporting is more important than not clicking!
• Only genuine reports count (not "what was that?")

Time to Report

• Definition: Time from phishing receipt to report
• Target: < 30 minutes
• Best-in-class: < 5 minutes (real-time defense!)

Repeat Clicker Rate

• Definition: % of users who click on phishing multiple times
• Target: < 3% after training
• Action: Repeat clickers - intensive training (1:1)

---

Training Completion Metrics:

Mandatory Training Completion

• Definition: % of employees who have completed the mandatory course
• Target: 100% (Compliance requirement!)
• Monthly report to management

Knowledge Retention

• Definition: Test score 30/60/90 days after training
• Measurement: Follow-up quiz (automatically in LMS)
• Target: >80% retention after 30 days

Training Effectiveness (Phishing)

• Definition: Click-through rate before training vs. 3 months afterward
• Calculation: (Before - After) / Before x 100
• Example: 23% -> 7% = 70% reduction - training works!

Compliance and Audit Metrics

Compliance KPIs:

Control Effectiveness Rate

• Definition: % of controls implemented in compliance with audit requirements
• Calculation: Implemented Controls / Total Controls x 100
• Target: >95% for critical controls

Open Audit Findings

• Definition: Number of unresolved audit findings by severity
• Critical: 0 open for over 30 days (Mandatory!)
• High: 0 open for over 90 days
• Medium: < 10 open

Remediation Rate

• Definition: % of audit findings resolved on time
• Target: >90%

Risk Register Health

• Definition: % of risks with a recent review (< 6 months old)
• Target: 100% (ISO 27001 requires regular review)

Policy Exceptions

• Definition: Number of active policy exceptions
• Target: Declining trend
• Exceptions > 6 months old - Escalation required

Management and Board Reporting

Board-Level Security Metrics (non-technical!):

1. Security Risk Rating (traffic light system):

• RED: Critical unpatched risk OR active incident
• YELLOW: SLA violation OR compliance gap
• GREEN: All KPIs within target range
• Board members want: trend, not details

2. Cyber Insurance Readiness:

• Are insurance requirements met?
• Impact on premium: which measures reduce it?
• Benchmark: where do we stand compared to the industry?

3. Cost of Security vs. Cost of Breach:

• Security budget: 150,000 EUR/year
• Average breach cost for SMEs: 3.5 million EUR (IBM 2024)
• ROI = (3,500,000 x probability reduction) / 150,000
• Concrete business case for security investments

4. Regulatory Compliance Status:

• NIS2: In implementation (75% of requirements met)
• ISO 27001: Certified, next audit: Q3 2026
• GDPR: Last data breach: none (18 months)

5. Incident Summary (Board-level language):

• "This month: 2 security incidents. Both contained within 4 hours. No data loss. No customer impact. Cause resolved."

---

Reporting Frequency: