The 5 baselines any CISO must use.

In our ever faster changing world, CISO’s need to know whether their security programs are keeping up with the evolving threat landscape. Here are 5 baselines that any CISO should use to measure the effectiveness of their security strategy.

The BOOM framework

This blog is nothing more than a shameless plug for the book “The metrics manifesto” written by Richard Seiersen. In this awesome book, Richard describes the BOOM framework (Baseline Objectives and Optimization Measurements). He describes 5 baselines that are foundational for measuring security performance and that allow you to articulate clear and measurable objectives to measure whether your security programs are actually making your environment more secure or not. Next to defining these terms, it also contains tons of details of how to implement the BOOM framework, including source code.

The BOOM framework consists of the following 5 baselines: survival analysis, burn-down metrics, arrival rates, wait times and escape rates. The baselines are based on the now common understanding among CISOs that events will happen and not all issues can be prevented. These baselines are designed to look beyond just counting events, but measure resiliency improvements instead.

Survival Analysis

The first baseline is about measuring the survival time of events. By understanding how long events (like vulnerabilities or other risk causes) survive in your environment, and how that depends on the type or severity of event, you are reducing the uncertainty about your risk exposure. An example of this baseline is: “50% of critical vulnerabilities live for 48 hours or longer”. You need to avoid using averages over clusters of types of events and gather as much fine-grained data as possible.

Burn-down Metrics

The second baseline is concerned with the ratio of removed risk (in a certain timeframe) over the total new risk that is out there. It measures whether you are mitigating risk faster or slower than the growth of risk. For example: if in the last month you had 100 new vulnerabilities, and your team was able to remove 60, the burn-down rate is 60%. Although this is lower than 100% and hence risk is increasing, if the burn-down rate was 50% the month before, you still know you are improving. This is a very useful performance metric that developer teams can use to measure their own performance, and for CISOs to compare teams to decide on which team to focus to make the biggest impact on security in the coming period.

Arrival Rates

Burn-down metrics meaure how fast you are removing risks, the third baseline is measuring the rate at which risks emerge. Predicting what will happen tomorrow is difficult, but by leveraging intel-feeds and historical data (for your environment), you can build probability-curves that show the chances of a vulnerability for one of your technology stacks being reported coming month. Arrival rates are useful to know because the arrival of a new vulnerability defines work for your team. This baseline therefore helps you make decisions on resource allocation.

Wait Times

The fourth baseline, wait times, is a well known measurement in operations management. It is measuring the time between arrivals of risk causes, like vulnerabilities. Knowing this metric, helps you optimise your security operations teams. But it should also be used as a leading indicator for risk: if wait times are decreasing, risk is increasing.

Escape Rates

The last baseline is measuring how risks migrate across your environment. In particular, it measures the rate at which risk-causes move from an environment with one state of control to an environment with a lesser state of control. For example, it measures the rate at which risks are moving from your development environment to the production environment. In other words: the rate at which those risks “escape”. Modern software development teams are increasing their release velocity. This would increase the escape rate too, unless your security program is able to reduce escape rates without reducing release velocity.

I would urge any CISO to read this book and start measuring these baselines, see the metrics change over time and tune your security strategy towards improving these metrics over time. Use these baselines to start setting goals for your security strategy and use the metrics to articulate information risk appetite. Any security program that can improve these metrics over time, is improving the security of your environment and now you can prove that to your management boards!