Made to Measure: How to Build Better Cybersecurity Metrics

Security isn’t a one-time process with a set end point: it’s an ongoing quest to stay ahead of the bad guys.
And metrics are a vital part of the cybersecurity arsenal. Get them right and you can quantify risks and vulnerabilities and communicate your findings through your organization.

Metrics are living and dynamic, and specific to narrow domains in your cybersecurity framework. No matter how many strides forward in maturity and efficiency you’ve made, you should constantly assess the value each metric gives you, eliminating or changing them as needed. So, if metrics are supposed to measure improvement across multiple dimensions, how should you decide which ones to track?

The Problem with Metrics

The biggest metrics challenge is that the threats they measure are always evolving. Almost two-thirds of organizations say that attacks are getting more sophisticated each year. With some industries facing regulatory changes, new issues are always bubbling up – and your choice of metrics must keep up with them.

In this context, alignment across your organization is essential for tracking cyber resilience. That means being judicious both in the way you analyze metrics and the way you share them. CISOs won’t always be able to present to the board directly (although, thankfully, this is changing). When they do, they may end up sharing reports that look good — such as the volume of blocked threats — but do not guide action. Others may end up sharing metrics that are too detailed for a non-technical audience, or don’t represent business goals.

There’s No Single Solution to Finding the Right Metrics Mix

It’s hard to give a single answer to the best metrics for a given organization. The best solutions will vary from sector to sector and company to company. Meanwhile, different board members and departments will have different tracking requirements – but trying to fulfil everyone’s wish list may leave you feeling like Santa without the sled.

Timeliness will also shape the reception metrics get – if threats are imminent, more technical detail will be required, while if your company focused on its hemorrhaging market share, hard cyber metrics may not get the most receptive audience.

But while there’s no single set of metrics that will work for every situation, there are approaches that can help you pick the ones most useful for you and your stakeholders.

Start with the End Goal in Mind

There are many models out there of the best metrics and reports to collect and share. Your metrics may be built around risk or maturity models, for example. You may present metrics based on the CARE (consistent, accurate, reasonable, effective) framework, or map reports to the CIA triad. Gap analysis can compare actual with desired performance, using a risk-focused framework to produce a detailed cybersecurity picture.

Other approaches focus on key operational metrics, such as the mean time to detect and respond, the number of systems with known vulnerabilities, traffic volume, and an assessment of user access levels.

There can be endless combinations, and it’s all too easy to get lost in the weeds. More isn’t always better when it comes to data (heresy, I know). Your starting point should always be: what is my organization trying to accomplish? Your business goals should be dictating which metrics you track. Are you looking to minimize risk? Are you looking to lower insurance costs? Or are you focusing on compliance? Having that kind of clarity up front will help you avoid vanity metrics and ensure you’re tracking the metrics that matter.

Consolidate and Automate Reporting Where Practical

The need to gather, format, and make data presentable can make reporting metrics a time-consuming chore, especially if data has to be collected from different sources. But it doesn’t have to be this way. Can data be extracted and normalized before being loaded into a central repository? Can the repository be set up so that metric consolidation can be automated, even partially?

While tremendously useful in certain environments, automation is not an end to pursue for its own sake. It might make more sense to prioritize particularly time-consuming tasks and automate these first, then consider automating less labor-intensive functions.

Let the Process Guide You

Some metrics are quick and easy to produce, while some require a lot of legwork and a lot of inputs before they’re ready to go into a report. If a given metric is labor-intensive to produce, that may be a sign that it’s not particularly worthwhile. The best solutions are often the simplest ones, and exploring the use case for individual metrics can often help you decide whether they can be simplified or junked altogether.

Similarly, time spent documenting metrics can be very valuable. By noting the source of the data, the steps required to create the metric, the metric’s frequency, and its intended audience, you can discover flaws or inefficiencies in the metrics’ data or production process.

Streamline Your Reports and Focus on Your Audience

We’ve noted how you can consolidate data, but how can you consolidate your reports? If you’re reporting a different set of metrics to different audiences on overlapping schedules, there may be serious efficiency savings to make. Can schedules be aligned? Do some metrics do a similar job?

That doesn’t mean you should stop personalizing reports for different parts of your company. Reports should be alive to their priorities, delivering metrics that are relevant and self-explanatory. Feedback from recipients can help you work out whether individual metrics deliver value or can be optimized or junked.

Such a process can help you hone in on a limited number (perhaps as few as four) of crucial, relevant metrics. This is particularly relevant for reports produced for your board, which should be geared towards overall business goals, rather than highlighting technical information. A few well-placed numbers or graphics with an executive summary will have far more sway with the board than endless columns and rows of data points.

The Bottom Line

As we’ve seen, just as the search for cybersecurity never ends, so the search for the ideal metrics never ends. If it did, cyber teams around the world would be out of a job!

Instead, the way we source, manage, and present metrics should be constantly scrutinized. Can we make our metric mix more efficient? Can reports be simplified? How do metrics contribute to our department’s goals and wider business goals? By keeping these questions in mind and encouraging our teams to think in whole-of-business terms, we can build metrics that help us stay one step ahead of the bad guys.

**This blog was originally published on April 20, 2022.