Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Cyber Security Metrics
Cyber security metrics are one of my favorite concepts to talk about. I talk about them so often that my colleagues must believe that this is a “Soapbox” that I visit frequently. Whenever someone asks me what my favorite metric is, I immediately think IT’S A TRAP! Many colleagues that I’ve talked with feel the same way, believing that the person asking is trying to “stump the chump” or find a way to discredit the security leader. Why do I feel this way? I might be a little professionally paranoid, but I believe that we should be wary whenever discussing cyber security metrics. In order to explain my perspective, let’s start with a textbook definition of security metrics to get the conversation started:
Security metrics are quantifiable measurements used to understand the status of systems and services through the collection, analysis and reporting of relevant data. Security metrics are based on security objectives that help inform decisions on how to improve the security of all components involved in delivering services and processing data. Source: Gartner
No, this blog is not sponsored by Gartner, but they have a reasonably good glossary that is generally useful for communicating cyber security concepts to the masses, so we’re going with it. This definition seems pretty straightforward, so what’s the problem? The problem lies in the fact that each organization’s security objectives may be different because each organization has different business risks, and just like business offerings, security capabilities evolve over time. Because of this, cyber security metrics that work for one organization will not necessarily work for another. There is also the concept of relevant data where organizations at times (intentionally or unintentionally) provide metrics that do not show the true cyber security risk that the company faces.
So how do we not fall for the trap? How do we explain cybersecurity metrics to a non-technical audience in a compelling way? To make sure that we don’t put our audience to sleep, we have to provide context to be relevant. So how do we do that? Great storytelling involves a concept, and the concept often answers a very simple question. To tell a cyber risk story, our metrics must answer simple questions about the operational effectiveness of our cybersecurity program. When we’re able to answer these questions and inspire action, then we’re doing the job.
Why all the questions?
Storytelling is certainly a skill that many cybersecurity professionals need to develop because it provides the backdrop regarding why a particular set of information should inspire action. The goal of metrics in the context of cyber governance is to tell you enough information to give you the confidence to relax in your chair, or to get up and do something about it. Note: I came up with that metaphor before standing desks…but I think it still works. Stories with context inspire action; data inspires narcolepsy. In order to inspire action and to tell a story, I am introducing to you the Goal, Question, Indicator, Metric (GQIM) methodology developed by Carnegie Mellon University’s Software Engineering Institute.
Before you scoff at the age of this resource (circa 2015) I want you to think about the number of articles, blogs, books, and online resources dedicated to security metrics written since then. New articles written about the “10 Metrics You Need Now” pop up all the time however very few organizations teach you how to create metrics. The GQIM methodology is my go-to resource when I teach others how to create meaningful metrics. If used properly, it provides built-in context because it is designed to answer simple questions. The approach is organized and is defined as follows:
In addition to defining GQIM itself, the GQIM report details how to create each dimension, providing examples along the way. It also offers introspective questions about whether you are addressing your audience and other considerations. One note about the Metric dimension: The original report doesn’t focus on what a “good” or “bad” value is for the metric chosen. I recommend that leaders add this qualitative dimension to clearly indicate if the organization is doing well (or not).
Security Awareness Metric Example
Ok, so how does this work? Let’s walk through the GQIM process for security awareness:
As far as refinements or improvements on the example, there are several. The questions that I posed above could be brought up a level or two higher to help your business leaders understand the story you are trying to tell. Also consider inverse indicators depending on company culture, meaning instead of identifying the percentage of employees that fail training, you might measure the inverse or the percentage of employees that pass the test. Your mileage may vary.
There was an update to GQIM?
Wait? There was an update? Why didn’t we just start there? Just like our cyber security metrics should tell a story, I want to guide you through the story of how our industry has evolved over time. While security metrics have improved over the years, and security leaders have developed and improved their Cyber Governance programs, something was still missing: Context. Didn’t answering senior management’s questions solve the context problem? It did, to a degree but most of our metrics were developed at the control level and honestly, business leaders don’t care much about security controls. They care about their business and in the intervening years, we have learned that we need to align our talking points and our cyber security metrics to business risk or we’ve lost the audience.
The update to GQIM came last year where researchers at Carnegie Mellon University showed that “Assets Support Critical Services.” Providing security risk context regarding the cyber security posture of underlying business systems, data, and people that support a particular business operation and revenue stream gets business leaders’ attention. It also helps create internal competition – “I run at a lower level of risk than you do,” helps drive investment, and creates the conditions for positive / candid conversations about risk.
So how might we break apart the example GQIM example to align with a business line? How do we highlight the biggest human risks that the organization faces, show off well-managed risk, and spotlight opportunities for improvement? Here is one approach:
The graphic above helps business leaders understand in which teams leaders need to get more involved in solving the problem. The example above also helps us identify candidates for more rigorous technical or financial controls to better manage the organization’s business risks.
Cyber Security Metrics 101
Now that we have a basic understanding of what cybersecurity metrics should look like and how we can tell a story with them, we’re ready to begin creating metrics! Right?!? We are, however I’m going to bring our Cyber Governance Mission Statement back to the conversation to see if we’ve forgotten anything:
Cyber Governance is a group of technologies, activities, and choices that achieve the following outcomes:
– Provide real-time telemetry regarding cyber security control coverage/effectiveness and the usage of organizational assets (data/applications/resources).
– Align security controls and asset usage to specific business risks and enable the organization to identify existing/evolving risk exposure and wargame what-if scenarios.
– Make informed decisions both at the tactical day-to-day operational level, during emergent scenarios, and at enterprise strategic planning events.
Source: Cybergovernance.net
The biggest concepts that we have not yet really explored are to make sure that when we’re developing metrics to govern our cyber programs, we need to take the following dimensions into account, from day one:
Before we close, I’d like to thank a few different organizations for their contributions to security metrics in general and for some high level ideas related to this topic. They include:
Key Metrics and Vulnerability Management Maturity Model Poster – Source: SANS Institute
Key Metrics for a CISO Dashboard – Source: IANS Research
SANS Security Awareness Blog Post | Security Awareness Metrics – What to Measure and How – Source: SANS Institute
Epilogue
Cyber Governance relies on information organized clearly into cyber security metrics in order to enable informed decision making. As leaders, we have to provide line of sight on the coverage, effectiveness, and validity of the security program, and we also must provide context in order to effectively communicate with our audience. Remember our mission statement – cyber security metrics should inspire and enable decision making – metrics and reporting is not a “check the box” activity. If it is, consider revisiting your metrics to ensure that it’s clearly communicating the cyber security risks you see in your business environment.
We’re not going to be able to solve cyber governance metrics in one go – and that’s ok! What deep-dive metrics do you want us to explore? What did you learn? What do you think? Share, Comment, Like Below!
Published:
Updated: