The Fault in Our Metrics: Rethinking How We Measure Detection & Response - Allyn Scott
2024-06-08, 14:30-15:15, Track 3 (Moody Rm 102)
Your metrics are boring and dangerous. Recycled slides with meaningless counts of alerts, incidents, true and false positives… SNOOZE. Even worse, it’s motivating your team to distort the truth and subvert progress. This talk is your wake-up call to rethink your detection and response metrics. You’ll get a practical framework for developing your own metrics, a new maturity model for measuring capabilities, and lots of visual examples of metrics that won’t put your audience to sleep.
Description
Metrics tell a story. But before we can describe the effectiveness of our capabilities, our audience first needs to grasp what modern detection and response is and its value. So, how do we tell that story, especially to leadership with a limited amount of time?
Measurements help us get results. But if you’re advocating for faster response times, you might be encouraging your team to make hasty decisions that lead to increased risk. So, how do we find a set of measurements, both qualitative and quantitative, that incentivizes progress and serves as a north star to modern detection and response?
Metrics help shape decisions. But legacy methods of evaluating and reporting are preventing you from getting the support and funding you need to succeed. At the end of this talk, you’ll walk away with a practical framework for developing your own metrics, a new maturity model for measuring detection and response capabilities, data gathering techniques that tell a convincing story using micro-purple testing, and lots of visual examples of metrics that won’t put your audience to sleep.
What’s new in this talk?
This talk presents a new approach to detection and response metrics. I propose moving away from the typical approach of measuring effectiveness solely based on quantitative indicators, such as event counts, which are often used by security operation centers or legacy detection and response programs. I introduce a new maturity model for measuring detection and response capabilities. I provide a methodology for utilizing micro-purple testing - tests that validate detection logic and analysis and response processes - to measure overall visibility into threats. Finally, I walk the audience through a practical framework that will help them develop their own metrics.
Key takeaways
A new maturity model that helps tell the story of modern detection and response, the value it provides, and how your current capabilities level against your goal state.
Visual examples of metrics you can use today to present across teams and leadership, along with a framework for developing your own detection and response metrics and practical advice on how to strategically move to these modern metrics when change is hard and leadership hates surprises.
Methods to measure and prioritize threat coverage with micro-purple testing - tests that validate detection logic and analysis and response processes.
Who will enjoy this talk?
A CISO that wants to better understand what modern detection and response metrics should look like and how to include them in their overall program metrics.
Managers and directors that present detection and response metrics to leadership and the rest of their organization.
Engineers and analysts that are tired of their work being misrepresented with sad, unmotivating metrics.
Anyone interested in learning more about detection and response.