Metricon X — Proceedings

Metricon X — Proceedings

March 27, 2019
metricon

Metricon X was held on March 21st and 22nd at the Stevens Institute of Technology in Jersey City, NJ. The theme of the conference was: “Metrics that Matters - Help Management with Decision Making and Improve Security Posture of the Organization.” The agenda, presented materials, notes, attendees and session descriptions follow. Chatham House Rules were in effect. Forty-eight (48) people attended.

Agenda #

Day 1: March 21, 2019 #

Day 2: March 22, 2019 #

Meeting notes are here.

Attendees #

  • Dennis J Amundson, US Bank
  • Josh Alvarez, Key Bank
  • Ed Amoroso, TAG Cyber
  • Christa Anderson, Microsoft
  • Wake Baker, Cyentia
  • Jennifer Bakuk, Decision Framework Systems
  • Nandita Chakraberty, Microsoft
  • Dan Chapman, Convertive
  • Jason Crabtree, Fractal Industries
  • Joseph Cronin, New York Medical College
  • Benjamin Charles Dean, Consultant to OECD Secretariat
  • Chris Eng, Veracode
  • Brian Gay, Northramp
  • Dan Geer, In-Q-Tel and co-founder, securitymetrics.org
  • Andrew Jaquith, JP Morgan Chase and co-founder, securitymetrics.org
  • Ryan Leirvik, Grimm
  • Micah Henning, Perceptive
  • Ajoy Kumar, Depository Trust Clearing Corporation
  • Jay Jacobs, Cyentia
  • Mike MacIntyre, Panaseer
  • Michael McCobb, Verizon
  • Sergue Mokhov, Concordia University
  • Fernando Montenegro, 451 Group
  • Kevin Montalto, Protiviti
  • John Murray, Convertive
  • Yael Nagler, BlackRock
  • Kevin Neely, Pure Storage
  • Sean Owen, Abt Associates
  • Mukul Pareek, Wells Fargo
  • Dipanjan Paul, Bank of America
  • Greg Pfeiffer, Estee Lauder
  • Leila Powell, Panaseer
  • Alex Proskura, Auspicatus Consulting
  • Ari Rabinowitz, Decision Framework Systems
  • Anupam Rawla, IT Risk Automation Consulting
  • Paul Rohmeyer, Stevens Institute of Technology
  • Sanaz Sadoughi, International Monetary Fund
  • Fred Scholl, Quinnipiac University
  • Abhey Singh, Refinitive
  • Aman Singh, Palidrome Technologies
  • John Sturgis, University of South Carolina
  • Travis Sugarbaker, Cisco
  • Pete Taylor, Morgan Stanley
  • Russell Thomas, RMS
  • Dan Trotter, Merck
  • Chris Veltsos aka Dr Infosec, University of Minnesota
  • Suzanne Widup, Verizon
  • Walt Williams, Monotype

Session Descriptions #

Opening Remarks #

Presenters: Andrew Jaquith, JP Morgan Chase and co-founder, securitymetrics.org

Abstract: The theme of the conference is “plus ça change…,” or: “the more things change, the more they stay the same.” This talk is about both constants and the change in security metrics over the last 12 years:

  • Data-driven security took root
  • “AI” has come to security, with uneven results
  • Success disasters are great teachers
  • Controls instrumentation offers terrific bang for the buck
  • Audience is everything

Written remarks · Presentation

A Calibrated Severity Score for Breach Impacts #

Presenters: Suzanne Widup, Verizon Business and Russell Thomas, Zions Bancorporation

Abstract: We present a method for scoring the severity of information security breaches based on observable evidence (“Indicators of Impact”) associated with post-breach activity and consequences. Our data is 3,620 US breach episodes recorded in the VERIS Community Database Project. Each breach episode has been hand-coded with one or more publicly reported Indicators of Impact (36 categories), e.g. “Consent decree”, “Executive churn”, “Language in 8K or 10K [report to the SEC]”, and “Business relationship ended”. Ideally, we want to use these Indicators of Impact help us estimate a probabilistic cost model for each breach episode. As a steppingstone toward this goal, we have 1) developed an interval-scale severity scoring system; and 2) calibrated scoring system by the estimating the relative contribution of each Indicator of Impact as well as their functional interactions. The resulting severity scores should be useful to practitioners and policy makers for those decisions that can be made based on categorical distinctions – i.e. “bigger than a bread box”. We will also share lessons we have learned regarding how to make quantitative inferences from sparse, incomplete, and perhaps erroneous open source data.

Presentation

Defensible Metrics for Improved Network Resilience Scoring to Include Lateral Movement Detection and Susceptibility #

Presenter: Jason Crabtree, Fractal Industries

Abstract: Traditional efforts to define metrics to describe the resilience of networks against different attacks have been plagued by a lack of generality, challenges with data availability or quality, and narrow effectiveness around specific types of attacks or vulnerabilities. We review an extensible approach to compiling both real and synthetic data from multiple sources, propose a generalized scoring methodology using graph-based methods, show a complementary and common approach to attack path determination and planning, apply the technique to several representative test networks, demonstrate the scaling of the methodology to larger paradigmatic networks, and explore how specific detection/response capabilities can be used to reduce the overall state space which must be considered during event set generation. The talk includes a demonstration of detecting complex credential compromise attacks (e.g. Golden Ticket, Silver Ticket, DC Sync and DC Shadow) and uses the presence of such detections on the same reference networks to demonstrate the impact on network resilience scores due to the increased confidence in authentication which constrains post-exploitation attack paths considered in the overall scoring methodology.

Presentation

Metrics and Standards: Report From the Trenches #

Presenter: Walt Williams, Monotype

Abstract: This presentation will provide a critical review of the state of compliance frameworks and information security metrics, as well as a discussion on what success within each looks like and if it is worth the journey to get to that destination.

Presentation

Gamifying Vulnerability Risk Data to Encourage Coordinated Disclosure: The Making of the MSRC Top 100 #

Presenter: Christa Anderson, Microsoft

Abstract: One of the ways the Microsoft Security Response Center (MSRC) encourages people to report security vulnerabilities to Microsoft is through public recognition. As part of this effort, for some years we have published the MSRC Top 100 at Black Hat USA to highlight the researchers who have done the most to contribute to the security of our customers and the broader ecosystem. That’s been our intention, anyway. In this session we’ll talk about how we’ve measured that contribution, potential pitfalls in designing gamification based on data collected for another purpose, how the algorithm for the top 100 has evolved over the past few years, and how we’re continuing to iterate on this algorithm (and on how we publish the data) to encourage the most valuable research.

Presentation

Integrating Cyber Insurance Into Your Cyber Security Arsenal #

Presenter: Serguei Mokhov, Concordia University

Abstract: Regardless of how the cyber-interloper gets into your network, the next step taken by your IT staff can determine the severity and consequences of the intrusion. It is generally acknowledged that better security and training are needed since the number of cyber attackers continue to overtake cyber defenders, it is becoming more and more difficult to improve the situation because attackers are looking for one flaw in a system’s defenses while defenders need to find and fix them all. As IT practitioners we can take all the precautions necessary for a safe and secure environment and still fail to keep unwanted intruders out. In these instances a new trend of insurance has slowly developed. This paper looks at the role of cyber insurance and its place in security environments.

Presentation

Metrics that Matter: Help Management Improve Decision-Making and Improve the Organization’s Security Posture #

Presenter: Sanaz Sadoughi, International Monetary Fund

Abstract: Information Security Metrics present a holistic view of the information security posture of the organization. it is critical to analyze and aggregate “metrics that matter” to provide an overall security risk scorecard to the Management to help them with decision making. This presentation explains how metrics were implemented at the International Monetary Fund to drive action and demonstrate return on investment.

Presentation

Assigning Probability to Cybersecurity Risk #

Presenter: Jennifer Bayuk, Decision Framework Systems

Abstract: The session describes a cybersecurity decision support framework using risk management methodology developed in the professional practice of operational risk management. Operational risk (“ops risk”) is inherently low on quantitative measures in comparison with its more mature risk industry counterparts: credit risk and market risk. However, in the past few decades, professionals in the field have developed systematic data collection methods, control evaluation criteria, and risk analysis techniques that are directly applicable to cybersecurity decision support. Cybersecurity risk managers have gained immediate value from adopting these techniques. An ops risk framework allows cybersecurity risk to be analyzed in the context of both industry standards and organizational attributes. It provides precise definitions for information relevant to decisions and a methodology for using that information in the context of cybersecurity risk management. This session will provide an overview of how an ops risk framework helps organizations with cyber risk identification, classification, quantification, and monitoring.

Presentation

Why Does Application Security Take So Long? #

Presenters: Chris Eng, Veracode and Jay Jacobs, Cyentia Institute

Abstract: Why does it take so long to fix insecure code? We pair new data about the lifecycle of a vulnerability with learnings from application security programs to answer this perennial question. Our data comprises 700,000 individual assessments and a population of over 22 million unique security findings over a 12-month period, easily the largest application security data set of its size. Chris will discuss outcomes of this study with a particular focus on identifying the factors that correlate most strongly (or not at all!) with fix rates. He’ll also provide data-backed insights into the contentious question of whether DevOps is a boon or a burden for security. Jay will do a deep dive into the analysis process and some of the techniques, such as survival analysis, he applied to the data set in order to measure and visualize the outcomes we were interested in. We’ll also describe how we identified and handled anomalous customer data that would have otherwise produced skewed representations of developer behaviors.

Presentation

Communicating Cyber Risk to the Board of Directors #

Presenter: Wade Baker, Cyentia Institute

Abstract: For the last two years, I’ve been doing research into communicating cyber risk to the Board of Directors. Metrics are a major part of this. While this research has been published (https://go.focal-point.com/cyber-balance-sheet-report), I think summarizing findings for and hearing feedback from a room of experts would make for a strong session. I’ve also had the opportunity to implement this research in at least one major organizations and can share some lessons learned from that experience.

Presentation

Metrics and Standards: Can Data Science Help Understand Privileged Access? #

Presenter: Mike MacIntire, Panaseer

Abstract: Privileged access is increasingly understood as a challenging problem for organizations to solve. Even Board members understand what privileges “superusers” possess and the potential impact they can have on critical business systems — for good or ill. As one CISO put it, privileged access is “at the intersection of human behavior and technical controls, and often brings IT and security into conflict”. Tools for privileged access management (PAM) exist to manage privileged access, but installing a tool is just the beginning. Once you’ve identified how people should be accessing assets, how do you clean up the tangled web of permissions that exists in your organization, without hindering by business as usual? In this talk, we’ll reframe PAM as a data science problem and explore what insight you can glean from your data, about where the problem lies and how to fix it.

Presentation

Open Mic / Rump Session #

Abstract: From 11:30 through lunch, we will be providing an open mic for on the spot or improv presentations, questions for the community, rants (within reason) and other discussion topics.

Metrics for Organizational Cybersecurity Practices #

Presenter: Benjamin Charles Dean, Columbia University

Abstract: Over 2016–18 the OECD undertook a project to improve the measurement of the digital security risk management practices of businesses. This project has yielded a set of tools that represent major progress for policymakers, national statistical offices and insurers: a measurement framework, a set of core indicators and a pilot survey instrument. The measurement framework and indicators are based on the Principles contained in the 2015 Recommendation of the Council on Digital Security Risk Management for Economic and Social Prosperity. Throughout the two-year process input was sought from a joint working group comprising Delegates from the OECD Working Party on Security and Privacy in the Digital Economy (SPDE) and the OECD Working Party on Measurement and Analysis of the Digital Economy (MADE).

Presentation

Tactical Metrics Don’t Lead to Strategic Investments #

Presenters: Brian Gay, Northramp LLC and Sean Owen, Abt Associates

Abstract: Traditional cybersecurity metrics programs are overloaded with streams of data that focus on tactical decisions that don’t allow senior leadership to understand how to make smart risk-focused decisions. In addition, industry has developed tools to reflect this same desire and cater to a highly technical audience primarily focused on self-measurement. In this session, we are proposing a different approach which has been successful at Abt Associates that focuses on a metrics program for non-technical decision makers and risk owners using uncomplicated metrics that are focused on communicating risk and guiding investments.

Presentation

comments powered by Disqus