Metricon 8 — Seven Metrics Challenges

March 29, 2013

Metricon 8 was a one-day event, Friday, March 1, 2013, co-located with the RSA Security Conference, in San Francisco, WA. This page contains a description of the event, official proceedings, presentations, and the original CFP.

Program #

Coffee and introductions – Pete Lindstrom
Plenary workgroup scenarios and instructions – Pete Lindstrom
Breakout sessions
- Data Breach Costs – Ben Shapiro, facilitator
- Malware Identification – Patrick Florer, facilitator
- Vulnerability Management – Andrew Jaquith, facilitator
- Systems Development Controls – Evan Wheeler, facilitator
- Information Security Program – Matthew Fleming, facilitator
- Cyber Security Risk – Bob Rudis, facilitator
- Business Impact – Myles Conley, facilitator
CISO panel
- Jennifer Bayuk, Jennifer L Bayuk, LLC
- Fred Doolittle, Chevron
- Steve Dotson, Travelport
Lightning talks, and lunch
- Sal Stolfo and Nathaniel Boggs, Columbia University – Measuring Defense in Depth
- Anton Chuvakin, Gartner – Can We Have Top 5 Security Metrics, Pleeeeeeeease?
- Mahesh Saptarshi, Symantec – Continued funding of SDLC
- Plus a few other talks we do not have presentations for
Data publishers panel
- Wade Baker, Verizon Business
- Andrew Jaquith, SilverSky
- Steve Christey, MITRE
Breakout sessions, Round Two
Beer

Venue

Metricon 8 was held at Moscone South in San Francisco on March 1, 2013.

Summary

The goal of Metricon 8 was to bring together practitioners, review both the state of the art, and leverage the collective wisdom of participants to take the first step towards a framework for metrics in areas of significant value to enterprise security programs.

The day began with a discussion of goals and objectives led by the program chair, Pete Lindstrom. Facilitators led seven break-out sessions that produced an initial set of metrics. Each group:

Created scenarios associated with the topic areas
Defined a set of metrics to inform decisions regarding the scenarios
Reviewed published data to see what could be “borrowed”
Defined metrics, where each metric contained:
- Name: descriptive label
- Measure: unit of quantitative measurement(s)
- Scenarios: where the metric would be useful
- Frequency: how often to collect data used for measuring changes over time
- Formula: the calculation that results in a numeric expression of a metric
- Indicators: information about the meaning of the metric and its performance trend

Each group presented its preliminary results to a simulated “enterprise” panel of CISO-equivalent practitioners, with discussion subsequently opened to all participants. Groups used the feedback to identify gaps and refine their metrics concepts.

The enterprise panel was followed by short (5-10 min) “lightning talks” on emerging issues.

Following the lightning talks, a panel of metrics data publishers (Wade, Andrew and Steve) discussed their data-gathering and analysis processes, and how they expect enterprise security practitioners to make use of the data in their reports to make decisions.

In the last session of the day, each group completed its work and presented findings to everyone.

Metricon 8’s formal Proceedings, written by Jennifer Bayuk , describes the day’s proceedings in detail. It also includes the set of metrics that each group produced.

Program Committee #

Chair: Pete Lindstrom

Members:

Gunnar Peterson, Artec Group
Chris Porter, Verizon Business
Bob Rudis, Liberty Mutual
Walt Williams,
Andrew Jaquith, SilverSky
Dan Geer, In-Q-Tel
Jennifer Bayuk, Jennifer L. Bayuk, LLC

Attendees #

Jim Acquaviva, nCircle
Phil Agacoli, Cox Communications
Anthony Arrott, Trend Micro
Wade Baker, Verizon
Jennifer L. Bayuk, Jennifer L. Bayuk, LLC
Chris Berry, Sensage Services
Nathaniel Boggs, Columbia University
Stephen Boyer, BitSight Technologies
Katherine Brocklehurst , nCircle
Krag Brotby, Brotby & Associates
David Charing, Canadian Imperial Bank of Commerce
Steve Christey, MITRE
Anton Chuvakin, Gartner
Myles Conley, Auspices LLC
Earl Crane, National Security Staff, The White House
Keesha M. Crosby, Tri-Guard Risk Solution, LTD
Fred Doolittle, Chevron Information Technology Company
Steve Dotson, Travelport
Thomas Elegante, Zions Bancorporation
Jussi Eronen, CERT-FI
Matthew H. Fleming, Homeland Security Studies and Analysis Institute
Patrick M. Florer, Risk Centric Security, Inc.
Doug Foster, USG
Summer C. Fowler, Carnegie Mellon University
Gary Golomb, Cylance, Inc.
Grant Hansen, Zions Bancorporation
Paula Hant, salesforce.com
Lance Hayden, Cisco
Josh Huston, Exultium
Jay Jacobs, Verizon
Andrew Jaquith, Silversky
Jack Jones, CXOWARE, Inc.
Ramon Krikken, Gartner
Jason Leuenberger, Starbucks
Pete Lindstrom, Spire Security, LLC
Ivan Macalintal, Trend Micro
Michael Makstman, Kaiser Permanente
Robert Markel, Virgin America
Raffael Marty, pixlcloud
Adam Montville, Tripwire, Inc.
Bill Telletier, LMIG
Alex Proskura, Auspicatus
Andy Rappaport, CORE Security
Michael Roytman, Risk I/O
Bob Rudis, Liberty Mutual
Ben Sapiro, The Dominion
Mahesh Saptarshi, Symantec
Aaron Schaub, State Auto Insurance
David F. Severski, Seattle Children’s
Lindsey Smith, Tripwire, Inc.
Wyman Stocks, NetApp
Salvatore J. Stolfo, Columbia University
Morey Straus, VMware
Russell Thomas, George Mason University
Ryan Ward, Avatier Corporation
Evan Wheeler, Omgeo
Suzanne Widup, Verizon
Walt Williams, Lattice Engines
Mathew Woodyard, Zions Bancorporation
Kai Yu, Trend Micro

Sponsors #

Original Call for Participation #

In our continuing efforts to infuse the infosec field with useful metrics, we are excited to announce Metricon 8 - an all-day metrics workshop on Friday, March 1, 2013!

Metricon 8 has a workshop-like format - complete with objectives - that will ultimately lead to a lasting contribution to our field. We sincerely believe that leveraging the collective wisdom of participants will lead us to a taxonomy or framework for metrics in areas that are important to enterprise security programs. Even more specifically, our approach will incorporate an evaluation of existing industry data sources (e.g. published reports) with an eye toward identifying alignments, gaps, and overlaps as these reports relate to the needs of the enterprise security professional.

The format for the workshop will include breakouts and full-group sessions as we move towards our objectives. We will ultimately cover 3-6 of the following subject areas, keeping in mind our enterprise interests as they dovetail with published reports:

Incident-oriented Metrics (sources, e.g. DBIR, Trustwave, Mandiant)
Malware-oriented Metrics (sources, e.g. most/all anti-malware companies)
AppSec-oriented Metrics (sources, e.g. Veracode, Whitehat, OWASP)
Vulnerability and System Config Metrics (sources e.g. MSFT, IBM)
Risk and/or Compliance Metrics (sources e.g. Vz PCI Report, Index of Cybersecurity)
Process / Management / Resource Metrics (sources e.g. NIST/GAO, various)
Other enterprise-oriented Metrics (help us out!)

Note: We will be leaning heavily towards recurring data reports rather than ad hoc empirical studies.

All prospective participants with an interest in security metrics are welcome. However, given the maturity level of most of our members and our intended objectives, the content will be oriented towards “seasoned” professionals.

It is crucial that we get a good sense up-front about participants, given our move to Friday of RSA, and the change in conference format. Get your name on the list now! There is no cost to participate.

We invite you to submit your request to participate via email to Metricon 8 @ securitymetrics.org. Please include the following:

Your name and affiliation
Three topic areas that are of interest in prioritized order (from list above or add your own)
Anticipated level of participation (heavy, moderate, light, observer)
A brief statement about your interest in security metrics

We are also very interested in regrets or disinterest along with the reason why.

We hope you are as excited as we are in contributing to the growing body of practical knowledge in our field.