Metricon 3 — An Idea Whose Time Has Come
January 8, 2009
Agenda #
Metricon 3 was held Tuesday, 29 July 2008 at San Jose, California.
- Dan Geer – Welcome words and housekeeping details
Four grouped sessions to follow; each has three at-most-20 minute presentations of ideas followed by 30 minutes of reaction from discussants and general interaction with all Metricon attendees. Breaks are short as is life. Lunch, which is in-room, is long enough but no longer. Dinner, which is in-room, is as long as people want though there is nothing “to do” that is more important than making the very utmost of the day and thus keeping at it until late. Any and all electronic materials that presenters or attendees wish to provide will be available online at the meeting and a digest account of all that transpires will be made available to all (and eventually published). There is both a lot to cover and the time to do it.
Models proposed and derived
- Thomas Heyman and Christophe Huygens, DistriNet, K.U.Leuven, Belgium – Using Model Checkers to Elicit Security Metrics
- Adam O’Donnell, CloudMark – Games, Metrics, and Emergent Threats. See also Adam’s IEE S&P Article.
- Fred Cohen, Fred Cohen & Associates – Bringing Clarity to Security Decision Making Using Qualitative Metrics in 2 Dimensions
- Discussants: Lloyd Ellam amd Elizabeth Nichols
Tools and their application
- Yolanta Beresnevichiene, HP Labs UK – Metrics Driving Security Analytics
- Alain Mayer, RedSeal – Security Risk Metrics: The View From the Trenches
- Sandy Hawke, BigFix – How to Define and Implement Operationally Actionable Security Metrics
- Discussants: Gunnar Peterson & Andrew Jaquith
In-room lunch, the final 30 minutes jointly from…
- Jennifer Bayuk – Comparing Metrics Designed for Risk-Management with Metrics Designed for Security
- Discussant: Bryan Ware
Scoring results and methods
- James Walden, Northern Kentucky University – Code Complexity and Static Analysis
- Karen Scarfone, NIST – Evidence-Based, Good Enough, & Open. See also the handout.
- Arshad Noor – Identity Protection Factor. See also the handout.
- Discussants: Fred Cohen & Dan Conway
Enterprise plans and lessons learned
- Caroline Wong – eBay’s Metrics Program
- Clint Kreitner, Center for Internet Security and Elizabeth Nichols, MetricsCenter – CIS Security Metrics & Benchmarking Program. See also the Nichols Slides and CIS handout
- Kevin Peuhkurinen, Great-West Life Assurance – Great-West’s Metrics Program
- Discussant: Dan Geer
Perimeters are the simplest possible thing to measure, right?
- Sandeep Bhatt – Metrics-Based Firewall Management
- Avishai Wool, AlgoSec – Firewall Configuration Errors Revisited
- Discussant: Bob Blakley
- Additional materials: George Cybenko, Dartmouth – Quantitative Evaluation of Risk for Investment Efficient Strategies in Cybersecurity: The QuERIES Methodology
Dan Geer – Minimalist closing remarks
Drinks, dinner, further talk
Drinks & dinner in room, and whatever happens next – which it is hoped includes lessons learned, volunteers for further episodes of Metricon, ideas on how we can best further support ourselves jointly, etc. Perhaps we will have someone stand up and lead such a discussion; consider that part of the program still fluid.
Dan Conway compiled a full digest of proceedings. Thanks for compiling it.
Program Committee #
Chair: Dan Geer, Geer Risk Services
Members:
- Bob Blakley, The Burton Group
- Fred Cohen, Fred Cohen & Associates & California Sciences Institute
- Dan Conway, Indiana University
- Lloyd Ellam, Iceberg Networks
- Andrew Jaquith, The Yankee Group
- Elizabeth Nichols, PlexLogic
- Gunnar Peterson, Arctec Group
- Bryan Ware, Digital Sandbox
- Christine Whalley, Pfizer
Original Call for Participation #
Security metrics – an idea whose time has come. No matter whether you read the technical or the business press, there is a desire for converting security from a world of adjectives to a world of numbers. The question is, of course, how exactly to do that. The advantage of starting early is, as ever, harder problems but a clearer field though it is very nearly too late to start early. Metricon is where hard progress is made and harder problems brought forward.
The Metricon workshops offer lively, practical discussion in the area of security metrics. It is a, if not the, forum for quantifiable approaches and results to problems afflicting information security today, with a bias towards practical, specific implementations. Topics and presentations will be selected for their potential to stimulate discussion in the Workshop. Past events are detailed here and here; see, especially, the meeting Digests on those pages.
Metricon 3.0 will be a one-day event, Tuesday, July 29, 2008, in San Jose, California, USA. The Workshop begins first thing in the morning, meals are taken in the meeting room, and work/discussion extends into the evening. As this is a workshop, attendance is by invitation (and limited to 60 participants). Participants are expected to “come with findings,” to “come with problems,” or, better still, both. Participants should be willing to discuss what they have and need, i.e., to address the group in some fashion, formally or not. Preference will naturally be given to the authors of position papers/presentations who have actual work in progress.
Presenters will each have a short 10-15 minutes to present his or her idea, followed by a another 10-15 minutes of discussion. If you would like to propose a panel or a group of related presentations on different approaches to the same problem, then please do so. Also consistent with a Workshop format, the Program Committee will be steered by what sorts of proposals come in response to this Call.
Goals and Topics
Our goal is to stimulate discussion of, and thinking about, security metrics and to do so in ways that lead to realistic, early results of lasting value. Potential attendees are invited to submit position papers to be shared with all, with or without discussion on the day of the Workshop. Such position papers are expected to address security metrics in one of the following categories:
- Benchmarking of security technologies
- Empirical studies in specific subject matter areas
- Financial planning
- Long-term trend analysis and forecasts
- Metrics definitions that can be operationalized
- Security and risk modeling including calibrations
- Tools, technologies, tips, and tricks
- Visualization methods both for insight and lay audiences
- Data and analyses emerging from ongoing metrics efforts
- Other novel areas where security metrics may apply
Practical implementations, real world case studies, and detailed models will be preferred over broader models or general ideas.
How to Participate
Submit a short position paper or description of work done or ongoing. Your submission must be brief – no longer than five (5) paragraphs or presentation slides. Author names and affiliations should appear first in or on the submission. Submissions may be in PDF, PowerPoint, HTML, or plaintext email and must be submitted to metricon3 AT securitymetrics.org. These requests to participate are due no later than noon GMT, Monday, May 12, 2008 (a hard deadline).
The Program Committee will invite both attendees and presenters. Participants of either sort will be notified of acceptance quickly – by June 2, 2008. Presenters who want hardcopy materials to be distributed at the Workshop must provide originals of those materials to the Program Committee by July 21, 2008. All slides, position papers, and what-not will be made available to all participants at the Workshop.
No formal academic proceedings are intended, but a digest of the meeting will be prepared and distributed to participants and the general public. (Digests for previous Metricon meetings are on the past event pages mentioned above.) Plagiarism is dishonest, and the organizers of this Workshop will take appropriate action if dishonesty of this sort is found. Submission of recent, previously published work as well as simultaneous submissions to multiple venues is entirely acceptable, but only if you disclose this in your proposal.
Location
Metricon 3.0 will be co-located with the [17th USENIX Security Symposium|http://www.usenix.org/events/sec08/] at the Fairmont Hotel in San Jose, California.
Cost
$225 all-inclusive of meeting space, materials preparation, and meals for the day.
Important Dates
- Requests to participate: by May 12, 2008
- Notification of acceptance: by June 2, 2008
- Materials for distribution: by July 21, 2008