RE: Counting on CVEs

["Mann, Dave" <damann@mitre.org>]

Kent et al,

Summarizing my take on the situation...

0) People use vulnerability data for statistics at their own peril.

1) CVE cannot solve the global vulnerability reporting problem.  We can be a part of the solution, but not *THE* solution.

2) To think clearly about the global vulnerability reporting problem and CVE, we need to think about "sources" of vulnerability disclosures and who is going to process which sources.  The analogy here is: sources are to vulnerability reporting as jurisdictions are to law enforcement.

3) We, the CVE community, need to finish our discussion on which sources CVE will cover. And we will need to discuss how fast and how accurately those sources are covered.

4) Once we agree on which sources need to be covered, and how fast and how well, then we can talk about ways to close any gaps such as resources, process improvements, expanding the CNA process and crowd sourcing.


More verbose ramblings on these points follow...


1) GLOBAL VULNERABILITY REPORTING - In my opinion, one thing that CVE cannot do is solve the global vulnerability reporting problem.  This was my position in the global vulnerability reporting discussions last fall and my convictions in this regard have only solidified based on discussions with Carsten at Secunia, more detailed discussions with the folks at JP-CERT and others in the international community.  The sets of vendors/products in play are too different, the relationships between software vendors and various national governments are too different, and of course, the language barriers are too big.  The discussions of the past 2 years on this subject have led me to conclude that when the CVE community set our goal of "all publicly known vulnerabilities" 12 years ago, we did so naively and with an incorrectly parochial view of the global software market.  There may or may not be a good solution to the global vulnerability reporting problem.  But one thing I'm very sure of is that this solution, if it exists, will need to evolve organically by knitting together various regional capabilities.  

I think the best thing that we, the CVE community, can do to help facilitate the emergence of a global vulnerability reporting capability is to be able to speak clearly about what we can and can't do and to try to make as many of our lessons learned available to others as possible.  



2) VULNERABILITY SOURCES - We've talked internally at great length on the subject of vendors, products and sources.  We've also talked a bit about this as a Board.  In my opinion, we'll drive ourselves bonkers if we talk about vendors and products.

The goal of law enforcement isn't to catch bad guys.  It's to create and sustain a law enforcement system that can effectively catch bad guys.  This is a critical distinction.   The first results in cops with guns running around pursuing bad guys with no regard to coordination and jurisdictional boundaries.  The latter takes seriously the idea of jurisdictional boundaries and uses this to create command and control systems to operate effectively within those boundaries.

In my opinion, we need to think in these terms regarding vulnerability reporting.  The only somewhat stable structure I can see that does this for us to think in terms of "sources" of vulnerability information.  So, instead of thinking about vendor X or the list of products produced by vendor X (all of the internationalized variants), we can talk about the English-based security bulletin web site run by vendor X.  That web site can be on the list of sites tracked by CVE or not.  The set of sources tracked by CVE become, effectively, CVE's jurisdiction.   This is the discussion we, as a Board, started last fall.

I can hear the groans of complaint already.  Vulnerabilities don't stay on a single set of sources.  Absolutely true.  In the same way, criminals don't stay in a single jurisdiction.  But we can't organize a police force around a single type of criminal or a particular gang and I see no way to structure a CVE-like capability around a set of vendors or products.   If other CVE-like capabilities emerge that can handle other sets of sources (different jurisdictions), I suggest we'll have to deal with vulnerabilities that cross jurisdictional boundaries in the same way that law enforcement types handle it.

If people can suggest other better ways to define jurisdictions (or swim lanes) I'm all ears.


3) CVE COVERAGE - This past fall, we had discussions on the Board list about what sources you all felt were "must-haves" and those you considered "nice-to-haves".  We're processing this internally and are considering what sources we're actually covering, to what extent and how fast.   We hope to present a summary of that in the next little while and I further expect that the summary will highlight some important gaps between our expectations and the realities.  We'll have more to talk about at that point.

In preparation for that discussion, I'll quote the sign that hangs on Steve Christey's office door.  Vulnerability IDs - Pick 2: Good, Fast, Cheap.

We'll need to talk about each of these dimensions more.


4) EVOLVING THE CVE PROCESS - The CVE ID assignment process has evolved over the past 12 years and will continue to evolve.  Once we gain some clarity on which sources we need to cover, how well we need to cover them and at what we speed, we as a community can discuss what, if any, changes are required of the current CVE production process. I believe that everything can be put on the table for discussion at that point, but we really need to agree on the goals in terms coverage.


0) STATISTICS - Statistics require stable social categories and stable social categories require stable shared practices.  I'm not going to shock anybody by suggesting that security practices in general, and vulnerability practices in particular are not stable.  In short, our field is too young.   

The best book on the subject that I know is "Standards and Their Stories: How Quantifying, Classifying, and Formalizing Practices Shape Everyday Life" by Lampland and Star.  The discussion of how "calendar age" became a standard in everyday life and how the age classification processes of the US census bureau evolved is particularly germane to the question of vulnerability statistics.   

I have no idea how to communicate this sort of troubling truth to upper management types but strongly suggest the book to everyone on this list.  You can thank or blame TK when you see him next.  He's the one who suggested it to me.  ;) 


-Dave
==================================================================
David Mann | Principal Infosec Scientist | The MITRE Corporation
------------------------------------------------------------------
e-mail:damann@mitre.org | cell:781.424.6003
==================================================================