Re: Counting on CVEs

[security curmudgeon <jericho@attrition.org>]

: I have just had a very concerning discussion about the usefulness of 
: CVEs as a means to measure vulnerabilities today and the decay of its 
: value if the trend continues.  The discussion centered around the 
: accuracy of the numbers of CVEs identified compared to those reported in 
: the community as a whole.  If we looked at just the CVE numbers, it 
: appears that the numbers of vulnerabilities have been dropping since a 
: high in 2008.  This is a rather important error. As we all know, this is 
: not accurate. Vulnerabilities have not been dropping, they are growing, 
: not dropping by 30%.

I can't find it right off, but this came up several years back when 
several people noticed the drop in vulnerability totals around 2008. After 
additional examination of CVE, OSVDB, Secunia, and I believe BID, all four 
databases showed roughly the same drop. That in turn lead to speculation 
about *why* it was happening. I don't recall seeing anyone showing a 5 
year trending of vulnerability counts, as seen through multiple VDBs, but 
I would honestly request to see some rough numbers before pursuing this 
line of discussion further. 

: information between fielded products and between research analysis 
: teams.  As the numbers decline, it means we are forced to look elsewhere 
: to provide the identification and communication that CVE provided in the 
: past. More proprietary ids are becoming more the norm.

Is it cheaper for your team to manufacturer proprietary IDs instead of 
looking to Secunia, ISS, BID, and OSVDB for external references? If the 
answer is 'yes', why rely on CVE at all any more? If the answer is 'no', 
then perhaps these other VDBs are doing some of the work for you, where 
CVE is currently not.

: "If the vulnerabilities have dropped 30% since 2008, why do I need to be 
: funding the security staff and efforts at the rate I am?  I see that 
: MITRE is reporting an overall drop each year since 2008 but yet you 
: folks keep coming to me saying that the threats are much worse and we 
: may be in the same relative situation we were when malware spiked a 
: couple years past?"

That quote alone is a non sequitur really. I know that the conversations 
are more detailed with these executives, but even as a summary, it is too 
simplified.

Just because the number of unique vulnerabilities tracked by a VDB drops, 
does not mean the number of attackers drop. Further, it does not mean risk 
has dropped. Even further, more companies are using custom web 
applications, and no VDB tracks site-specific vulnerabilities (one current 
OSF project wants to, but we simply don't have funding or volunteers to 
make it happen). These are just a few issues of why an overall 
vulnerability count may drop, but their need to hire you and any other 
security company is stronger than ever. I certainly hope you can steer 
them past this limited view re: CVE.

: For those that actively work in this environment, we know 
: vulnerabilities have not dropped 30% since 2008. Quite the contrary, our 
: internal numbers indicate an increasing trend similar to a 30% rise.  
: Symantec has also reported a similar trend.

Could you provide links to the Symantec report? Again, I am just curious 
about the ~ 2008 thing where most VDBs reported a drop.

: Another problem is the CVE format itself.  There has been an active 
: discussion about the format limitations as well. The CVE format is 
: CVE-YYYY-NNNN. This means that currently we cannot have more than 10,000 
: CVEs reported in a single year.  At the rates we are seeing internally, 
: we are already there.

As I said in a previous reply, OSVDB hit the 10k mark in 2006 due to our 
abstraction method.

: Then there are the limitations of CVE process in general.  The focus is 
: English only although some non-US vulnerabilities do get assigned from 
: time to time.  CVE does not support the international community and 
: software written for non-English geo-centric areas are not included.  
: While this has not been a concern for US-only software vendors, there is 
: a great deal of regional software written for major emerging markets.  
: None of those vulnerabilities are identified by a CVE.

This is very difficult to manage. Even if CVE could keep up with all 
English-based disclosures from all sources (e.g., think of their team 
being triple in size at the very least), trying to keep up with foreign on 
top of it is another huge jump in resources. Carsten Eiram over at Secunia 
could probably give some insight on this better than anyone. I have 
noticed over the years that he appears to have several analysts that speak 
other languages, as they frequently pull up vulnerabilities with no 
English disclosure.

: Given these constraints, it seems like it is time to figure out how to 
: address the issues in a more of a creative way.  We know the 
: constraints. Is there something we can do to augment the MITRE staff in 
: certain areas that would help?  I can see the format issue being a 
: rather easy one to attack but it is the coverage issue that is most 
: concerning.  Or we should ignore it and slowly let the value of CVE to 
: the community and vendors decay?

If MITRE can't get funding to keep up (and catch up), looking to augment 
the effort from external staff would certainly be interesting. However, I 
would like to make two points on that.

1. Having non-MITRE resources work on CVE would require an entire overhaul 
of the process and policies around the project. Further, MITRE would have 
to divert some resources away from the already struggling CVE to 
implement some form of training program. I know this because Steve 
Christey and I 'traded roles' for *one* entry many years back. He gave me 
a link to a vulnerability and said "write a CVE". I gave him a link to a 
vulnerability and said "mangle the OSVDB entry to 100%". We both learned a 
lot from that process, and we both walked away with newfound respect for 
each other's process. Trying to train up a dozen people to meet their 
standards would not be easy.

2. OSVDB has been around for some time now, and I have been involved with 
it since 2004. We have been an 'open' database, where anyone could sign up 
for an account and contribute to our entries. We provided an extensive 
walk-through, near 24 hour support for a while should anyone have 
questions, reminder help screens for entries, templates to help ensure the 
text was readable, and more. Over the last 8 or so years, less than 1% of 
our data has come from volunteers. Off hand, I would guess that the bottom 
1,000 volunteers who signed up have contributed less than the #8 ranking 
contributor (http://osvdb.org/contributors). In short, the idea of an open 
community-driven database simply did not work. We tried a wide variety of 
campaigns to encourage people to help ranging from a reward system to 
internships to resume experience working for a 501(c)(3). In all of that 
time, less than 5 have stuck with the project and become core 
contributors. While we don't quite have the exposure of CVE, we are 
certainly not 'unknown'. We've had the framework to do this for a long 
time, and I have to think that if the community wanted it, they would have 
supported us. I know dozens of researchers and penetration testers that 
"don't have the time", but are overjoyed when I can create and mangle an 
entry for a vulnerability that no VDB has, so they can provide an external 
reference in their reports.

Brian
OSF / OSVDB