Re: procedure for penalizing or revoking CNA status?

[jericho <jericho@attrition.org>]

On Fri, 10 Oct 2014, Steven M. Christey wrote:

: On Thu, 25 Sep 2014, jericho wrote:
: 
: > For the rest of the board, there has been an increasing reason to better
: > monitor and restict CVE assignment. Both from researchers requesting them,
: > and from CNAs who don't understand CVE or the abstraction process.
: 
: The former issue - about researchers requesting CVEs - is separable from how
: CNAs manage CVEs.  

Absolutely. We can address one more readily than the other, which is the 
point of my mail.

: Some context for CNA-related errors: traditionally, we've had 
: approximately a 0.5% REJECT rate for CVEs overall, but that percentage 
: has gone up in recent years, although I don't track these stats 
: regularly or precisely (yet).  While I personally dislike REJECTs, the 
: 0.5% rate doesn't indicate a systemic problem.  But since the raw number 
: of CVE assignments has also risen along with the rate, the raw number of 
: REJECTs has increased noticeably.  REJECTs, for us and I believe for 
: many CVE consumers, can cause confusion and be time-consuming to 
: resolve.

Screw the numbers. You cannot look at the OVERALL reject rate and somehow 
magically apply that to the few CNAs who may be the cause of most of them. 
Further, a CNA can not follow guidelines and do a variety of assignments 
that are wrong, and CVE may not notice. If you go off pure 'reject' 
numbers, they are all over the place.

1999 - 33
2013 - 102
2014 - 35 (so far)

You and I both know the bias of statistics. I can selectively pick out 
stats to prove my point, just like you. My point is that every few weeks 
we run into a CNA that has botched an assignment. Personally, I don't feel 
that the entire CVE team looks at this angle, while I know that you 
(Steve) would. You simply cannot analyze all CVE just as I cannot analyze 
all OSVDB unfortunately.

: We do not have any formal procedures for warning, penalizing, and/or 
: revoking CNA status, but we agree that we should develop some.  One 

Good. I think we're about 2 years overdue on this at the very least, and I 
think CVE should make up for lost time. At the very least, if a CNA issues 
a CNA in violation of the standards, they should be expected to answer for 
it. For example, Oracle issues a CVE on a CVSSv2 0.0 issue a while back. I 
choose that because it is the most amusing example, and it involved the 
GOPHER protocol which is absolutely epic. Said issue wasn't a 0.0, but it 
was very low and warranted a CVE. The fact that they said "not a vuln, no 
risk, no issue" by assigning a 0.0 score and then assigning a CVE to it is 
problematic to me. Same organization has also botched numerous other 
assignments (they do almost every release).

: issue is that things have gotten much more complex, and what might 
: appear to be a CNA error could in fact be due to limitations of the CNA 
: process, many of which were discussed in the early days of CVE, if I 

Again, problematic. 'Early days of CVE' is honestly a dark period. Until 
2013, CVE still had the same 'data sources' page listing the sources of 
vuln information they tracked, the same as it listed in 2007. Likely 
before but that is as far back as archive.org goes. We simply cannot go 
off 'early days' any more. That is purely the BID model of "we're stuck in 
1999 and doing fine". That shit VDB doesn't even have 100% coverage of 
CVE, which has become the no-child-left-behind model of VDBs. I'd really 
like to see CVE not become as bad as BID.

: recall correctly. When developing procedures, we also need to ensure 
: that any disciplinary measures - when necessary - are not out of balance 
: with the offense.

Agreed. And I know we will butt heads on this. For example, if we punished 
any organization that didn't know how to use CVSSv2 and said "you can't 
score any more", then most major vendors would never use it again 
including IBM, HP, ICS-CERT, Oracle, and more.

: However, we also need to be clear on what is causing the errors.  The 
: errors that occur are rarely due to carelesness.  For example, we've 

Maybe. But if it isn't carelessness, it is absolute ignorance. That is 
worse. Screw them if they didn't read the fine print. If there wasn't fine 
print, screw CVE for not giving it to them.

Speaking of, where are the CNA standards exactly? I don't recall EVER 
reading what is sent to a new CNA ... and that bothers me too. That should 
not only be public, but trivial to find on the CVE site. Which as 
mentioned, can go a DECADE without being updated.

: learned that over time, people's jobs (naturally) shift; and the 
: original technical lead for a CNA might move to a different role, and 

Don't care, no excuse. CVE is a 'cornerstone' of the industry. CVE puts 
itself out there as "a dictionary of publicly known information security 
vulnerabilities and exposures" with the intent to "allow vulnerability 
databases and other capabilities to be linked together, and to facilitate 
the comparison of security tools and services".

Almost every security offering, be it vuln scanner, IPS, IDS, firewall, or 
anything else is based on CVE. We have a moral and ethical obligation to 
hold our work to the highest standard. If anyone disagrees, we are happy 
to accept your immediate resignation. That goes for CVE staff or any 
Editorial board member. (by that I mean, more than half the board should 
immediately step down).

: the replacement is not as well-trained.  As another example, there are 

Replacement? MITRE gets as much as 5 million a year to run CVE. We know 
that isn't reality, but as I have shown you, that is the best figure I can 
give you due to MITRE's shitty book-keeping. Even if it is 1 million a 
year, which is about the lowest it could be... that is 10x more than other 
VDBs use to cover *more vulns* yearly. Point is, if a CNA isn't cutting 
it, MITRE should be dealing with this problem. It shouldn't take an 
Editorial Board member to notice and bring this up.

If MITRE can't do it, why not? Speaking of, did CVE ask for additional 
funding this year related to medical vulnerabilities? Why is 1 - 5 million 
not enough to trach vulns, when other VDBs have medical vuln coverage 
going back to the 80's and have made such vulns a priority? Given the 
importance of CVE, there has to be more transparency here.

: researchers who contact multiple CNAs for CVEs and effectively introduce 
: duplicates that way (not maliciously, as far as I can tell); many 

We're skipping the researcher problem, as you said, entirely different 
beast. Not sure any of us can come up with a solution to that stupidity 
(and it is getting worse). I did mail you off-list about setting standards 
for when a researcher asks for a CVE ID, to help ensure it meets criteria. 
Curious if that has been implemented yet?

: our fairly extensive documentation.  As a third example, the significant 
: media attention and urgency given to some issues, along with 
: non-coordinated disclosure, introduces room for error.  Incomplete 

As seen with 'Shellshock'! That has as many as 5 CVEs attached to it 
depending on abstraction. By that I mean if you abstract to the same issue 
not properly fixed less than 24 hours later, it only has 3 dupes and 2 
assignments.

That is an entirely different conversation the board should be having. 
Yet... no one else on the board has brought it up? Why is that exactly! I 
know one other board member noticed it and we have had extensive 
discussion. How about the rest of you? 

: and was a factor in the confusion - for which CVE was a symptom and not 
: a cause (and, partially, a cure because we did REJECTs pretty quickly.)  
: There are many more situations besides these.

True, and while CVE was quick to get HB and SS out, and deal with the 
dupes, I would bet a dollar it was due to YOU personally recognizing the 
severity and handling it. That is called a single point of failure. Every 
VDB has one =) With OSVDB, we finally addressed that to some degree after 
10 years by recruiting a 2nd person that is just as knowledgeable about 
VDBs. While he doesn't have the history I do on OSVDB, he has the history 
on VDBs to make the same decisions I did. In addition, I have gone to 
great lengths to document all of my madness for that very reason. In our 
world, we call it the "what if jericho gets hit by a bus" scenario.

What does CVE call it? Has CVE worked to address it? If not, why not? You 
are a fucking cornerstone of the industry. Every mouth-breathing idiot 
treats CVE like the vuln-bible. The ethical and moral obligation to deal 
with this is considerably higher than what OSVDB faces.

: In the coming months, we will improve our tracking for REJECTs and why 
: they happen; consult more closely with CNAs; and consult with the Board 
: on ways forward.

Uh, do you need a year-by-year breakdown of REJECTs? I have it. A 
break-down of the year-by-year RESERVED? Need a percentage of public vulns 
covered by CVE as compared to other VDBs? Got that too. Again, it isn't 
about raw numbers, they don't paint the real picture.

Last, and I know this is a teaser to the rest of the board who barely 
reads this mail, and certainly doesn't give a shit... and this is entirely 
a setup, since I have a thing about being honest:

Anything else you want to "bring up to the board" before I ask the same 
question of the board, with details of what you told people you would 
bring to the board, and haven't yet? =) I have been waiting for it for 
over a week now because it is a fascinating discussion, and has serious 
ramifications for CVE. Yet nothing so far... Maybe you are the real tease 
here.

.b