Re: Problematic assignments for subpar reports via CVE request form

[Carsten Eiram <che@riskbasedsecurity.com>]

On Mon, Oct 23, 2017 at 6:51 PM, Art Manion <amanion@cert.org> wrote:

On 2017-10-23 04:13, Carsten Eiram wrote:

...
> Maybe it would be worth having a discussion on the list about what
> this bar should be and how someone gets on the #ignore list? The CVE
> request form has made a lot of things easier, but it also seems to be
> the cause of a lot of problematic and invalid assignments.

Bug bounty/disclosure management platforms measure reputation and S/N ratio.

We track the same in our VulnDB solution. That's why the vulnerability reports from him stand out as being questionable. He is rapidly becoming the most untrustworthy vulnerability reporter. That and I had the "pleasure" of personally dealing with a big chunk of them, so I got to experience first hand how problematic they are.

While we aren’t suggesting that MITRE tracks vulnerability reporters, I would definitely expect it to be something they keep an eye on behind the scenes just like we do. At least there should be a process to address it when feedback on a vulnerability reporter is provided. Kurt details in his follow-up response, which I'll just address here, exactly the type of process I'd expect to go on behind the scenes of such CNAs.

> However, I do suggest some minimum level of
> vetting is performed. 

Not to put the burden entirely on the messenger, but can't anyone (or at least a CNA like RBS) DISPUTE a CVE entry?  This would be the crowd-based approach, assuming some vendor CNAs don't notice or care about inaccurate CVE entries.

Just for clarification's sake, we (RBS) are not a CNA, though that shouldn't stop us from disputing issues and hasn't. We have myself on the CVE board, though. However, being able to dispute CVE assignments is fine for a dispute here and there, but in this case we're speaking to a much bigger underlying problem of insufficient vetting before assigning CVE identifiers + assignments for issues with zero proof of a security impact.

First off, I'm not going to dispute 100+ CVEs to clean things up, because a CNA (MITRE) didn't do proper vetting before CVE assignments were made. I am, however, going to warn about the problem and provide guidance in my role on the CVE board. It is then up to MITRE to act on it as they see fit.

A significant concern is the effort required to refute questionable assignments.

One end of the spectrum is to assign liberally and wait until vendors DISPUTE.  Lots of cruft would build up though.  What about liberally accepting DISPUTE claims?  Any CNA or named vendor (reasonably trusted/known source) can change status to DISPUTED with a simple request.  Then, the alleged cruft is flagged, and burden goes on the Lin Wang class researcher.

Exactly, which leads me to: While CVEs can be disputed after the fact, it is much better to have a decent process to ensure at least obviously questionable requests aren't given CVEs in the first place. Assigning CVEs without any real vetting not only forces vendors and others to prove the CVE assignments are invalid and go through the process of getting them disputed or REJECTed, it also causes CVE consumers to scramble to address issues that either aren't as severe as claimed or don't exist at all. If this behavior continues or gets worse, it has the potential to call all of CVE data into question.

Back in the old days, the CVE team might have been slower and coverage may have been lesser, but at least CVE consumers could have a certain level of trust in the quality of the assignments. Lately we've noticed an increasing inconsistency and poor CVE assignment decisions. CVE has never claimed or aimed to be 100% correct (and shouldn’t), yet I know first hand that CVE consumers have an expectation that if a vulnerability has a CVE, it's legitimate. This is obviously a misconception, but to the extent possible and within reason, we should have processes in place to at least ensure a decent minimum level of CVE assignment quality.

 

Based on Carsten/RBS claims, I'd support a temporary ban.  But, we'll need to develop consistent rules/metrics (which could start as "a reliable CNA says you're wrong").

Also, keep in mind that the cruft and diffusion in quality and detail is a direct effect of choosing expansion.  Unavoidable, but still not something we should leave unchecked.

Agree.

To highlight the reason for me raising this as a serious issue, here are some metrics on Lin Wang. Our VulnDB solution currently has 244 entries credited to him. He reported more “vulnerabilities”, but the lower number is due to us having merged obvious duplicates. We are confident that if it was possible to test his claims, an additional significant chunk would end up being merged.

Out of these 244 entries, 110 are currently marked as either not covering a valid vulnerability e.g. stability bug or being false claims. That’s a failure rate of 45% and only covers the ones possible to flag as non-issues with reasonable certainty without testing (since no PoCs are available). There could easily be more, and if he keeps up the rate he is reporting, this will only get worse.

/Carsten