Re: Problematic assignments for subpar reports via CVE request form

[Carsten Eiram <che@riskbasedsecurity.com>]

On Thu, Oct 26, 2017 at 3:36 PM, Coffin, Chris <ccoffin@mitre.org> wrote:

Carsten,

 

In answer specifically to the CVE IDs assigned by the MITRE CNA for Lin Wang, the rationale for a CVE ID assignment often depends on non-public information. It could be, but isn't always, a non-public PoC. There have also been cases where we have requested that Lin provide justification for his vulnerabilities. In each case he has provided the justification required. Whether or not Lin makes this additional information public would be up to him. Maybe we can nudge Lin to provide more of this information (including PoCs) in the future once the vulnerability has been fixed by the vendor.

I am aware that MITRE sometimes receives additional details in private, which is good. However, I am also confident based on our analysis that this particular vulnerability reporter would not have been able to provide MITRE with any additional details or PoCs for many of these issues that would justify the CVE assignments. This is evident from the crash reports and extra analysis we conducted for VulnDB.

Discussing Lin Wang has served its purpose to bring attention to the bigger, underlying problem. I won’t spent much more time debating him specifically. We have evidence that the CVE assignments tied to many of his reports are invalid, while MITRE seems to overall consider them fine. There is little reason to discuss that any further.

Instead, I'd appreciate feedback on the following questions related to this:

1) Based on the above, it appears that MITRE already has some process for challenging vulnerability reporters. Could you please elaborate on the current process?

2) What made MITRE decide to challenge Lin Wang to provide justification in certain cases? Before I brought this issue up, there was nothing to suggest MITRE was skeptical of assigning CVEs to Lin Wang; even the very first response from MITRE made no mention of it. 

3) Which specific CVEs did MITRE challenge Lin Wang on? It is good to hear it was done, but we consider it important to understand for which CVEs additional vetting was performed by MITRE.

4) What specific details did Lin Wang provide that was considered justification for the CVE assignments? I presume PoCs from the wording. If so, was the fact that PoCs were provided all that was required or to which extent were they tested and crashes analyzed? Did you just confirm a crash and based on that considered the reports legit with more severe impacts plausible? Or was time spent determining that these could indeed have a more severe impact as speculated (randomly guessed) by the vulnerability reporter?

 

The CVE team has been able to pick out the following list of separate issues and questions in this thread. It might help if we separate and discuss them individually.

1. Should we ban requesters when they have repeatedly provided questionable vulnerability details when requesting a CVE ID?

1. MITRE will work with the Board to decide on a path forward here. One concern here is that banning a requester completely might send a negative message to the community in general and might be something we should avoid.
2. Do other Board members feel that banning requesters is ever appropriate, or should we just put more pressure on them to justify their work when these cases crop up?

My view is that the CNA should initially ask for more information and proof when dealing with reports that are considered questionable. The key is then being able to spot such reports. If the vulnerability reporter is unable to or refuses to provide more information, no CVE should be assigned for that specific issue. If the same vulnerability reporter in the future requests CVEs for additional questionable reports and still fails to provide sufficient information, the person would be put forward to discuss being banned.

In such cases, this process would ensure that the CNA has been forthcoming and tried to work with the vulnerability reporter, but to no avail. While some vulnerability reporters may complain, I’m sure the larger community and - more importantly CVE consumers - will appreciate it and view the process as fair.

If CVE would rather avoid ruffling a few feathers and assign CVEs to questionable or even obviously invalid reports, then the trustworthiness of CVE IDs is tarnished. We believe firmly that CVE should focus on the interests of CVE consumers; not vulnerability reporters, who have been identified as repeat offenders and just want as many CVEs as possible tied to their name without making an effort.

2. How should we handle researchers like Lin Wang in the future?

If “researchers like Lin Wang” refers to unreliable vulnerability reporters with questionable reports lacking sufficient information, I think some type of restriction is required. I’d also suggest that MITRE would do well to invest and help educate reporters that have quality issues to further their research and provide valuable work.

 

1. A better path forward might be to do as suggested and flag the requester for use in future requests. Maybe we could develop some simple process for CNAs around what additional information or details might be requested in these cases. If the requester is flagged, a CNA such as MITRE could treat the request slightly different and request additional information such as a PoC. This would likely cause extra work on the part of the CNA, but the assumption is that this case would not be the norm and wouldn’t happen often. In addition to the process, we’d have to define the parameters for when to flag a requester, and what would be required from them to get the flag removed.
2. Do other folks have suggestions for what might be required of the requester in these cases?

As long as the aspiration is that the validation is done properly, then this approach sounds good to me. I concur that it is something that should only impact very few vulnerability reporters. As such, the extra work should be minimal.

The goal of the CNA is not and should not be to do the research on behalf of the vulnerability reporter, but simply double check the validity of the report based on existing information. Should e.g. a PoC be provided and quick testing does not suggest an impact worse than a benign crash, it’s up to the vulnerability reporter to make a better effort. He or she has to prove that the issue is within scope instead of just guessing “or possibly have unspecified other impact” as Lin Wang does.

I think it’s important to also consider having a way to flag CVEs that were assigned based on additional scrutiny by the CNA. That way other CNAs and CVE consumers know that additional vetting was performed. Perhaps even include a comment about what this vetting was e.g. “PoC provided privately and crash demonstrating a buffer overflow verified.” This would signal that even though the report appears to be questionable, further analysis has been conducted and the CVE ID can be trusted.

 

3. MITRE's inclusion criteria for what is a vulnerability.

1. MITRE currently uses CNT2.2A as part of the criterion to decide if something is a vulnerability.  When the rule was proposed, there was discussion around issues such as this where a researcher was claiming a vulnerability that might be received as questionable by others. The consensus was that these assignments would be acceptable so long as remediation steps were available to dispute and reject.  If the Board no longer feels CNT2.2A is a valid criteria for deciding if something is a vulnerability, then a discussion may be needed in regards to using only CNT2.2B.

I think this approach is a disservice to CVE consumers. It’s good you suggest us revisiting that discussion. This causes organizations to waste time on issues that are either less severe or outright invalid. Such assignments should be kept to a minimum within a reasonable amount of effort.

As mentioned previously, I don’t think it’s the goal of CVE to aim for 100% accuracy, but some vetting should be performed to weed out questionable and obviously invalid requests. It’s problematic that CVEs are assigned to such requests with the (honestly “naive”) hope that someone else is going to dispute/reject it if wrong. This becomes more problematic when there is not even a requirement for at least public PoCs in such cases to make crowd-sourced disputes more likely.

That said, even if there were public PoCs, we know that crowd-sourced disputes still won’t happen to the necessary degree.

 

4. The community does not seem incentivized to dispute CVE assignments

1. As mentioned above, the use of CNT2.2A seemed acceptable so long as the dispute/reject processes worked.  However, as Carsten demonstrates, those who do the research that could be used to dispute or reject a CVE aren't incentivized to provide that information to CVE. Does this change how Board members feel about CNT2.2A?

As commented in a previous response, this only works (and even then not really) when it’s a CVE here and there; not when it’s a significant chunk of CVEs. The new approach with the CVE request form is generating too many of these. We shouldn’t expect a crowd-sourced approach to dispute/reject CVEs. There is no value whatsoever in it for people to spend time on it. Vulnerability reporters are focused on how they can request the most CVEs for their own findings; including making money off bug bounties; not “waste time” disputing the findings of others.

The extremely few CVE consumers, who do spend resources to validate CVE assignments, are not going to go through the whole dispute process and provide sufficient evidence. In fact, it currently seems like much more of an effort to dispute a CVE than request one. That’s worth pondering.

 

5. Preventing duplicates

1. The comments about duplicates largely relate to some deficits that existed in CNT1 (Independently fixable). In particular, CNT1 did not provide guidance on how to handle situations where there is some evidence that two issues are the same vulnerability, but you are not certain. MITRE's policy in these cases was to follow the groupings the requester used when making the request. However, in the most recent rules revision, we proposed a change to CNT1 in which the issues would be merged into a single CVE ID if there is uncertainty on whether they are duplicates.  The change was approved and MITRE will be following it going forward.  The new rules should stop these types of possibly duplicate assignments from being made in the future. This issue partially explains why CVE-2017-15773 and CVE-2017-15738 are separate.

This is a good change. When in doubt, a merge is better than a split. We can always split later if more details become available (rarely happens) to justify it.

 

6. How much certainty is required before making a decision on assigning a CVE ID?

1. Taking a step back, much of this discussion circles around the degree of certainty needed to assign a CVE ID. How is this certainty obtained and what is the cost of obtaining the certainty? Is it worth the results?
2. Carsten: What process did you use to determine that CVE-2017-15738 and CVE-2017-15773 were duplicates? How long did it take?

I simply looked at the crash output provided with one of the reports and compared it to others with similar characteristics. It was obvious pretty much immediately when processing one of them that it was likely a duplicate of the other. After that I double checked my suspicions by disassembling the affected library from the two products. Standard practice for us really; no fancy or elaborate process needed.

It’s hard to say how long it took, as it was part of a larger process to invalidate a large portion of that chunk of assigned CVEs. This is something we do regularly for VulnDB, so we’re talking only a few minutes, though. If I had just needed to invalidate that one issue, it would have taken longer to download and install the two products than spot the dupes and open the libraries up to confirm. In some cases, it could take longer, but anyone familiar with validating vulnerability reports should be able to perform this type of vetting within very short time.

/Carsten