Re: Problematic assignments for subpar reports via CVE request form

[Pascal Meunier <pmeunier@cerias.purdue.edu>]

This sort of event saps at a foundation of the CVE.  In a prior Board
call, we discussed the issue of references that disappear over time due
to aging and abandonment.  Both are use cases that would justify trying
to archive CVE references somehow.  

Pascal


On Mon, 2017-11-13 at 00:12 +0100, Carsten Eiram wrote:
> As a quick follow-up to this discussion, I noticed today that Lin
> Wang
> seemingly deleted all his reports on 2017/10/31:
> https://github.com/wlinzi/security_advisories/commit/d571e1c1e2829483
> 3a96a070b494a7eeeb6fd516
> 
> Unless I overlooked something they're all blank e.g:
> https://github.com/wlinzi/security_advisories/tree/master/CVE-2017-14
> 302
> 
> Naturally, we cannot guarantee or expect for vulnerability reports to
> exist
> indefinitely, but it is curious when deleted shortly after
> publication from
> the sole source they were available at. Unless this was some mistake,
> looking at his Github repository it seems like he for unknown reasons
> decided to clean out the whole repository to prevent reading the old
> reports and just put up blank dummy entries instead. Perhaps he
> suddenly
> questioned the quality / validity of his reports.
> 
> I think this adds to why assigning CVEs for any of his reports going
> forward should be a concern. Many of them were questionable to begin
> with
> and have now also been deleted, which is a problem since they were
> the only
> sources referenced in the assigned CVEs. We now have 330 CVEs from
> 2017
> with dead references from a single unreliable vulnerability
> discoverer.
> Unless he plans to republish the information, this makes 3rd party
> vetting
> even harder. If CVEs are assigned to any of his future reports, we
> should
> at least consider capturing public copies of these elsewhere.
> 
> /Carsten
> 
> 
> On Fri, Nov 3, 2017 at 5:06 PM, Coffin, Chris <ccoffin@mitre.org>
> wrote:
> 
> > Carsten,
> > 
> > 
> > 
> > Below are some answers and thoughts on your numbered questions.
> > 
> > 
> > 
> >    - 1) Based on the above, it appears that MITRE already has some
> >    process for challenging vulnerability reporters. Could you
> > please elaborate
> >    on the current process?
> > 
> > The CVE analysts making CVE ID assignments on behalf of the MITRE
> > CNA have
> > many years of experience in vulnerability analysis and use this
> > coupled
> > with the current CVE counting rules to determine whether a CVE ID
> > should be
> > assigned. The CVE analysts in this role are able to detect hundreds
> > of
> > different scenarios in which a submission does not "provide a
> > demonstrated
> > negative impact for the bug" (CNT2.2A). A simple step-by-step
> > checklist
> > does not exist for this process.
> > 
> > 
> > 
> >    - 2) What made MITRE decide to challenge Lin Wang to provide
> >    justification in certain cases? Before I brought this issue up,
> > there was
> >    nothing to suggest MITRE was skeptical of assigning CVEs to Lin
> > Wang; even
> >    the very first response from MITRE made no mention of it.
> > 
> > The analyst wanted to know whether a crash was a negative impact
> > for the
> > use cases of a product, and also wanted to know whether specific
> > file types
> > could be presented to a product in a way that crossed a privilege
> > boundary.
> > 
> > 
> > 
> > For example, we would not assign a CVE ID for a "benign crash" in a
> > command-line program that is intended to read an image file, do
> > something
> > with it, and then stop execution. If a product is not limited to
> > that type
> > of command-line usage, and is instead a long-running desktop
> > application,
> > then it is possible that all crashes (for any reasons) have either
> > a
> > negative availability impact or a negative confidentiality impact
> > (or
> > both). It is also possible that a different long-running desktop
> > application, if it has sufficiently limited use cases, would never
> > be
> > characterized as vulnerable on the basis of a "benign crash."
> > 
> > 
> > 
> > Lin Wang also sent us the reports about DLL files that became
> > CVE-2017-15790 through CVE-2017-15803. We initially challenged all
> > of them
> > because of insufficient information about the interaction between
> > the base
> > product and the DLL file. For example, it is not a vulnerability if
> > the
> > threat model is an attacker who already has the ability to choose
> > what code
> > exists in a DLL file, and then cause the product to execute code
> > from that
> > DLL file. Lin Wang told us that the threat model for DLL files was
> > the same
> > as the threat model for any type of image file, because the product
> > is only
> > trying to render an icon picture contained within the DLL file.
> > 
> > 
> > 
> >    - 3) Which specific CVEs did MITRE challenge Lin Wang on? It is
> > good
> >    to hear it was done, but we consider it important to understand
> > for which
> >    CVEs additional vetting was performed by MITRE.
> > 
> > See the prior answer.
> > 
> > 
> > 
> >    - 4) What specific details did Lin Wang provide that was
> > considered
> >    justification for the CVE assignments? I presume PoCs from the
> > wording. If
> >    so, was the fact that PoCs were provided all that was required
> > or to which
> >    extent were they tested and crashes analyzed? Did you just
> > confirm a crash
> >    and based on that considered the reports legit with more severe
> > impacts
> >    plausible? Or was time spent determining that these could indeed
> > have a
> >    more severe impact as speculated (randomly guessed) by the
> > vulnerability
> >    reporter?
> > 
> > A PoC was provided privately in each and every case. It was
> > confirmed in
> > each case that the CVE was not identical to the PoC for any other
> > CVE.
> > 
> > 
> > 
> > 
> > 
> > Chris
> > 
> > 
> > 
> > *From:* Carsten Eiram [mailto:che@riskbasedsecurity.com]
> > *Sent:* Tuesday, October 31, 2017 12:40 AM
> > *To:* Coffin, Chris <ccoffin@mitre.org>
> > *Cc:* Art Manion <amanion@cert.org>; cve-editorial-board-list <
> > cve-editorial-board-list@lists.mitre.org>
> > *Subject:* Re: Problematic assignments for subpar reports via CVE
> > request
> > form
> > 
> > 
> > 
> > On Thu, Oct 26, 2017 at 3:36 PM, Coffin, Chris <ccoffin@mitre.org>
> > wrote:
> > 
> > Carsten,
> > 
> > 
> > 
> > In answer specifically to the CVE IDs assigned by the MITRE CNA for
> > Lin
> > Wang, the rationale for a CVE ID assignment often depends on non-
> > public
> > information. It could be, but isn't always, a non-public PoC. There
> > have
> > also been cases where we have requested that Lin provide
> > justification for
> > his vulnerabilities. In each case he has provided the justification
> > required. Whether or not Lin makes this additional information
> > public would
> > be up to him. Maybe we can nudge Lin to provide more of this
> > information
> > (including PoCs) in the future once the vulnerability has been
> > fixed by the
> > vendor.
> > 
> > 
> > 
> > 
> > 
> > I am aware that MITRE sometimes receives additional details in
> > private,
> > which is good. However, I am also confident based on our analysis
> > that this
> > particular vulnerability reporter would not have been able to
> > provide MITRE
> > with any additional details or PoCs for many of these issues that
> > would
> > justify the CVE assignments. This is evident from the crash reports
> > and
> > extra analysis we conducted for VulnDB.
> > 
> > 
> > 
> > Discussing Lin Wang has served its purpose to bring attention to
> > the
> > bigger, underlying problem. I won’t spent much more time debating
> > him
> > specifically. We have evidence that the CVE assignments tied to
> > many of his
> > reports are invalid, while MITRE seems to overall consider them
> > fine. There
> > is little reason to discuss that any further.
> > 
> > 
> > 
> > Instead, I'd appreciate feedback on the following questions related
> > to
> > this:
> > 
> > 
> > 
> > 1) Based on the above, it appears that MITRE already has some
> > process for
> > challenging vulnerability reporters. Could you please elaborate on
> > the
> > current process?
> > 
> > 
> > 
> > 2) What made MITRE decide to challenge Lin Wang to provide
> > justification
> > in certain cases? Before I brought this issue up, there was nothing
> > to
> > suggest MITRE was skeptical of assigning CVEs to Lin Wang; even the
> > very
> > first response from MITRE made no mention of it.
> > 
> > 
> > 
> > 3) Which specific CVEs did MITRE challenge Lin Wang on? It is good
> > to hear
> > it was done, but we consider it important to understand for which
> > CVEs
> > additional vetting was performed by MITRE.
> > 
> > 
> > 
> > 4) What specific details did Lin Wang provide that was considered
> > justification for the CVE assignments? I presume PoCs from the
> > wording. If
> > so, was the fact that PoCs were provided all that was required or
> > to which
> > extent were they tested and crashes analyzed? Did you just confirm
> > a crash
> > and based on that considered the reports legit with more severe
> > impacts
> > plausible? Or was time spent determining that these could indeed
> > have a
> > more severe impact as speculated (randomly guessed) by the
> > vulnerability
> > reporter?
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > The CVE team has been able to pick out the following list of
> > separate
> > issues and questions in this thread. It might help if we separate
> > and
> > discuss them individually.
> > 
> > 1. Should we ban requesters when they have repeatedly provided
> > questionable vulnerability details when requesting a CVE ID?
> > 
> >    1. MITRE will work with the Board to decide on a path forward
> > here.
> >    One concern here is that banning a requester completely might
> > send a
> >    negative message to the community in general and might be
> > something we
> >    should avoid.
> >    2. Do other Board members feel that banning requesters is ever
> >    appropriate, or should we just put more pressure on them to
> > justify their
> >    work when these cases crop up?
> > 
> > 
> > 
> > My view is that the CNA should initially ask for more information
> > and
> > proof when dealing with reports that are considered questionable.
> > The key
> > is then being able to spot such reports. If the vulnerability
> > reporter is
> > unable to or refuses to provide more information, no CVE should be
> > assigned
> > for that specific issue. If the same vulnerability reporter in the
> > future
> > requests CVEs for additional questionable reports and still fails
> > to
> > provide sufficient information, the person would be put forward to
> > discuss
> > being banned.
> > 
> > 
> > 
> > In such cases, this process would ensure that the CNA has been
> > forthcoming
> > and tried to work with the vulnerability reporter, but to no avail.
> > While
> > some vulnerability reporters may complain, I’m sure the larger
> > community
> > and - more importantly CVE consumers - will appreciate it and view
> > the
> > process as fair.
> > 
> > 
> > 
> > If CVE would rather avoid ruffling a few feathers and assign CVEs
> > to
> > questionable or even obviously invalid reports, then the
> > trustworthiness of
> > CVE IDs is tarnished. We believe firmly that CVE should focus on
> > the
> > interests of CVE consumers; not vulnerability reporters, who have
> > been
> > identified as repeat offenders and just want as many CVEs as
> > possible tied
> > to their name without making an effort.
> > 
> > 
> > 
> > 
> > 
> > 2. How should we handle researchers like Lin Wang in the future?
> > 
> > 
> > 
> > If “researchers like Lin Wang” refers to unreliable vulnerability
> > reporters with questionable reports lacking sufficient information,
> > I think
> > some type of restriction is required. I’d also suggest that MITRE
> > would do
> > well to invest and help educate reporters that have quality issues
> > to
> > further their research and provide valuable work.
> > 
> > 
> > 
> > 
> > 
> > 
> >    1. A better path forward might be to do as suggested and flag
> > the
> >    requester for use in future requests. Maybe we could develop
> > some simple
> >    process for CNAs around what additional information or details
> > might be
> >    requested in these cases. If the requester is flagged, a CNA
> > such as MITRE
> >    could treat the request slightly different and request
> > additional
> >    information such as a PoC. This would likely cause extra work on
> > the part
> >    of the CNA, but the assumption is that this case would not be
> > the norm and
> >    wouldn’t happen often. In addition to the process, we’d have to
> > define the
> >    parameters for when to flag a requester, and what would be
> > required from
> >    them to get the flag removed.
> >    2. Do other folks have suggestions for what might be required of
> > the
> >    requester in these cases?
> > 
> > 
> > 
> > As long as the aspiration is that the validation is done properly,
> > then
> > this approach sounds good to me. I concur that it is something that
> > should
> > only impact very few vulnerability reporters. As such, the extra
> > work
> > should be minimal.
> > 
> > 
> > 
> > The goal of the CNA is not and should not be to do the research on
> > behalf
> > of the vulnerability reporter, but simply double check the validity
> > of the
> > report based on existing information. Should e.g. a PoC be provided
> > and
> > quick testing does not suggest an impact worse than a benign crash,
> > it’s up
> > to the vulnerability reporter to make a better effort. He or she
> > has to
> > prove that the issue is within scope instead of just guessing “or
> > possibly
> > have unspecified other impact” as Lin Wang does.
> > 
> > 
> > 
> > I think it’s important to also consider having a way to flag CVEs
> > that
> > were assigned based on additional scrutiny by the CNA. That way
> > other CNAs
> > and CVE consumers know that additional vetting was performed.
> > Perhaps even
> > include a comment about what this vetting was e.g. “PoC provided
> > privately
> > and crash demonstrating a buffer overflow verified.” This would
> > signal that
> > even though the report appears to be questionable, further analysis
> > has
> > been conducted and the CVE ID can be trusted.
> > 
> > 
> > 
> > 
> > 
> > 3. MITRE's inclusion criteria for what is a vulnerability.
> > 
> >    1. MITRE currently uses CNT2.2A as part of the criterion to
> > decide if
> >    something is a vulnerability.  When the rule was proposed, there
> > was
> >    discussion around issues such as this where a researcher was
> > claiming a
> >    vulnerability that might be received as questionable by others.
> > The
> >    consensus was that these assignments would be acceptable so long
> > as
> >    remediation steps were available to dispute and reject.  If the
> > Board no
> >    longer feels CNT2.2A is a valid criteria for deciding if
> > something is a
> >    vulnerability, then a discussion may be needed in regards to
> > using only
> >    CNT2.2B.
> > 
> > 
> > 
> > I think this approach is a disservice to CVE consumers. It’s good
> > you
> > suggest us revisiting that discussion. This causes organizations to
> > waste
> > time on issues that are either less severe or outright invalid.
> > Such
> > assignments should be kept to a minimum within a reasonable amount
> > of
> > effort.
> > 
> > 
> > 
> > As mentioned previously, I don’t think it’s the goal of CVE to aim
> > for
> > 100% accuracy, but some vetting should be performed to weed out
> > questionable and obviously invalid requests. It’s problematic that
> > CVEs are
> > assigned to such requests with the (honestly “naive”) hope that
> > someone
> > else is going to dispute/reject it if wrong. This becomes more
> > problematic
> > when there is not even a requirement for at least public PoCs in
> > such cases
> > to make crowd-sourced disputes more likely.
> > 
> > 
> > 
> > That said, even if there were public PoCs, we know that crowd-
> > sourced
> > disputes still won’t happen to the necessary degree.
> > 
> > 
> > 
> > 
> > 
> > 4. The community does not seem incentivized to dispute CVE
> > assignments
> > 
> >    1. As mentioned above, the use of CNT2.2A seemed acceptable so
> > long as
> >    the dispute/reject processes worked.  However, as Carsten
> > demonstrates,
> >    those who do the research that could be used to dispute or
> > reject a CVE
> >    aren't incentivized to provide that information to CVE. Does
> > this change
> >    how Board members feel about CNT2.2A?
> > 
> > 
> > 
> > As commented in a previous response, this only works (and even then
> > not
> > really) when it’s a CVE here and there; not when it’s a significant
> > chunk
> > of CVEs. The new approach with the CVE request form is generating
> > too many
> > of these. We shouldn’t expect a crowd-sourced approach to
> > dispute/reject
> > CVEs. There is no value whatsoever in it for people to spend time
> > on it.
> > Vulnerability reporters are focused on how they can request the
> > most CVEs
> > for their own findings; including making money off bug bounties;
> > not “waste
> > time” disputing the findings of others.
> > 
> > 
> > 
> > The extremely few CVE consumers, who do spend resources to validate
> > CVE
> > assignments, are not going to go through the whole dispute process
> > and
> > provide sufficient evidence. In fact, it currently seems like much
> > more of
> > an effort to dispute a CVE than request one. That’s worth
> > pondering.
> > 
> > 
> > 
> > 
> > 
> > 5. Preventing duplicates
> > 
> >    1. The comments about duplicates largely relate to some deficits
> > that
> >    existed in CNT1 (Independently fixable). In particular, CNT1 did
> > not
> >    provide guidance on how to handle situations where there is some
> > evidence
> >    that two issues are the same vulnerability, but you are not
> >    certain. MITRE's policy in these cases was to follow the
> > groupings the
> >    requester used when making the request. However, in the most
> > recent rules
> >    revision, we proposed a change to CNT1 in which the issues would
> > be merged
> >    into a single CVE ID if there is uncertainty on whether they are
> >    duplicates.  The change was approved and MITRE will be following
> > it going
> >    forward.  The new rules should stop these types of possibly
> > duplicate
> >    assignments from being made in the future. This issue partially
> > explains
> >    why CVE-2017-15773 and CVE-2017-15738 are separate.
> > 
> > 
> > 
> > This is a good change. When in doubt, a merge is better than a
> > split. We
> > can always split later if more details become available (rarely
> > happens) to
> > justify it.
> > 
> > 
> > 
> > 
> > 
> > 6. How much certainty is required before making a decision on
> > assigning a
> > CVE ID?
> > 
> >    1. Taking a step back, much of this discussion circles around
> > the
> >    degree of certainty needed to assign a CVE ID. How is this
> > certainty
> >    obtained and what is the cost of obtaining the certainty? Is it
> > worth the
> >    results?
> >    2. Carsten: What process did you use to determine that CVE-2017-
> > 15738
> >    and CVE-2017-15773 were duplicates? How long did it take?
> > 
> > 
> > 
> > I simply looked at the crash output provided with one of the
> > reports and
> > compared it to others with similar characteristics. It was obvious
> > pretty
> > much immediately when processing one of them that it was likely a
> > duplicate
> > of the other. After that I double checked my suspicions by
> > disassembling
> > the affected library from the two products. Standard practice for
> > us
> > really; no fancy or elaborate process needed.
> > 
> > 
> > 
> > It’s hard to say how long it took, as it was part of a larger
> > process to
> > invalidate a large portion of that chunk of assigned CVEs. This is
> > something we do regularly for VulnDB, so we’re talking only a few
> > minutes,
> > though. If I had just needed to invalidate that one issue, it would
> > have
> > taken longer to download and install the two products than spot the
> > dupes
> > and open the libraries up to confirm. In some cases, it could take
> > longer,
> > but anyone familiar with validating vulnerability reports should be
> > able to
> > perform this type of vetting within very short time.
> > 
> > 
> > 
> > /Carsten
> > 
> > 
> >