When is a vulnerability sufficiently verified to exist?

["Steven M. Christey" <coley@linus.mitre.org>]

The VERIFY clusters identify candidates that are not sufficiently
verified, based on my research.  I don't know for certain that these
candidates identify vulnerabilities that really exist and/or are
exploitable.  Thus I would either REJECT or NOOP them unless other
Editorial Board members were able to provide sufficient confirmation.

The question is, when is a candidate sufficiently verified?  Here are
my own rules.

For me, a candidate could be a verified problem if one of the
following occurs:

1) I can test and verify the problem myself.

2) The vulnerability is acknowledged by the software vendor, either in
an advisory or a patch.

3) There is a posted advisory by a "trusted" entity (e.g. a CERT, or a
vulnerability analysis team which consistently produces accurate
reports)

4) There are two independent sources of information that discuss the
vulnerability.

5) There is independent verification by two different Editorial Board
members.


Many of the VERIFY-BUGTRAQ vulnerabilities were listed in Bugtraq, but
I couldn't find any other sources of information besides the Bugtraq
postings themselves.  Elias and Russ have different approaches for
verifying vulnerabilities before they approve postings to their
respective lists, and maybe something will slip through that isn't
necessarily a problem; or, they might post something that is a
problem, but won't satisfy the CVE vulnerability definition.  This is
not to criticize Elias and Russ' roles as moderators, of course; their
lists deal with newly emerging and volatile information, and the lists
have a broader role than just confirming that something is a problem.
But it does illustrate the question of when the Editorial Board can be
confident that a vulnerability exists.

I imagine that many vulnerability analysis teams often have strong
motivations for making certain that their advisories are accurate, but
their methodology is not discussed, and they rarely indicate how much
verification they've done.

Security tool vendors, as I understand it, have their own problems
with respect to verifying a vulnerability.  Not only do they have to
make sure it exists, but they also have to design tests for it, and
may not have the appropriate resources to do so, especially for
vulnerabilities on unusual platforms.  But does the fact that a
security tool vendor have a test for the vulnerability, sufficiently
confirm that it exists?  What if the fix information for the
vulnerability only says, "Ask your vendor"?

Incident response teams may have a different timeline for
verification; if a vulnerability is exploited in an incident, then
that's proof positive that there is truly a vulnerability.  But how
much verification does the team do itself?  Perhaps it relies on other
teams for the verification.

There are other information sources that may be unreliable.  Consider
exploit sites, where someone's code may not properly exploit the
vulnerability, and therefore doesn't necessarily prove that the
vulnerability exists.

Related to this question of whether a vulnerability exists, is if we
know all the different platforms/OSes/applications where it exists.
Once we know something is truly a vulnerability, when do we know
*enough* about it to say that we understand it enough to place it into
the CVE?

Since the CVE must represent mature information, we should make sure
that every vulnerability in the CVE is sufficiently verified to exist.
What is the best approach?  How do you verify that a vulnerability
exists?  Can the Editorial Board verify all the VERIFY-* candidates,
and if not, why not?

- Steve