[TECH] Possible CVE Confidence Levels

["Steven M. Christey" <coley@linus.mitre.org>]

Following is a proposal for different approaches to CVE confidence
levels.  This proposal includes 2 different options:

- 3 levels of confidence (supports simplicity)
- 4 levels of confidence (supports the "web of confidence")

Thanks to Barbara Pease (of the CVE content team) and Andre Frech for
developing these levels with me and writing various sections of this
description.

- Steve


Background rationale
--------------------

In the fields of biological and medical research, the process for
confidence building for an investigator's work is formal and well
established.  First, a laboratory builds its own confidence that it
correctly understands the nature of the problem and has given the best
possible interpretation of experimental results by reproducing the
results using similar or entirely different experimental techniques.
The varied approaches increase confidence through the constancy and
the consistency of the data.  This would be analogous to serious
testing and analysis where no source code is available, since
biological systems under investigation are black boxes with only
partial characterization.

At that point, the laboratory team submits a paper to a reputable
journal.  The paper goes out for peer review by other researchers
engaged in work in the same field.  The reviewers often make
suggestions for additional experimentation.  The laboratory carries
them out and then resubmits a revised paper.

A lone publication within the research community is still not enough.
Once the paper is published, other interested laboratories using the
methodology section of the paper try to reproduce the results.  Often
they do additional work supporting the original contention.  Those
laboratories publish their papers just as rigorously refereed
confirming the original publication.  The original workers become
established with the highest confidence in the literature, and people
can accept the work and build on it to further elucidate a line of
investigation.

The vulnerability analysis community does not come close to this
approach for establishing confidence.  There is no formal peer review.
There are varying methods for verifying that a vulnerability exists,
but they are not standardized or even well-documented.  In addition,
the volume of work, and the speed with which the body of knowledge is
growing, suggest that such an established academic approach is not
feasible in the short term, if ever.


3 Levels of Confidence
----------------------

Andre, Barbara, and I considered more granular levels of confidence
(between 5 and 6 levels), but they were too specific to our own
individual perspectives and intended uses of those levels.  They could
be too difficult to easily describe to the average user.

We arrived at 3 levels of confidence that were agreeable to all of us,
with minor variations.

Note that the levels of confidence are described from the perspective
of the individual or organization who is establishing the level of
confidence in the vulnerability information.  Different entities may
have different levels of confidence in the same report based on their
own expertise, who they trust, etc.  There is also the assumption that
the entity who is establishing the confidence is experienced in
information security analysis and/or testing.

In the context of CVE, each voter could assign the amount of
confidence that the voter has in the specific candidate.


Highest level - CONFIRMED
-------------------------
- a security-aware vendor has confirmed the problem
- I or my organization has confirmed the problem
- an organization trusted within the community for its experience in
  information security has confirmed the problem with testing.

The core rationale behind the CONFIRMED level is that individuals or
organizations experienced in information security issues have proven
that the problem exists with hands on testing.

This would be the highest confidence level based on the existence of
vendor security advisories demonstrating astute security awareness and
knowledge, in combination with either vendor patches or bug fix
references, e.g. developers' README files.  Vendor reports of default
configurations creating a security problem should also be included.
Reputable organizations that are experienced and knowledgeable in the
field of information security could be included in this level.

RATIONALE -- When duplicating a problem, one may instrument the
systems while reviewing hardware logic and source code where relevant.
Some bugs are exceedingly complex and defy a simple source code
analysis.  Even if source is available, the source code reviewer needs
a certain level of expertise to analyze the problem.  The affected
software vendor may not have the expertise to recognize that there is
a problem in the first place - or they may not even acknowledge that
the problem exists.  If a fix is available, it has to be tested to
make sure that the problem was really fixed.  This process can confirm
both the complex cases and those fixes where source code analysis
identified the problem in simpler cases.

Preparing patches and bug fixing for releases is done with source code
available, experience in the product and the opportunity to discuss
the matter with others also experienced in the product.  In the case
of security vulnerabilities, as long as the vendor is knowledgeable in
security matters, then there is reasonable confirmation that the bug
fixed presents a real security problem.

For a CONFIRMED level of confidence, we could include technical
reports, email, verbal communication, and technical memoranda and
advisories reporting well thought out tests from organizations other
than the vendor, if their line of business uses such testing, and the
organization is reputable.  The organization would stand on its own
reputation and extensive experience.


Middle level - LIKELY
---------------------
- multiple sources have confirmed the problem with testing, but their
  experience level is unknown (proof of testing is a shared exploit or
  test)
- multiple sources report a problem, but without proof of testing
- vendor reports a problem but there is no proof of testing (i.e no
  patch, work around or statement of tests is available).
- vendor reports a problem, but vendor is inexperienced in information
  security and is clearly responding to recommended fixes from sources
  established at the LIKELY level, rather than the CONFIRMED level
- exploit source code exists, but hasn't been tested by me or my
  organization
- a reliable source (community-trusted) has reported a problem

This middle level would indicate a reproducible but informal
vulnerability testing where exploitation is reported by several
people.  The experience and background of the announcers (or those who
replicate the issue) may not be known.  Reports showing astute
observation and speculation about a configuration problem may also be
included.  Vendor reports and fixes in which the issue is addressed
without sufficient proof that it is exploitable, or in cases in which
the vendor does not necessarily have the skills to confirm the issue
themselves, could be recorded at this confidence level.

RATIONALE -- The reporting mechanisms and methods of verification are
informal but reasonable.  There should be an exploit or test that
people are using to confirm the original report.  The technical
content should show observation and insight, but since access to
source code may not be possible and the experience level of the
testers may not be known, and it cannot be confirmed with the
available data.

These references would more than likely be informal mailing list
discussions, private communications to colleagues at other
institutions about some hacking around with the problem, etc.  The
work is ad hoc but indicates that the reporters have some skills in
identifying vulnerabilities.

Vendor advisories that do not have patches available might be included
here.  The hope is that some discussion at least has taken place with
knowledgeable people at the vendor site, but without a patch it is not
clear that any tests were run, unless there is confirmation from a
reliable source.  One could also include vendor advisories and patches
from vendors where the level of their security expertise is not clear.
They may fix a bug that others at the same confidence level report,
without insight as to whether or not it presents a security problem.



Lowest level - UNCERTAIN
------------------------
- disputed without independent confirmation at the CONFIRMED level
- announced without any references or discussion of testing
- no references

This level could include reports with poor references or no
references.  It could include reports in which no test or exploit is
included, and where no one else reports that they duplicated the
problem.  This could include reports that are in dispute, but the
evidence cited is not at the CONFIRMED level.

One issue is the case in which a security-aware vendor disputes a
claim from a CONFIRMED-level, trusted organization.

RATIONALE -- there is insufficient evidence to prove that the issue is
real.



4 levels of confidence
----------------------

An alternate approach is to establish another "middle" level of
confidence, the TRUSTED level.  The TRUSTED level basically separates
the items in the CONFIRMED and LIKELY levels that deal with an
organization's trust in third party reporters.

Here's a simplified writeup of the 4-level approach.

CONFIRMED:
  - the vendor has confirmed the problem
  - I or my organization has confirmed the problem

TRUSTED:
  - I trust the entity who announced the problem
  - I trust an entity who said they confirmed the problem

LIKELY:
  - multiple sources have confirmed the problem, but they are not
    trusted
  - there is a plausible exploit that hasn't been tested by me or
    my organization

UNCERTAIN:
  - disputed
  - announced without any confirmation
  - no references

With a TRUSTED level, there is no need for the community to decide
which organizations are to be trusted at the CONFIRMED level.


Pro's and Con's of 3-level versus 4-level confidence
----------------------------------------------------

If there is not a TRUSTED level, then the 3-level approach is easier
to describe to consumers of vulnerability information.  It can avoid
"politicizing" which organizations can be trusted and more cleanly
separates *who* is to be trusted from *how* those trusted entities are
used in assigning confidence.

If there is a TRUSTED level, then each organization that applies a
confidence level to vulnerability information could effectively
"formalize" which organizations they trust.  If a person trusts
organization X, and X trusts Y, then the person might be more willing
to trust reports from Y.  This could be used to more cleanly represent
a "web of trust" or "web of confidence" and differentiate between the
vendor, the entity that assigns the confidence to the issue, and all
other third parties.