[CVEPRI] Increasing numbers and timeliness of candidates

["Steven M. Christey" <coley@linus.mitre.org>]

All,

Later tonight, I will be proposing about 330 candidates to the
Editorial Board.  Note that this number of candidates was generated in
6 weeks.  We have significantly increased our rate of candidate
production.

January 1 - May 31, 2001:  496 candidates
January 1 - May 1, 2002:   820+ candidates

The yearly totals are as follows:

  2000 -  1189 candidates
  2001 -  1460 candidates
  2002 -  2460 candidates (estimated)


I expect that MITRE will be producing more content at a faster rate
now, because (a) the number of issues being reported is rapidly
increasing, (b) the content team is becoming better and faster at
creating candidates, and (c) MITRE is making a commitment to
generating content more frequently and in a more timely fashion.  For
all practical purposes, (c) means that I am personally generating more
content than I was in late 2001.

123 of today's candidates are "leftovers" from the late-2001 backlog
of candidates.  These are combined into 3 MISC-2001-xxx clusters.
Other "leftovers" from 2001 will continue to trickle in as MITRE
processes the more complex or vague issues.  However, that backlog is
basically gone (though the more complex and daunting legacy backlog
remains).

Note that some of these 2001-era candidates were complex enough that
only consultation with the vendor was sufficient to iron out the
details.  Mark Cox of Red Hat has helped out greatly in this area!  I
have found that in some cases, a painstaking attention to detail, a
high level of expertise, and a detailed discussion with the vendor is
the only way to generate a usable CAN.

The other 208 candidates being proposed tonight, represent some
portion of the issues that have been announced this year, the bulk of
them before March.  So far, we have 100+ candidates for January 2002
and 150+ for February.

Now that we have overcome the most recent backlog, the CVE content
team can concentrate on becoming more timely.  This may include major
modifications to our process and tools for generating content.  We
have already made significant changes in recent months, including (a)
better error checking during refinement, (b) enhancing incoming
submissions to collect and pre-format CVE style references, which
reduces refinement time and reduces the number of duplicates, (c)
better feedback mechanisms from the Editor (me) to members of the
content team, (d) improvements to the refinement GUI, and (e) better
documentation.

As CVE becomes more timely, I believe that this may have some
noticeable impact on the initial quality of the candidates, in the
following ways:

1) Application of content decisions - having a 1-2 month "perspective"
   on multiple, closely related vulnerabilities helps us to use the
   right level of abstraction for candidates.  With less time for the
   community to find all closely related vulnerabilities for an issue,
   or for details to be "leaked out" after an initial delay, we are
   more likely to make abstraction errors.  In the past year and a
   half, the content decisions have been modified so that they are
   less "brittle" with respect to the amount of information that is
   available at any one time, but they cannot be expected to handle
   imperfect or incomplete information, especially without clear
   vendor acknowledgement of the issue.

   The result may include: (a) an increase in the number of RECAST
   operations that would be required to SPLIT or MERGE items, which
   may increase database maintenance for CVE compatible vendors, as
   well as confusion for CVE end users; or (b) merging newly
   discovered issues with existing CANs if the CDs already dictate it,
   sort of a "soft recast" to use parlance from an old Board meeting.

   Some Board members may argue that content decisions related to
   abstraction are not that essential, and that we should tolerate
   some error in abstraction.  However, I believe that as product
   liability and security becomes more important and quantifiable,
   good metrics will become more important.  Content decisions ensure
   that CVE-based metrics are as reliable as possible.  This will help
   ensure that comparisons between products - whether software
   products or security products - remain as fair as possible.

2) CANs will be proposed with fewer references, since not every data
   source will have created the references at the time the candidate
   is created.  This is especially the case with vendor bulletins.

3) We will begin proposing more CANs before vendors have provided a
   patch or other acknowledgement (typically between 7 and 45 days, if
   the issue was released before a fix was available).  This in turn
   will impact the vendor acknowledgement field that we provide to
   voters.  Subsequently, this impacts the level of confidence that a
   voter may have in an issue; without vendor acknowledgement, a voter
   may be more likely to NOOP an issue.

   This will result in more "permanent" candidates and/or delays in
   moving candidates to the entry stage, because there will likely be
   fewer supporting votes, overall.

   There will be several ways to combat this:

   (a) more frequent voting by more Editorial Board members

   (b) notifying voters when the vendor acknowledgement status has
       changed, in case some want to change their NOOP.

   (c) figuring out a method that allows voters to do a "conditional
       acceptance" if one or more Board members replicate the issue.


It is likely that we had all 3 of these problems to some degree in
late 1999 and 2000, when things were a little more timely.  With the
perspective of having extensive and mature data (i.e. the backlog), I
am better able to understand the impact of *not* having all that data
available.

Although there will be some growing pains, increasing CVE's timeliness
and capacity is an important goal.  As MITRE modifies its processes
over the coming months, we will be considering many factors, such as
the ones I outlined above.  We will try to balance speed, correctness,
and the maintenance costs for voting Editorial Board members, CVE
compatible products, and CVE end users.  Really though, it's just
another day in the life of CVE ;-)


- Steve