RE: Question about robots.txt

["Coffin, Chris" <ccoffin@mitre.org>]

Brian,

> That said, after Kurt's mail in December of 2015... in the last ~ 30 
> - 60 days, I noticed that MITRE finally changed that. Google is now 
> indexing and caching the CVE pages.

We made the change to allow indexing back in Feb of 2016, which was a 
few months after Kurt had pointed out the issue. We apologize to all 
for not replying to the original thread at that time. Dan also 
mentioned the same in a response to you back in April of this year 
(http://common-vulnerabilities-and-exposures-cve-board.1128451.n5.nabble.com/Re-CVENEW-New-CVE-CANs-2017-04-23-19-00-count-1-td722.html#a727).

> Just like you didn't ask us about the 3k+ RESERVED fiasco that got 
> several of us talking about this morning, figuring out how we'd 
> handle it. When NVD spoke up, we all collectively said "hell yeah!"
>
> The fact that NVD called you out, and has since said they will be 
> 'ignoring' those IDs, is also very significant in CVE history. This 
> is the first *real* break that NVD has had from CVE ever. There have 
> been other breaks the last year+, but they were more pedantic and 
> favored NVD > over MITRE/CVE, based on the time of entries becoming 
> public (e.g. NVD published before MITRE did).

We are not absolutely certain what concern you have in the case of the 
RESERVED CVE IDs moving to REJECT status. Please let us know if the 
following explanation does not clear up your concerns.

We have had multiple conversations during Board conference calls 
regarding the fact that there are many RESERVED CVE IDs within the 
current CVE list, and there was a general consensus that they should be 
cleaned up (i.e., REJECT or populate). As you are probably aware, there 
are multiple reasons that a CVE ID might be stuck in a RESERVED status. 
One of those reasons could be that the CNA obtained a block of CVE IDs, 
but never actually assigned some of those IDs to vulnerabilities.

As a first step in tackling the larger cleanup effort, we began 
contacting CNAs in March of this year to determine what CVE IDs they 
had not used from their previously assigned CVE ID blocks. All but a 
couple of CNAs responded and pointed out which CVE IDs were not used. 
In every case, the CVE ID in question moved from a status of RESERVED 
to a status of REJECT. The CVE IDs in question were moved to REJECT 
status earlier today.

You are correct and Dave at NIST had sent a message in regards to this 
first step and he was not clear on exactly what the end result would 
be. Dave and I spoke on the phone, we cleared up the gaps in 
understanding, and even decided to hold off for a day to give the NIST 
NVD folks a bit more time to analyze the impact. 

Dave can correct me if I'm wrong, but we didn't interpret the comment 
"ignored by the NVD" to mean that the NVD team would not publish the 
REJECT CVE entries. Our interpretation is that the NVD team does not 
see a need to analyze the entries and will simply publish them as is, 
with no significant effort on their part.

Regards,

Chris Coffin
The CVE Team

-----Original Message-----
From: jericho [mailto:jericho@attrition.org] 
Sent: Thursday, May 11, 2017 12:32 AM
To: Coffin, Chris <ccoffin@mitre.org>
Cc: Kurt Seifried <kseifried@redhat.com>; cve-editorial-board-list 
<cve-editorial-board-list@lists.mitre.org>
Subject: RE: Question about robots.txt
Importance: High


On Tue, 8 Dec 2015, Coffin, Chris wrote:

: We made the choice a long time ago to not allow indexing of the
: cve.mitre.org web site. At least part of that decision was simply
: resource constraints ? when CVE was in its toddler years, search 
engine
: indexers were very resource intensive.

That 'decision' was based on crap excuses, even back then. =) As 
someone who ran two sites over the time MITRE ran CVE, and intensively 
watched logs on one of them (attrition.org, since 1998-10-07), search 
engines were NOT resource intensive back then. Attrition staff talked 
about that issue and didn't block any of our content in robots.txt 
because search engine spam was present, but not heavy. For those 
interested in Internet history...

forced ~$ more /home/admin/util/list.filter
72.14.203.104
forced.attrition.org
images.search.yahoo.com
casualgamer.org
myspace.com
stumbleupon.com
f-mai.gif
f-bak.gif
f-att.gif
thefiles.gif
panopta.com
divinelanguage.com
forced ~$ grep -i google /home/admin/util/list.* 
/home/admin/util/list.bot:googlebot.com
/home/admin/util/list.bot:Feedfetcher-Google
/home/admin/util/list.filter-old:google.com
/home/admin/util/list.filter-old:google.co.jp/search
/home/admin/util/list.filter-old:google.de
/home/admin/util/list.filter-old:google.fr
/home/admin/util/list.filter-old:google.co.uk
forced ~$

"list.filter-old" is from 2003-08-25. The limited set of Google domains 
should be very telling, given the year and traffic generated.

We actually *stopped* filtering Google at some point, while ignoring 
Yahoo early on. Why? Because they were simply not hammering sites and 
causing any undue burden, to a random desktop machine bought at the 
local computer store. Those were "ignore displaying those entries in 
our log parser", not "block them from reaching our web server" via 
iptables.

That was Attrition when it was run on a ~ $500 box bought in 1998 and 
hosted on a consumer link, compared to MITRE's resources and CVE 
contract money from the government at the time. So to be clear, MITRE's 
answer in 2015, is based on people forgetting what it was like in 1997 
- 1999.

That said, after Kurt's mail in December of 2015... in the last ~ 30 - 
60 days, I noticed that MITRE finally changed that. Google is now 
indexing and caching the CVE pages.

Thank you, as a long-time taxpayer funding MITRE's projects, including 
CVE, to the tune of $1,487,334,000 in MITRE income last year. Good to 
see you making these small changes to help the industry.

: We are currently re-examining this policy and will keep the Board
: posted.

Except... you didn't. Just like you didn't ask us about the 3k+ 
RESERVED fiasco that got several of us talking about this morning, 
figuring out how we'd handle it. When NVD spoke up, we all collectively 
said "hell yeah!"

The fact that NVD called you out, and has since said they will be 
'ignoring' those IDs, is also very significant in CVE history. This is 
the first *real* break that NVD has had from CVE ever. There have been 
other breaks the last year+, but they were more pedantic and favored 
NVD over MITRE/CVE, based on the time of entries becoming public (e.g. 
NVD published before MITRE did).

Brian