RE: Question about robots.txt

["Coffin, Chris" <ccoffin@mitre.org>]

Hi Kurt,

 

We made the choice a long time ago to not allow indexing of the cve.mitre.org web site. At least part of that decision was simply resource constraints – when CVE was in its toddler years, search engine indexers were very resource intensive.

 

We are currently re-examining this policy and will keep the Board posted.

 

Chris Coffin

The CVE Team

The MITRE Corporation

 

From: owner-cve-editorial-board-list@lists.mitre.org [mailto:owner-cve-editorial-board-list@lists.mitre.org] On Behalf Of Kurt Seifried
Sent: Wednesday, December 02, 2015 11:32 AM
To: cve-editorial-board-list <cve-editorial-board-list@lists.mitre.org>
Subject: Question about robots.txt

 

http://cve.mitre.org/robots.txt

 

User-agent: *

Disallow: /cgi-bin/

 

This means all the CVE's in the database, e.g.

 

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-0160

 

are not really searchable/indexed via any search engine, is this intentional?

 

--
Kurt Seifried -- Red Hat -- Product Security -- Cloud
PGP A90B F995 7350 148F 66BF 7554 160D 4553 5E26 7993
Red Hat Product Security contact: secalert@redhat.com