# robots.txt for the virtual hosts at heasarc.gsfc.nasa.gov # We really don't need 18 virtual hosts indexed round the clock. # Having heasarc.gsfc.nasa.gov indexed should be enough. # Modified the virtual hosts robots.txt file specifically for swift # to allow wget to download data. User-Agent: * Disallow: / User-Agent: Wget Disallow: