doc.rust-lang.org/robots.txt doesn’t match src/doc/robots.txt

The intention in #59213 was to add robots.txt to what gets uploaded to doc.rust-lang.org, but it appears that either the manual step of copying this over for a release hasn’t been happening, or that the automation was never done, or the automation isn’t working.

The contents of src/doc/robots.txt doesn’t match what I’m currently seeing on https://doc.rust-lang.org/robots.txt:

User-agent: *
Disallow: /0.3/
Disallow: /0.4/
Disallow: /0.5/
Disallow: /0.6/
Disallow: /0.7/
Disallow: /0.8/
Disallow: /0.9/
Disallow: /0.10/
Disallow: /0.11.0/
Disallow: /0.12.0/
Disallow: /1.0.0-alpha/
Disallow: /1.0.0-alpha.2/
Disallow: /1.0.0-beta/
Disallow: /1.0.0-beta.2/
Disallow: /1.0.0-beta.3/
Disallow: /1.0.0-beta.4/
Disallow: /1.0.0-beta.5/

(lines 20-27 in the repo are not present on the live site)

Thanks!

1 thought on “doc.rust-lang.org/robots.txt doesn’t match src/doc/robots.txt

Comments are closed.