Rollup merge of #59213 - kornelski:robots, r=Mark-Simulacrum

Track changes to robots.txt

Currently `robots.txt` of doc.rust-lang.org is not part of any repo, so there's [no way to contribute any changes to it](https://internals.rust-lang.org/t/deadlock-about-fixing-outdated-documentation-links-in-search-engines/9374), such as needed for #44894 and countless dupes of this issue.

I propose adding it to this repo. I'm not in control of the infrastructure, so I can't help to automate deployment of it, but even just having the file under source control is IMHO a step forward.

Author: kennytm

src/doc/robots.txt

@@ -0,0 +1,19 @@
+# NB: This file is not automatically deployed. After changes, it needs to be uploaded manually to doc.rust-lang.org
+User-agent: *
+Disallow: /0.3/
+Disallow: /0.4/
+Disallow: /0.5/
+Disallow: /0.6/
+Disallow: /0.7/
+Disallow: /0.8/
+Disallow: /0.9/
+Disallow: /0.10/
+Disallow: /0.11.0/
+Disallow: /0.12.0/
+Disallow: /1.0.0-alpha/
+Disallow: /1.0.0-alpha.2/
+Disallow: /1.0.0-beta/
+Disallow: /1.0.0-beta.2/
+Disallow: /1.0.0-beta.3/
+Disallow: /1.0.0-beta.4/
+Disallow: /1.0.0-beta.5/