ci: add robots.txt and sitemap to Netlify docs build

[chatman-media]

Summary

• generate robots.txt for the deployed docs site
• generate a sitemap for the current main and versioned docs paths
• keep internal Netlify build paths excluded from indexing

Closes #4855

Checks

• python3 -m compileall -q noxfile.py
• synthetic generator check for sitemap/robots output
• git diff --check

[davidhewitt]

Thanks, this looks good. Please fix the fmt issues.

[codspeed-hq]

Merging this PR will improve performance by 11.84%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 124 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚡	critical_section_creation	762.5 ns	675 ns	+12.96%
⚡	test_empty_class_init_py	11.7 µs	10.6 µs	+10.74%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing chatman-media:codex/pyo3-docs-indexing (7252413) with main (5c6807d)}

[chatman-media]

Thanks! Fixed the ruff formatting in noxfile.py — the robots.txt block is now a triple-quoted string (matching the existing _redirects block style), and ruff format --check + ruff check are both green locally. I also merged main to un-stale the branch.

Heads-up on guide-build: it's failing on lychee hitting https://www.unicode.org/faq/normalization returning 404. That link lives in dependency rustdoc (not in this PR — check-guide doesn't touch the netlify/robots/sitemap code path), and it's flaky on Unicode's end: the latest main run checked the same URL and got 0 errors. Looks like a transient external link-check failure rather than anything from this change — a re-run should clear it.