fix(backend): robots.txtで返却する内容の調整

[kakkokari-gtyih]

What

• すべてのパスを許可していたが、設定やコントロールパネルなどの一部のルートを既定で除外するように
• さらに、ugcVisibilityForVisitorがnoneの場合は除外するルートを追加するように
  • ユーザーが個別にクローラー除外の設定を入れているのに関しては、既存の仕組み（X-Robots-Tag や meta robots タグ）でカバーされている
  • リモートのユーザーに関しては、既定でmeta robotsタグが入るようになっているのでそれでカバーされている

Why

#17143 で述べられている懸念事項は概ねこれで解決できる可能性がある

Additional info (optional)

Checklist

• Read the contribution guide
• Test working in a local environment
• (If needed) Add story of storybook
• (If needed) Update CHANGELOG.md
• (If possible) Add tests

[github-actions]

このPRによるapi.jsonの差分
差分はありません。
Get diff files from Workflow Page

[codecov]

Codecov Report

❌ Patch coverage is 0% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.48%. Comparing base (cdb958c) to head (4007a9f).
⚠️ Report is 62 commits behind head on develop.

Files with missing lines	Patch %	Lines
...ages/backend/src/server/web/ClientServerService.ts	0.00%	26 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           develop   #17165       +/-   ##
============================================
+ Coverage    14.03%   63.48%   +49.45%     
============================================
  Files          238     1161      +923     
  Lines        11296   115963   +104667     
  Branches      3736     8348     +4612     
============================================
+ Hits          1585    73619    +72034     
- Misses        7586    40158    +32572     
- Partials      2125     2186       +61     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Backend memory usage comparison

Before GC

Metric	base (MB)	head (MB)	Diff (MB)	Diff (%)
VmRSS	313.67 MB	308.72 MB	-4.94 MB	-1.57%
VmHWM	313.67 MB	308.72 MB	-4.94 MB	-1.57%
VmSize	23108.19 MB	23103.63 MB	-4.56 MB	-0.01%
VmData	1379.58 MB	1374.29 MB	-5.28 MB	-0.38%

After GC

Metric	base (MB)	head (MB)	Diff (MB)	Diff (%)
VmRSS	314.38 MB	308.72 MB	-5.65 MB	-1.79%
VmHWM	314.38 MB	308.72 MB	-5.65 MB	-1.79%
VmSize	23109.19 MB	23104.04 MB	-5.14 MB	-0.02%
VmData	1380.58 MB	1374.71 MB	-5.87 MB	-0.42%

After Request

Metric	base (MB)	head (MB)	Diff (MB)	Diff (%)
VmRSS	314.64 MB	309.39 MB	-5.24 MB	-1.66%
VmHWM	314.72 MB	309.39 MB	-5.33 MB	-1.69%
VmSize	23109.19 MB	23104.46 MB	-4.72 MB	-0.02%
VmData	1380.58 MB	1375.13 MB	-5.45 MB	-0.39%

See workflow logs for details

[ghost]

対応ありがとうございます！こういう形で開発に貢献できるのはすごく嬉しいです☺️