Generate LLMs text during site build

[koriym]

Summary

• generate llms-full.txt in the Pages workflow and local serve setup
• stop committing generated llms-full.txt from the scheduled update workflow
• ignore generated llms-full.txt and update bin documentation

Verification

• zsh -ic "sphp85; php -l bin/generate_llms_full.php"
• zsh -ic "sphp85; php bin/generate_llms_full.php"
• git diff --cached --check

Note: local Jekyll build was not run successfully because required bundle gems are not installed in this local environment.

Summary by CodeRabbit

• Chores

  • CI workflows updated to run the documentation generator during site builds and to tighten publish permissions to the deploy job only.
  • Local developer start script now runs the doc generator before building and fails loudly on errors.
  • Link-checker installation steps improved for more reliable execution.

• Documentation

  • Tracking and auto-commit now focus only on single-page English and Japanese documentation.
  • Removed the large aggregated documentation artifact and updated docs guidance accordingly.

• Bug Fixes

  • Generator enhanced with stronger I/O and content-validation error handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6e5b0b52-36de-408e-bd3d-2f6d854fe07e

📥 Commits

Reviewing files that changed from the base of the PR and between 79da511 and 18a1c0e.

📒 Files selected for processing (2)

• .github/workflows/link-checker.yml
• .github/workflows/validate-documentation.yml

✅ Files skipped from review due to trivial changes (1)

• .github/workflows/link-checker.yml

---

📝 Walkthrough

Walkthrough

Integrates PHP llms-full generation into the primary Jekyll build and local serve, removes llms-full tracking from the documentation-update workflow (now tracking only English/Japanese single-page files), adds llms-full.txt to .gitignore, updates README examples, and hardens the PHP generator's I/O and regex error checks.

Changes

LLM Documentation Generation and Workflow Consolidation

Layer / File(s)	Summary
Integrate PHP generation into builds and local serve .github/workflows/jekyll.yml, bin/serve_local.sh	Jekyll workflow sets up PHP 8.3 and runs php bin/generate_llms_full.php during the build; bin/serve_local.sh runs the PHP generator before bundle exec jekyll build. Deploy job permissions for Pages are scoped to the deploy job.
Refocus update-documentation workflow to single-page outputs .github/workflows/update-documentation.yml, .gitignore	Remove baseline copy, change detection, staging, and summary entries for llms-full.txt; change detection and commits now consider only English and Japanese single-page files. Add llms-full.txt to .gitignore.
Harden PHP generator I/O and regex handling bin/generate_llms_full.php	Add checks after file_get_contents, verify preg_replace/preg_replace_callback results are not null, and validate file_put_contents return and written byte count; throw RuntimeException on failures.
Update README docs and examples bin/README.md	Move PHP generation into the main build run example, rename automated updates heading to emphasize single-page docs, and remove llms-full regeneration from the update-documentation example.
Adjust link-checker installs and minor workflow formatting .github/workflows/link-checker.yml, .github/workflows/validate-documentation.yml	Install lychee by moving the binary from the extracted architecture-specific directory into /usr/local/bin/; minor whitespace/placement edits in summaries and loop closings.

Sequence Diagram

sequenceDiagram
  participant GH_Actions as GitHub Actions (jekyll.yml)
  participant SetupPHP as shivammathur/setup-php
  participant PHP_Generator as php bin/generate_llms_full.php
  participant Jekyll as bundle exec jekyll build
  participant Local_Serve as bin/serve_local.sh
  participant Deploy as deploy job (pages)

  GH_Actions->>SetupPHP: setup php 8.3
  GH_Actions->>PHP_Generator: run php bin/generate_llms_full.php (build step)
  Local_Serve->>PHP_Generator: run php bin/generate_llms_full.php (pre-build)
  PHP_Generator->>Jekyll: generate llms-full content for site build
  Jekyll->>Deploy: produce site artifacts
  Deploy->>Deploy: pages: write / id-token: write (scoped to deploy job)

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 I hopped through docs and build-time rails,
PHP warmed up, then told its tales,
Single pages sing, the big file hides,
Generators check, no silent tides,
A tidy build — hop on, no fails!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Generate LLMs text during site build' accurately describes the primary change: moving llms-full.txt generation from a scheduled workflow to the Jekyll site build process.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/ci-generate-llms-full

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

.github/workflows/update-documentation.yml (1)

46-50: ⚡ Quick win

Drop llms-full.txt regeneration from this workflow.

This job now commits only single-page docs, so regenerating llms-full.txt here is redundant and increases failure surface.

Proposed simplification

       - name: Regenerate documentation
         run: |
-          echo "Regenerating llms-full.txt..."
-          php bin/generate_llms_full.php
-          
           echo "Regenerating single-page documentation..."
           ruby bin/merge_md_files.rb

(After this, you can also remove the PHP setup step from this workflow.)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/update-documentation.yml around lines 46 - 50, Remove the
"Regenerate documentation" step that echoes and runs php
bin/generate_llms_full.php from the workflow (the step with name "Regenerate
documentation") because llms-full.txt regeneration is redundant; delete the two
lines that run the echo and the php command and remove the whole step block, and
then also remove the PHP setup step (the job/step that installs PHP) since it
will no longer be needed.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/jekyll.yml:
- Line 38: The workflow currently uses tag-pinned actions (e.g.,
shivammathur/setup-php@v2, actions/checkout@v4, ruby/setup-ruby@v1,
actions/configure-pages@v5, actions/upload-pages-artifact@v3,
actions/deploy-pages@v4); replace each tag reference with the corresponding
commit SHA for that action (pin to a specific commit) to harden the supply
chain, updating the `uses:` lines to the SHA-pinned form for
shivammathur/setup-php, actions/checkout, ruby/setup-ruby,
actions/configure-pages, actions/upload-pages-artifact and actions/deploy-pages.

In `@bin/serve_local.sh`:
- Around line 9-10: The script runs php bin/generate_llms_full.php before Jekyll
but doesn't fail fast; update bin/serve_local.sh to abort if php
bin/generate_llms_full.php fails by enabling strict exit-on-error (e.g., set -e
at top) or chaining the commands so bundle exec jekyll build/serve only runs
when php bin/generate_llms_full.php succeeds; ensure references to the commands
php bin/generate_llms_full.php and bundle exec jekyll build (and/or bundle exec
jekyll serve) are adjusted accordingly so stale/missing artifacts cannot be
used.

---

Nitpick comments:
In @.github/workflows/update-documentation.yml:
- Around line 46-50: Remove the "Regenerate documentation" step that echoes and
runs php bin/generate_llms_full.php from the workflow (the step with name
"Regenerate documentation") because llms-full.txt regeneration is redundant;
delete the two lines that run the echo and the php command and remove the whole
step block, and then also remove the PHP setup step (the job/step that installs
PHP) since it will no longer be needed.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a10f4967-9176-47c6-823d-8796544c332f

📥 Commits

Reviewing files that changed from the base of the PR and between e977cbc and 003c6b8.

📒 Files selected for processing (6)

• .github/workflows/jekyll.yml
• .github/workflows/update-documentation.yml
• .gitignore
• bin/README.md
• bin/serve_local.sh
• llms-full.txt

💤 Files with no reviewable changes (1)

• llms-full.txt

[koriym]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[koriym]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.