[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-09

[github-actions[bot]]

Summary

• Date of test: 2026-06-09
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: As a complete beginner, the home page makes a strong first impression with a clear headline and "Quick Start with CLI" button. However, the quick-start guide assumes considerable prior knowledge (PATs, frontmatter, .lock.yml files) that isn't explained, which creates stumbling blocks before a user can get their first workflow running.
• Workflow run: §27185047094

---

🔴 Critical Issues Found

1. COPILOT_GITHUB_TOKEN Setup Is Cryptic for Beginners

The Quick Start Step 2 mentions setting up a COPILOT_GITHUB_TOKEN that is "a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN." A beginner won't know:

• What a "fine-grained PAT" is
• Why there are two different tokens
• The difference between "Account permissions" and "Repository permissions"

The Note callout reads: "Under Permissions → Account permissions, set Copilot Requests to Read, then generate the token." — but the PAT creation UI is complex and a single wrong choice silently breaks things.

Suggestion: Add a short dedicated subsection or a linked mini-guide titled "Getting your AI token" with numbered screenshots showing exactly where to click.

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8c04d9bccc9bb61f09e6a92778f11e5aa648ce2c1b8573ce4f78bf6e3a0b0474.png?raw=true

2. githubnext/agentics Repository Is Unexplained

Step 2 runs gh aw add-wizard githubnext/agentics/daily-repo-status with no explanation of where this repository comes from or who maintains it. A beginner will wonder:

• Do I need access to githubnext/agentics?
• Is this the only available workflow?
• Why am I running code from someone else's repo?

Suggestion: Add a single sentence like "This references the daily-repo-status workflow hosted in the public githubnext/agentics examples repository."

3. Undefined Jargon: "frontmatter" and ".lock.yml"

Step 4 uses "frontmatter" with a parenthetical explanation — good! — but the .lock.yml file is introduced in Step 2 with a dense paragraph explaining the compilation model, right in the middle of the setup flow. This is too much new information at the wrong moment.

Suggestion: Replace the .lock.yml explanation in Step 2 with a one-liner ("The wizard generates a compiled workflow file automatically — you don't need to edit it") and link to the full explanation in the reference docs.

---

🟡 Confusing Areas

4. "Peli's Agent Factory" Is an Internal Reference

The "What's next?" section links to "Peli's Agent Factory" — a beginner has no idea who Peli is. This reads as an internal joke, not a public onboarding resource.

Suggestion: Either rename the link to something self-explanatory like "Example Workflows Gallery" or add a brief description: "Peli's Agent Factory — a community collection of ready-to-use workflows maintained by a gh-aw contributor."

5. CLI Commands Page Overwhelms Beginners with Advanced Content

The CLI page opens with a great "Most Common Commands" table — excellent! But it immediately cascades into GitHub Enterprise Server setup, version pinning, and configure_gh_for_ghe.sh details. A new user hitting this page after the quick start will feel lost.

Suggestion: Add a "🚀 Just getting started?" callout at the top pointing to the Quick Start, and consider collapsing the GHES section behind a <details> toggle.

📎 [cli.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/3cd141d7b8503d0867c78f88fe3ee49c2a79952458b439be2441dc9622343662.png?raw=true

6. Step 3 Image Does Not Load in Dev Environment

Step 3 promises "The report will look something like this:" but the example screenshot image (daily-repo-report-result.png) fails to load in the Astro dev server (loaded: false, naturalHeight: 0). A beginner following the docs will see broken white space instead of a motivating preview.

📎 [quick-start-image-area.png] (scroll shows where image should appear) — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/cdcb599a83a147953cdc66d2f686629741a9c375499e0d6cd413e0ac6a9babfe.png?raw=true

7. Home Page Security Section May Be Off-Putting for New Users

The home page has a large "Guardrails Built-In" section with a mermaid flowchart showing the security pipeline before any explanation of what the tool does or what a user gets out of it. Security is important, but for a first-time visitor the benefit needs to come before the risk mitigation.

Suggestion: Move the security/guardrails section below the "Key Features" section, and add a brief "What you get" section with a concrete example outcome (e.g., "Wake up to a daily issue summarizing your repository's health").

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/a34b0faa4e2d7d9ca2f0f1d85125b5722b90eed06dd5f4da59482ff9fa34473c.png?raw=true

View minor confusing areas

gh auth login --scopes repo,workflow: The prerequisites mention this command but don't explain when it's needed vs. a standard gh auth login. A beginner may already be logged in and not know if they need to re-authenticate.

Step 2 information density: The explanation of add-wizard and all it does (check prerequisites, select AI engine, set up secret, add workflow, trigger run) is helpful but dense. A visual step-by-step progress bar or numbered sub-steps would help.

"Repository root" not explained: Step 2 says "From your repository root run:" — a complete beginner may not know this means cd into their cloned repo folder.

---

🟢 What Worked Well

• ✅ Clear hero headline: "Repository automation, running the coding agents you know and love" immediately communicates the value proposition.
• ✅ Prominent "Quick Start with CLI" CTA: The home page makes it obvious where to begin.
• ✅ Estimated time: 10 minutes: Sets realistic expectations upfront — greatly appreciated by new users.
• ✅ Prerequisites checklist: Well-structured list. Each item has a verification command (e.g., gh --version, gh auth status). Very beginner-friendly.
• ✅ "Most Common Commands" table on CLI page: Perfect quick reference — exactly what a beginner needs at the top of the page.
• ✅ Step-by-step numbered structure: The 4-step quick start flows logically.
• ✅ Multiple AI engine support clearly stated: Both home page and quick start are clear that Copilot, Claude, Codex, and Gemini are all supported.
• ✅ Inline Tip/Note callouts: The callout boxes for common issues and authentication tips are well-placed.
• ✅ Early development warning: Appreciated transparency, though placement could be reviewed.

---

Recommendations

Quick Wins (High Impact, Low Effort)

1. 🔧 Add one sentence explaining githubnext/agentics is a public examples repo in Step 2.
2. �� Replace the .lock.yml paragraph in Step 2 with a one-liner + link to reference docs.
3. 🔧 Rename "Peli's Agent Factory" to something self-descriptive in the "What's next?" section.
4. 🔧 Investigate why the example report image fails to load in the dev server (Step 3).

Medium-Term Improvements

5. 📖 Create a "Getting your first AI token" mini-guide linked from the Prerequisites section.
6. 📖 Add a "🚀 Just getting started?" callout at the top of the CLI Commands page.
7. 📖 Add a collapsible section or separate page for GitHub Enterprise Server content on the CLI page.

Longer-Term Documentation Improvements

8. 🗺️ Add a "What is GitHub Agentic Workflows in 60 seconds?" video or animated GIF on the home page showing the end result (the daily report issue).
9. 🗺️ Consider a glossary page for terms like "frontmatter", ".lock.yml", "safe outputs", and "agentic workflow" — link to it inline.
10. 🗺️ Consider reordering the home page: benefits → key features → how it works → security, rather than the current mixed order.

---

References:

• §27185047094

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · ⌖ 32.3 AIC · ⊞ 18.9K · ◷

• expires on Jun 9, 2026, 9:22 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-10T05:22:09.750Z.

Closed by Workflow