Skip to content

feat/agent only command for app store requirement retrieval#7427

Closed
brandonreid wants to merge 1 commit into
mainfrom
04-29-feat_agent_only_command_for_app_store_requirement_retrieval
Closed

feat/agent only command for app store requirement retrieval#7427
brandonreid wants to merge 1 commit into
mainfrom
04-29-feat_agent_only_command_for_app_store_requirement_retrieval

Conversation

@brandonreid

@brandonreid brandonreid commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

We're introducing a new pattern for agent-only commands in the Shopify CLI: commands that are hidden from --help and "did you mean" suggestions, intended to be invoked by AI coding agents when prompted by the Shopify AI Toolkit.

The first command in this pattern, app agentget-self-review-requirements, returns an AI agent a filtered set of the Shopify App Store requirements that are checkable against a developer's codebase and come with tailored agent verification guidance. The agent uses the verification guidance in each requirement to inspect the developer's codebase and report back which requirements are likely passing, likely failing, or need review, helping the developer catch issues before submitting for review.

This initial implementation is simple, it requests the requirements from a dedicated page in dev docs. This is temporary. The next iteration will be retrieving the requirements directly from app store, where the authenticated request can track which app has made the request (useful as a trust signal) and filter the requirements for direct relevancy to the app's selected category (if they have declared one).

WHAT is this pull request doing?

New hidden topic + command

  • Registers app:agent as a hidden oclif topic in packages/cli/package.json. Both the topic and individual commands set hidden: true, so nothing under it appears in shopify --help, topic listings, or "did you mean" suggestions, but the command remains directly invokable when called by name.
  • Adds shopify app agent get-app-store-requirements, which fetches the App Store self-review requirements from dev docs (already served as Content-Type: text/markdown, no auth required) and writes the result to stdout via outputResult.

Establishes the agent-only contract

  • Sentinel header on stdout: <!-- shopify-agent: <command-id> --> so agents can verify they parsed the expected payload.
  • Errors go to stderr as a single shopify-agent-error: <message> line.
  • Stable exit codes: 0 success, 1 generic failure, 2 not linked, 3 auth failure, 4 server unavailable.
  • The command extends plain Command (not AppLinkedCommand) — the requirements page is universal, and forcing an app link would trigger a prompt that breaks agent invocation. Future agent commands should only opt into app context if their response actually depends on it.

Postrun hook adjustment

  • Excludes app:agent:* commands from autoUpgradeIfNeeded() in packages/cli-kit/src/public/node/hooks/postrun.ts, alongside the existing notifications and upgrade exclusions. This is mainly to prevent an agent invocation of a fast metadata command from triggering a global npm install -g @shopify/cli@latest as a side effect, which would slow the call down by 10–30s and write spawn output to stdout/stderr.

How to test your changes?

# Build the affected packages
pnpm nx run-many --target=build --projects=cli-kit,app,cli

# Run the command
pnpm shopify app agent get-app-store-requirements

Expected: ~60KB of markdown to stdout starting with <!-- shopify-agent: get-app-store-requirements v1 -->, followed by the live requirements from shopify.dev.

Verify the agent contract

pnpm shopify app agent get-app-store-requirements > /tmp/out.md 2> /tmp/err.txt
wc -c /tmp/out.md /tmp/err.txt
head -1 /tmp/out.md

Expect: stdout ~60KB, stderr 0 bytes, first line is the sentinel.

Verify the command is hidden

pnpm shopify --help | grep -i agent          # should print nothing
pnpm shopify app --help | grep -i agent      # should print nothing
pnpm shopify app aget-app-store-requirements 2>&1 | grep -i "did you mean"  # should NOT suggest the agent command
pnpm shopify app agent get-app-store-requirements --help  # works (direct path)

Verify the error path — temporarily edit REQUIREMENTS_URL in packages/app/src/cli/services/agent/app-store-requirements.ts to a 404 path on shopify.dev, rebuild app, and rerun:

pnpm shopify app agent get-app-store-requirements 2>/tmp/err.txt; echo "exit=$?"
cat /tmp/err.txt

Expect: exit=4, stderr starts with shopify-agent-error:, stdout empty. Revert the URL when done.

Verify the postrun guard didn't over-match — on a setup where the upgrade nudge would normally fire, confirm a non-agent command still emits it on stderr:

pnpm shopify version 2>/tmp/err.txt; cat /tmp/err.txt

Post-release steps

None for this PR. The command is hidden and the new namespace is unused outside of it. Once the upstream "copy-paste prompt for your agent" UI surface lands (separately, likely in the Partner Dashboard / dev.shopify.com), it will reference this command by name.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows) — pure Node, no platform-specific APIs; uses shopifyFetch from cli-kit which is already cross-platform.
  • I've considered possible documentation changes — intentionally not adding public docs. The command is hidden from docs:generate (which excludes hidden: true commands) and from the packages/cli/README.md regen. The agent-facing "documentation" is the markdown output itself.
  • I've considered analytics changes to measure impact — the existing reportAnalyticsEvent in the postrun hook still runs for agent commands (only the auto-upgrade leg is skipped), so usage of app:agent:* commands will show up in standard CLI analytics like any other command.
  • The change is user-facing — no changeset. The new command is hidden from end users and has no impact on existing public commands. The postrun hook change is additive and only affects the new hidden namespace.is simple,the

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@github-actions github-actions Bot added the no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. label Apr 29, 2026
@brandonreid brandonreid force-pushed the 04-29-feat_agent_only_command_for_app_store_requirement_retrieval branch 2 times, most recently from 5dccb4c to 8987915 Compare April 29, 2026 16:33
@brandonreid brandonreid force-pushed the 04-29-feat_agent_only_command_for_app_store_requirement_retrieval branch from 8987915 to 6c40d5c Compare April 29, 2026 16:42
@github-actions

Copy link
Copy Markdown
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/monorail.d.ts
@@ -74,6 +74,8 @@ export interface Schemas {
             cmd_app_validate_valid?: Optional<boolean>;
             cmd_app_validate_issue_count?: Optional<number>;
             cmd_app_validate_file_count?: Optional<number>;
+            cmd_app_agent_upstream_status?: Optional<number>;
+            cmd_app_agent_response_bytes?: Optional<number>;
             cmd_dev_tunnel_type?: Optional<string>;
             cmd_dev_tunnel_custom_hash?: Optional<string>;
             cmd_dev_urls_updated?: Optional<boolean>;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant