Conversation
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
🟡 Acceptable - Solid concept and structure, but fix the criteria.md duplication before merge. "Bad programmers worry about the code. Good programmers worry about data structures."
plugins/onboarding/skills/agent-readiness-report/scripts/scan_agent_instructions.sh
Show resolved
Hide resolved
|
|
||
| --- | ||
|
|
||
| ## Pillar 2 · Feedback Loops |
There was a problem hiding this comment.
We will want to replace this later when @xingyaoww is done with our recommended verification stack.
| | 3 | **Multi-model support** | Instructions that work across different AI models/tools, not locked to one vendor | 2+ distinct agent config types from features 1–2 present in same repo | | ||
| | 4 | **Agent skills or capabilities** | Packaged, reusable abilities the agent can invoke | `.claude/skills/`, `.factory/skills/`, `skill.md` files, tool definition files | | ||
| | 5 | **Tool server configuration** | Config for agent tool protocols (lets agents use external tools) | `.mcp.json`, `mcp.config.js`, tool server manifests | | ||
| | 6 | **Agent prompt library** | Pre-built prompts for common tasks in this repo | `.github/prompts/`, `prompts/` directory, prompt template files | |
There was a problem hiding this comment.
Maybe not necessary if we have skills?
|
|
||
| | # | Feature | What to look for | Evidence | | ||
| |---|---------|------------------|----------| | ||
| | 1 | **Agent instruction file** | A dedicated file telling agents how to work in this repo — conventions, banned patterns, common commands | `AGENTS.md`, `CLAUDE.md`, `COPILOT.md`, `CONVENTIONS.md` at root | |
There was a problem hiding this comment.
AGENTS.md is the standard now, probably just need that.
| | # | Feature | What to look for | Evidence | | ||
| |---|---------|------------------|----------| | ||
| | 1 | **Agent instruction file** | A dedicated file telling agents how to work in this repo — conventions, banned patterns, common commands | `AGENTS.md`, `CLAUDE.md`, `COPILOT.md`, `CONVENTIONS.md` at root | | ||
| | 2 | **AI IDE configuration** | Settings or rules for AI-powered editors/IDEs | `.cursor/rules/`, `.cursorrules`, `.github/copilot-instructions.md`, `.github/instructions/`, `.claude/settings.json` | |
There was a problem hiding this comment.
Can probably remove, we don't necessarily need to recommend that people use IDEs.
| |---|---------|------------------|----------| | ||
| | 1 | **Agent instruction file** | A dedicated file telling agents how to work in this repo — conventions, banned patterns, common commands | `AGENTS.md`, `CLAUDE.md`, `COPILOT.md`, `CONVENTIONS.md` at root | | ||
| | 2 | **AI IDE configuration** | Settings or rules for AI-powered editors/IDEs | `.cursor/rules/`, `.cursorrules`, `.github/copilot-instructions.md`, `.github/instructions/`, `.claude/settings.json` | | ||
| | 3 | **Multi-model support** | Instructions that work across different AI models/tools, not locked to one vendor | 2+ distinct agent config types from features 1–2 present in same repo | |
There was a problem hiding this comment.
Not necessary, since the purpose of this is mainly to tell that this is ready for openhands.
| | 4 | **Agent skills or capabilities** | Packaged, reusable abilities the agent can invoke | `.claude/skills/`, `.factory/skills/`, `skill.md` files, tool definition files | | ||
| | 5 | **Tool server configuration** | Config for agent tool protocols (lets agents use external tools) | `.mcp.json`, `mcp.config.js`, tool server manifests | | ||
| | 6 | **Agent prompt library** | Pre-built prompts for common tasks in this repo | `.github/prompts/`, `prompts/` directory, prompt template files | | ||
| | 7 | **Component-level agent guidance** | Different parts of the codebase have their own agent instructions | `AGENTS.md` or instruction files in subdirectories (e.g. `frontend/AGENTS.md`, `api/CLAUDE.md`) | |
There was a problem hiding this comment.
| | 7 | **Component-level agent guidance** | Different parts of the codebase have their own agent instructions | `AGENTS.md` or instruction files in subdirectories (e.g. `frontend/AGENTS.md`, `api/CLAUDE.md`) | | |
| | 7 | **Component-level agent guidance** | Different parts of the codebase have their own agent instructions | `AGENTS.md` or instruction files in subdirectories (e.g. `frontend/AGENTS.md`, `api/AGENTS.md`) | |
| | 11 | **API documentation** | Reference docs for the project's interfaces | `openapi.yaml`, generated HTML docs, `doc.go` files, Swagger UI, `api-docs/` | | ||
| | 12 | **Inline code documentation** | Doc comments, docstrings — agents read these to understand intent | JSDoc `/** */` blocks, Python docstrings, GoDoc comments, RDoc `#` blocks, Rust `///` | | ||
| | 13 | **Runnable examples** | Working example code the agent can study and imitate | `examples/` directory, `_examples/`, example apps with their own READMEs | | ||
| | 14 | **Changelog** | History of what changed and how entries should be written | `CHANGELOG.md`, `CHANGES.md`, `HISTORY.md`, release notes in GitHub Releases | |
There was a problem hiding this comment.
This can also just be tracked in github, so maybe not necessary.
| | 46 | **Label automation** | Automatic PR/issue labeling based on paths or content | `.github/labeler.yml`, label-sync config, auto-label workflows | | ||
| | 47 | **Multi-platform CI** | CI matrix covering multiple OS, arch, or runtime versions | `matrix:` in CI with `os: [ubuntu, macos, windows]` or multiple language versions | | ||
| | 48 | **Deployment automation** | Automated deployment pipeline | Deploy workflow triggered on merge/tag, staging + production environments in CI | | ||
| | 49 | **Automated code review checks** | Bot-driven review checks beyond CI | Danger.js config, review bot config, required review assignments, CODEOWNERS + required reviews | |
There was a problem hiding this comment.
This should recommend the openhands review?
| | 57 | **Security scanning** | Automated vulnerability scanning in CI | `.github/workflows/codeql.yml`, Snyk config, `gosec` in CI, Trivy, Dependabot security alerts | | ||
| | 58 | **Git attributes** | Line endings, diff drivers, LFS, linguist overrides | `.gitattributes` with `text=auto`, `linguist-generated`, LFS tracking patterns | | ||
| | 59 | **Contributor agreement** | DCO sign-off or CLA process | DCO bot config, `Signed-off-by` requirement in contributing guide, CLA-assistant config | | ||
| | 60 | **Governance model** | Documented maintainer roles, decision-making process | `GOVERNANCE.md`, `MAINTAINERS.md`, governance section in docs, team/role descriptions | |
There was a problem hiding this comment.
This seems duplicated with codeowners
|
|
||
| ## Five Pillars | ||
|
|
||
| | Pillar | Question | Features | |
There was a problem hiding this comment.
If we change anything based on my above comments, change here too.
|
@neubig I've updated this PR if you want to take another look. The idea is that the Instead, the main entry-point is the |
3 participants
This PR adds a plugin that defines a set of skills to help get a repository ready for OpenHands to start being productive.
New skills:
setup-openhands-- initial configuration and integrations for OpenHandssetup-agents-md-- advice for creating reasonableAGENTS.mdfiles from scratchsetup-pr-review-- instructions on setting up the OpenHands review workflowagent-readiness-report-- moved fromreadiness-report, optimized (see below)improve-agent-readiness-- generates high-impact fixes to improve agent readinessThe intended flow to getting a repository ready for OpenHands is now just running the
/setup-openhandsskill. This references the other setup skills and follows the remaining advice in this documentation on SDLC setup.Agent Readiness
If the step-by-step workflow in
/setup-openhandsisn't enough, we also have a process for hardening the repo based on current best practices.Running the
/agent-readiness-reportskill will examine the repository, score it across a variety of factors, and generate a report. Running the/improve-agent-readinessskill will examine the report, identify high-impact fixes, and update the report when done.The features and fixes identified by this loop have been sourced from what other state-of-the-art AI projects are doing. Not all the features are applicable to all repos, and many identify technologies/config options that are not OpenHands-specific.
Updating the Readiness Report
The previous readiness report was based (loosely) on the CMMI approach and had some strong assumptions on the repository domain (web-based apps). Defining maturity levels for such a fast-moving field is a bit strange, so the new approach is based on the BSIMM approach instead.
To that end, I gathered ~125 open-source repos across a mix of languages, domains, apparent AI readiness, and size. OpenHands inspected each repo for evidence of AI readiness across four pillars (along with an "other"):
agents.mdfiles,.cursorconfigs, skills, etc.This produced >1.2k features that act as evidence for or against a repo's AI readiness. Some multi-resolution clustering of embeddings and manual tweaks produced the features in
criteria.md. Once fed to/agent-readiness-reportand run against a sample of repos from the initial collection, the scores in the report cleanly separated those that were AI ready from those that weren't.