From 663ceffd93cb6b891226859b10e39a9255ac29ee Mon Sep 17 00:00:00 2001 From: Rhuan Barreto Date: Fri, 27 Mar 2026 12:32:47 +0100 Subject: [PATCH 1/2] docs: add archgate opportunity score product plan Document the full product and implementation plan for a repository analysis experience that estimates governance friction and potential Archgate impact. Made-with: Cursor --- .../archgate-repo-opportunity-score-plan.mdx | 210 ++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx diff --git a/docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx b/docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx new file mode 100644 index 00000000..00f7031e --- /dev/null +++ b/docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx @@ -0,0 +1,210 @@ +--- +title: "Plan: Archgate Repository Opportunity Score" +description: "Product and implementation plan for a web experience that evaluates repository governance friction and estimates Archgate impact." +--- + +# Plan: Archgate Repository Opportunity Score + +## Goal + +Build a public web experience that analyzes a GitHub repository and returns a clear **Archgate Opportunity Score** plus a prioritized ADR/rule starter pack. + +This should work as both: + +- a practical governance diagnostic for engineering teams, and +- a product-led growth funnel for Archgate adoption. + +## Product Outcome + +For a given repository, the system should provide: + +- `Opportunity Score` (0-100) with confidence level +- friction breakdown by dimension +- hidden-debt indicators (abandoned and stale PR patterns) +- projected impact from standardizing decisions via ADRs/rules +- recommended "first 5 ADRs" for that repository profile +- exportable report (markdown + JSON) + +## Design Principles + +- **Deterministic score core:** numeric score must be auditable and repeatable. +- **Agentic interpretation layer:** optional AI insights can explain patterns and propose ADRs. +- **Evidence-linked claims:** every recommendation links to specific PR evidence. +- **Explainable scoring:** each point in the score maps to explicit metric contributions. +- **Fast first value:** default run should complete in minutes on public repos. + +## Analysis Dimensions + +### 1) Review Friction + +- review events per PR +- p75/p90 merge latency +- percentage of PRs with high discussion volume + +### 2) Decision Debt + +- high-traction closed-unmerged PR share +- stale open PR share (14+ and 30+ days) +- stale PR discussion intensity + +### 3) Scope and Change Risk + +- oversized PR share (file count/churn thresholds) +- cross-domain PR complexity (frontend/backend mix) +- high-review-volume PR concentration + +### 4) Policy Gap Signals + +- recurring review themes: + - API contract ambiguity + - test expectation mismatches + - nullability/error-path issues + - permission/security guardrails + - UX behavioral invariants + +## Scoring Model (Deterministic Core) + +Recommended v1 weighted composition: + +- Review Friction: 30% +- Decision Debt: 25% +- Scope/Change Risk: 20% +- Policy Gap Signals: 25% + +Each component is normalized to 0-100 from bounded metric ranges. +Final score: + +`OpportunityScore = sum(weight_i * normalized_component_i)` + +### Confidence score + +Separate confidence metric based on: + +- PR sample size sufficiency +- recency and coverage window completeness +- bot/noise filtering quality +- missing-data rate + +## Agentic Layer (Optional, Non-Authoritative) + +Use agentic analysis to: + +- cluster and summarize review themes semantically +- generate repository-specific ADR drafts +- generate "likely root causes" narrative for high-friction cohorts + +Guardrails: + +- agent output never changes deterministic score directly +- every insight must include evidence links +- require minimum evidence threshold per claimed theme +- mark insights as "AI Interpretation" + +## Architecture + +## Data flow + +1. User submits `owner/repo`. +2. Backend collector fetches PR/review metadata via GitHub API. +3. Deterministic scorer computes normalized metrics and score. +4. Optional AI worker generates interpretation + ADR proposals. +5. Result persisted and served with shareable URL. + +## Components + +- **Frontend app:** score dashboard, evidence explorer, report export +- **API/backend:** ingestion orchestration and result retrieval +- **Scoring engine:** deterministic metric transforms +- **Insights engine:** agentic synthesis with evidence binding +- **Storage/cache:** run outputs keyed by repo and timestamp + +## MVP Scope + +### In + +- public repositories +- single-run analysis over last 90 days +- deterministic score + breakdown +- abandoned/stale/merged cohorts included +- recommended ADR starter pack + +### Out (later phases) + +- private repos (OAuth/GitHub App) +- org-wide aggregate analytics +- time-series tracking and alerts +- one-click ADR file generation into target repos + +## UX Flow + +1. Enter repository (`owner/repo`). +2. Run analysis (progress visible). +3. View score + confidence. +4. Expand each dimension to inspect evidence PRs. +5. View recommended ADR pack and expected impact. +6. Download report or copy share link. + +## Success Metrics + +Product metrics: + +- analysis completion rate +- report export/share rate +- trial-to-install conversion to Archgate +- % users who request ADR starter pack + +Diagnostic quality metrics: + +- score stability across reruns +- false-positive rate in theme detection +- user-rated recommendation usefulness + +## Implementation Phases + +### Phase 1: Scoring MVP + +- implement deterministic metric collector and scorer +- deliver web dashboard for one repo +- publish scoring methodology docs + +### Phase 2: Evidence + Recommendation Quality + +- add deeper evidence explorer +- add ADR recommendation mapping +- add confidence indicators and caveat messaging + +### Phase 3: Agentic Insights + +- semantic clustering of review discussions +- AI-generated ADR drafts with citations +- "what to standardize first" narratives + +### Phase 4: Growth and Scale + +- private repo support +- result persistence and trend views +- onboarding path into Archgate initialization + +## Risks and Mitigations + +- **Rate limits / API quotas:** caching, async job queueing, retry budget. +- **Noisy bot comments:** robust bot filtering and account classification. +- **Overclaiming ROI:** show ranges and assumptions, not single-point guarantees. +- **Sample bias:** explicitly report coverage and confidence. +- **Trust in AI insights:** strict evidence requirements and deterministic separation. + +## Deliverables + +- v1 scoring spec document (formula, thresholds, normalization) +- backend analysis service contract +- frontend score report UI +- ADR mapping catalog (theme -> recommended ADR templates) +- public methodology transparency page + +## Immediate Next Steps + +1. Freeze metric definitions and normalization ranges. +2. Implement deterministic scoring module as reusable package. +3. Build a minimal web UI with one sample repository run. +4. Validate score quality against known repos (including Sentry baseline). +5. Add optional AI insight generation behind a feature flag. From 997a63383197ecd568acd26a45be62ba23e38cb7 Mon Sep 17 00:00:00 2001 From: Rhuan Barreto Date: Fri, 27 Mar 2026 12:36:22 +0100 Subject: [PATCH 2/2] revert: remove opportunity score plan from cli docs The plan has been moved to archgate/internal and should not live in the public CLI repository. Made-with: Cursor --- .../archgate-repo-opportunity-score-plan.mdx | 210 ------------------ 1 file changed, 210 deletions(-) delete mode 100644 docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx diff --git a/docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx b/docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx deleted file mode 100644 index 00f7031e..00000000 --- a/docs/src/content/docs/studies/archgate-repo-opportunity-score-plan.mdx +++ /dev/null @@ -1,210 +0,0 @@ ---- -title: "Plan: Archgate Repository Opportunity Score" -description: "Product and implementation plan for a web experience that evaluates repository governance friction and estimates Archgate impact." ---- - -# Plan: Archgate Repository Opportunity Score - -## Goal - -Build a public web experience that analyzes a GitHub repository and returns a clear **Archgate Opportunity Score** plus a prioritized ADR/rule starter pack. - -This should work as both: - -- a practical governance diagnostic for engineering teams, and -- a product-led growth funnel for Archgate adoption. - -## Product Outcome - -For a given repository, the system should provide: - -- `Opportunity Score` (0-100) with confidence level -- friction breakdown by dimension -- hidden-debt indicators (abandoned and stale PR patterns) -- projected impact from standardizing decisions via ADRs/rules -- recommended "first 5 ADRs" for that repository profile -- exportable report (markdown + JSON) - -## Design Principles - -- **Deterministic score core:** numeric score must be auditable and repeatable. -- **Agentic interpretation layer:** optional AI insights can explain patterns and propose ADRs. -- **Evidence-linked claims:** every recommendation links to specific PR evidence. -- **Explainable scoring:** each point in the score maps to explicit metric contributions. -- **Fast first value:** default run should complete in minutes on public repos. - -## Analysis Dimensions - -### 1) Review Friction - -- review events per PR -- p75/p90 merge latency -- percentage of PRs with high discussion volume - -### 2) Decision Debt - -- high-traction closed-unmerged PR share -- stale open PR share (14+ and 30+ days) -- stale PR discussion intensity - -### 3) Scope and Change Risk - -- oversized PR share (file count/churn thresholds) -- cross-domain PR complexity (frontend/backend mix) -- high-review-volume PR concentration - -### 4) Policy Gap Signals - -- recurring review themes: - - API contract ambiguity - - test expectation mismatches - - nullability/error-path issues - - permission/security guardrails - - UX behavioral invariants - -## Scoring Model (Deterministic Core) - -Recommended v1 weighted composition: - -- Review Friction: 30% -- Decision Debt: 25% -- Scope/Change Risk: 20% -- Policy Gap Signals: 25% - -Each component is normalized to 0-100 from bounded metric ranges. -Final score: - -`OpportunityScore = sum(weight_i * normalized_component_i)` - -### Confidence score - -Separate confidence metric based on: - -- PR sample size sufficiency -- recency and coverage window completeness -- bot/noise filtering quality -- missing-data rate - -## Agentic Layer (Optional, Non-Authoritative) - -Use agentic analysis to: - -- cluster and summarize review themes semantically -- generate repository-specific ADR drafts -- generate "likely root causes" narrative for high-friction cohorts - -Guardrails: - -- agent output never changes deterministic score directly -- every insight must include evidence links -- require minimum evidence threshold per claimed theme -- mark insights as "AI Interpretation" - -## Architecture - -## Data flow - -1. User submits `owner/repo`. -2. Backend collector fetches PR/review metadata via GitHub API. -3. Deterministic scorer computes normalized metrics and score. -4. Optional AI worker generates interpretation + ADR proposals. -5. Result persisted and served with shareable URL. - -## Components - -- **Frontend app:** score dashboard, evidence explorer, report export -- **API/backend:** ingestion orchestration and result retrieval -- **Scoring engine:** deterministic metric transforms -- **Insights engine:** agentic synthesis with evidence binding -- **Storage/cache:** run outputs keyed by repo and timestamp - -## MVP Scope - -### In - -- public repositories -- single-run analysis over last 90 days -- deterministic score + breakdown -- abandoned/stale/merged cohorts included -- recommended ADR starter pack - -### Out (later phases) - -- private repos (OAuth/GitHub App) -- org-wide aggregate analytics -- time-series tracking and alerts -- one-click ADR file generation into target repos - -## UX Flow - -1. Enter repository (`owner/repo`). -2. Run analysis (progress visible). -3. View score + confidence. -4. Expand each dimension to inspect evidence PRs. -5. View recommended ADR pack and expected impact. -6. Download report or copy share link. - -## Success Metrics - -Product metrics: - -- analysis completion rate -- report export/share rate -- trial-to-install conversion to Archgate -- % users who request ADR starter pack - -Diagnostic quality metrics: - -- score stability across reruns -- false-positive rate in theme detection -- user-rated recommendation usefulness - -## Implementation Phases - -### Phase 1: Scoring MVP - -- implement deterministic metric collector and scorer -- deliver web dashboard for one repo -- publish scoring methodology docs - -### Phase 2: Evidence + Recommendation Quality - -- add deeper evidence explorer -- add ADR recommendation mapping -- add confidence indicators and caveat messaging - -### Phase 3: Agentic Insights - -- semantic clustering of review discussions -- AI-generated ADR drafts with citations -- "what to standardize first" narratives - -### Phase 4: Growth and Scale - -- private repo support -- result persistence and trend views -- onboarding path into Archgate initialization - -## Risks and Mitigations - -- **Rate limits / API quotas:** caching, async job queueing, retry budget. -- **Noisy bot comments:** robust bot filtering and account classification. -- **Overclaiming ROI:** show ranges and assumptions, not single-point guarantees. -- **Sample bias:** explicitly report coverage and confidence. -- **Trust in AI insights:** strict evidence requirements and deterministic separation. - -## Deliverables - -- v1 scoring spec document (formula, thresholds, normalization) -- backend analysis service contract -- frontend score report UI -- ADR mapping catalog (theme -> recommended ADR templates) -- public methodology transparency page - -## Immediate Next Steps - -1. Freeze metric definitions and normalization ranges. -2. Implement deterministic scoring module as reusable package. -3. Build a minimal web UI with one sample repository run. -4. Validate score quality against known repos (including Sentry baseline). -5. Add optional AI insight generation behind a feature flag.