Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 38 additions & 14 deletions .github/workflows/refresh-profile.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,20 @@
name: Update Profile README
name: Update Profile and Deploy Pages

on:
schedule:
- cron: '0 5 * * *'
workflow_dispatch:

permissions:
contents: read
concurrency:
group: pages
cancel-in-progress: false

jobs:
refresh:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
steps:
- uses: actions/checkout@v4

Expand All @@ -23,29 +27,49 @@ jobs:

- run: uv sync --all-groups --frozen

- name: Update README
env:
GITHUB_TOKEN: ${{ secrets.SCORE_BOT_PAT }}
run: uv run generate-profile-readme
- name: Restore snapshot cache
uses: actions/cache@v4
with:
path: profile/cache/repo_overview.json
key: snapshot-${{ github.run_id }}
restore-keys: snapshot-

- name: Collect metrics
env:
GITHUB_TOKEN: ${{ secrets.SCORE_BOT_PAT }}
run: uv run python scripts/collect_metrics.py
run: uv run generate-repo-overview collect

- name: Render overview (MD)
run: uv run generate-repo-overview render-overview

- name: Create Pull Request
uses: peter-evans/create-pull-request@v7
with:
title: Update profile README and metrics
title: Update profile README
author: eclipse-score-bot <187756813+eclipse-score-bot@users.noreply.github.com>
committer: eclipse-score-bot <187756813+eclipse-score-bot@users.noreply.github.com>
body: |
This PR updates:
- The repository descriptions and grouping in `profile/README.md`
- The cross-repo metrics report in `profile/metrics.md`
Please review and merge if everything looks good.
commit-message: "chore: auto-refresh profile readme and metrics"
Auto-generated update of `profile/README.md` from the latest snapshot.
The HTML metrics dashboard is deployed separately to GitHub Pages.
commit-message: "chore: auto-refresh profile readme"
base: main
branch: bot/status-update
env:
GITHUB_TOKEN: ${{ secrets.SCORE_BOT_PAT }}

- name: Render details (HTML)
run: uv run generate-repo-overview render-details

- uses: actions/upload-pages-artifact@v5
with:
path: _site

deploy:
needs: refresh
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
steps:
- id: deployment
uses: actions/deploy-pages@v5
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,8 @@ __pycache__/
*.py[cod]
.pytest_cache/
.ruff_cache/
/.codex
/profile/cache/bazel_registry_checkout/
/profile/cache/reference_integration_checkout/
/profile/cache/repo_overview.json
/_site/
37 changes: 37 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# AGENTS.md

Context file for AI coding assistants. See `docs/repo-overview-tool-design.md` for architecture, data model, and caching details.

## Quick reference

```sh
uv sync --all-groups --frozen # install deps
uv run generate-repo-overview collect # GitHub API → snapshot JSON
uv run generate-repo-overview render-overview # snapshot → profile/README.md
uv run generate-repo-overview render-details # snapshot → _site/ (index + per-repo pages)
uv run pytest # run tests
uv run ruff check src/ tests/ # lint
uv run basedpyright src/ # type check
```

## Key files for website work

```
src/generate_repo_overview/
metrics_html.py — HTML renderer (index + per-repo detail pages)
metrics_report.py — shared helpers: grouping, version comparison, badges
models.py — RepoEntry, RepoSnapshot, signal dataclasses
cli.py — render-details writes all pages from render_all_pages()
constants.py — default paths (DEFAULT_METRICS_HTML_OUTPUT = _site/)
tests/
test_cli_render.py — render output tests
```

## Website rendering notes

- No static site generator or template engine — pure Python string concatenation.
- CSS is inlined per page via the `CSS` constant in `_html_common.py`. Dark theme using CSS variables.
- `render_all_pages(snapshot)` returns `dict[str, str]` of relative path to HTML content.
- Index page: tabs, filters, sortable columns — all client-side JS in `_render_script()`.
- Detail pages (`<repo>/index.html`): static HTML, no JS.
- Repo name links on the index go to detail pages; GitHub links use a separate icon.
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
See [AGENTS.md](AGENTS.md) for project context, key files, and dev commands.
47 changes: 42 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

This repository hosts the start page when you visit the eclipse-score GitHub organization. It contains links to the Eclipse Score website, documentation, and other resources related to the Eclipse Score project.

The Python tool in this repo now acts as a small repo-overview generator: it collects a cached snapshot of organization metadata once, then renders multiple Markdown views from that shared snapshot.

## Development

Expand All @@ -11,18 +12,54 @@ Use `uv` to create a virtual environment and install the project dependencies:
uv sync --all-groups
```

To generate the organization profile README:
The CLI now has a built-in overview:

```sh
uv run generate-repo-overview
```

For a cache-only re-render of both built-in reports:

```sh
uv run generate-repo-overview render
```

For a fresh GitHub pull before rendering, run:

```sh
uv run generate-repo-overview collect
```

By default, `collect` now does a cache-aware refresh: it checks fast, high-level
repository state and reuses cached deep details for repositories whose default
branch SHA has not changed. Use this for regular updates.

For volatile repository metrics (open PRs/issues, release counters, and recent
activity), fast mode keeps a per-repository fetch timestamp and refreshes those
values automatically when they are older than 1 hour.

You can tune this freshness window with `REPO_OVERVIEW_VOLATILE_TTL_MINUTES`
(default: `60`).

If you need a full deep refresh for every repository, run:

```sh
uv run generate-repo-overview collect --deep
```
uv run generate-profile-readme

If you only want the profile README:

```sh
uv run generate-repo-overview generate-profile-readme
```

Category order and category descriptions are configured in
`src/profile_readme_generator/profile_readme_config.toml`. Pass
`src/generate_repo_overview/profile_readme_config.toml`. Pass
`--config /path/to/file.toml` to use a different config file.

The generator reads repository custom properties from GitHub and expects `GITHUB_TOKEN` to be set.
If `GITHUB_TOKEN` is not set, it falls back to `gh auth token`.
The generator reads repository custom properties from GitHub and expects `GITHUB_TOKEN` to be set. If `GITHUB_TOKEN` is not set, it falls back to `gh auth token`.

Architecture notes for the package live in [src/generate_repo_overview/README.md](src/generate_repo_overview/README.md). The broader design notes are in [docs/repo-overview-tool-design.md](docs/repo-overview-tool-design.md).

To run the local checks:

Expand Down
105 changes: 105 additions & 0 deletions docs/repo-overview-tool-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Repo Overview Tool Design

## Goals

- Collect GitHub organization data once and reuse it across multiple reports.
- Keep local iteration fast by rendering from a cached snapshot instead of re-querying GitHub on every run.
- Separate GitHub collection, content enrichment, and rendering so new views are easy to add.
- Extend the profile README workflow with cross-repo metrics — Markdown, HTML dashboard, and GitHub Pages deployment — using a shared snapshot.

## Architecture

The tool is split into three layers:

1. `collector/`
- Connects to GitHub.
- Loads active repositories and custom properties.
- Derives content-based signals such as `has_ci`, `has_lint_config`, `has_coverage_config`, `bazel_version`, and `referenced_by_reference_integration`.
- Writes and reads a local JSON snapshot cache.
2. `profile_readme.py`, `metrics_report.py`, `metrics_html.py` (with `_html_index.py`, `_html_detail.py`, `_html_common.py`)
- Render different views (Markdown and HTML) from the same normalized data model.
- Keep presentation decisions out of the collection layer.
3. `cli.py`
- Orchestrates cache-aware commands: `collect`, `render-overview`, and `render-details`.

## Data Model

The shared model lives in `models.py`.

- `RepoEntry` contains both grouping metadata and overview metrics.
- `RepoSnapshot` stores:
- schema version
- organization name
- generation timestamp
- normalized repositories

The snapshot is intentionally renderer-agnostic. It stores neutral values such as booleans and plain strings rather than Markdown-specific markers.

## Caching Strategy

The default cache file is `profile/cache/repo_overview.json`.

The cache is used in two ways:

- Render commands read the snapshot directly and never contact GitHub.
- Collection commands reuse content-derived signals for repositories whose default-branch SHA has not changed.

That means changing a template or report layout is a local-only operation, and refreshing the snapshot only re-fetches file-tree data for repositories whose content likely changed.

## Why The Tool Uses The GitHub API Instead Of Cloning Repositories

The current report set mainly needs:

- repository metadata
- custom properties
- release dates
- open pull request counts
- file-presence checks
- a few small file contents such as `.bazelversion`
- cloned shared metadata repositories such as `bazel_registry` and `reference_integration`

For those needs, API access is cheaper and simpler than cloning every repository.

The collector uses:

- repository metadata from the organization API
- repository trees to detect whether files or directories exist
- targeted file-content reads only when a detector needs a small file

Cloning remains a future option if the project later needs heavyweight analysis such as line counting, local static analysis, or parsing large groups of files.

## Command Surface

The generic entry point is:

```sh
uv run generate-repo-overview <command>
```

Built-in commands:

- `collect`
- Sync the cached snapshot from GitHub and write it to disk.
- Use `--deep` to force a full refresh for every repository instead of reusing cached signals for unchanged ones.
- `render-overview`
- Render the profile README from an existing snapshot.
- `render-details`
- Render the HTML metrics page from an existing snapshot.

The `collect` command always performs a sync. The render commands never contact GitHub.

## Extension Points

To add a new view:

1. Extend `RepoEntry` only if the new view needs new normalized data.
2. Add or update detectors in `collector/` if new collection logic is required.
3. Create a new renderer that accepts `RepoSnapshot` or `list[RepoEntry]`.
4. Add a CLI command that reads the cached snapshot and calls the renderer.

To add a new detector, prefer:

- tree-based file existence checks for simple presence signals
- targeted small-file reads for version or config parsing

Avoid coupling detectors directly to output format. The collector should produce plain data; the renderer should decide how that data is displayed.
9 changes: 9 additions & 0 deletions profile/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Welcome to eclipse-score

**[Metrics, Status, etc](https://eclipse-score.github.io/.github/)** — for all repositories in this organization.

This Github organization contains artifacts developed by the [Eclipse S-CORE Project](https://projects.eclipse.org/projects/automotive.score).

## Introduction to Eclipse S-CORE Project
Expand Down Expand Up @@ -32,6 +34,7 @@ Core S-CORE modules, libraries, and APIs.
| [communication](https://github.com/eclipse-score/communication) | Repository for the communication module LoLa |
| [config_management](https://github.com/eclipse-score/config_management) | Repository for config management |
| [feo](https://github.com/eclipse-score/feo) | Repository for the Fixed Order Execution (FEO) framework |
| [inc_security_crypto](https://github.com/eclipse-score/inc_security_crypto) | Incubation repository for Security & Cryptography feature |
| [kyron](https://github.com/eclipse-score/kyron) | Safe async runtime for Rust |
| [lifecycle](https://github.com/eclipse-score/lifecycle) | Repository for the lifecycle feature |
| [logging](https://github.com/eclipse-score/logging) | Repository for logging daemon |
Expand Down Expand Up @@ -93,6 +96,12 @@ Integration repositories for various systems and components.
| [rules_imagefs](https://github.com/eclipse-score/rules_imagefs) | Repository for Image FileSystem Bazel rules and toolchains definitions |
| [testing_tools](https://github.com/eclipse-score/testing_tools) | Repository for testing utilities |

#### testing

| Repository | Description |
|------------|-------------|
| [qnx_unit_tests](https://github.com/eclipse-score/qnx_unit_tests) | Infrastructure for running unit tests in QNX VMs |

#### Toolchains

Toolchain repositories for compilers, linters, and other development tools.
Expand Down
Loading
Loading