Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 37 additions & 7 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,24 @@ Standard Knowledge is a multi-language library for programmatically augmenting C
## Working Effectively

### Bootstrap, build, and test the repository

- `cargo build --verbose` -- builds all Rust components. Takes ~55 seconds. NEVER CANCEL. Set timeout to 120+ seconds.
- `cargo test --verbose` -- runs all Rust tests. Takes ~25 seconds. NEVER CANCEL. Set timeout to 60+ seconds.
- `cd py && uv run pytest` -- runs Python tests. Takes ~25 seconds. NEVER CANCEL. Set timeout to 60+ seconds.

### Install CLI tool

- `cargo install --path cli` -- installs the standard_knowledge CLI tool. Takes ~32 seconds. NEVER CANCEL. Set timeout to 90+ seconds.

### Run applications

Python library (interactive testing):

- `cd py && uv run python`

```python
import standard_knowledge

library = standard_knowledge.StandardsLibrary()
library.load_cf_standards()
library.load_knowledge()
Expand All @@ -27,13 +33,15 @@ print(standard.attrs()) # Note: use standard.unit, not standard.units
```

CLI usage:

- `standard_knowledge --help` -- show CLI help
- `standard_knowledge get air_pressure_at_mean_sea_level` -- get standard details
- `standard_knowledge get -f xarray air_pressure_at_mean_sea_level` -- get xarray format
- `standard_knowledge filter --search temp` -- search for standards
- `standard_knowledge filter --ioos-category Meteorology` -- filter by category

### Linting and formatting

- `cargo fmt --check` -- check Rust formatting (~0.6 seconds)
- `cargo fmt --all` -- apply Rust formatting
- `cargo clippy --all-targets --all-features` -- run Rust linting. Takes ~12 seconds. NEVER CANCEL. Set timeout to 30+ seconds.
Expand All @@ -42,16 +50,19 @@ CLI usage:
## Validation

Always manually validate changes by:

1. **Build and test**: Run both Rust and Python test suites after making changes
2. **CLI functionality**: Test CLI commands with real examples from the README
3. **Python library**: Test Python library functionality interactively
4. **Standards data**: Verify standards can be loaded and queried correctly
1. **CLI functionality**: Test CLI commands with real examples from the README
1. **Python library**: Test Python library functionality interactively
1. **Standards data**: Verify standards can be loaded and queried correctly

### Test CLI changes

- Update CLI tests: `TRYCMD=overwrite cargo test --package standard_knowledge_cli --test cmd` -- Takes ~11 seconds. NEVER CANCEL. Set timeout to 30+ seconds.
- For new CLI features, copy existing test files in `cli/tests/cmd/` and run the TRYCMD command

### Update standards

- Run `uv run --script utils/update_standards.py` to refresh CF Standards from official sources. Takes ~10 seconds.

## Repository Structure
Expand All @@ -62,14 +73,17 @@ Always manually validate changes by:
- **utils/**: Python utility scripts for maintaining standards data

### Key files and directories

- `core/standards/`: YAML files containing community knowledge for each standard
- `core/src/lib.rs`: Main Rust library entry point
- `cli/src/main.rs`: CLI application entry point
- `py/src/lib.rs`: Python bindings entry point
- `.pre-commit-config.yaml`: Pre-commit hooks configuration

### Standards Knowledge Structure

Each standard has a YAML file in `core/standards/<standard_name>.yaml` with:

- `ioos_category`: IOOS measurement category
- `long_name`: Human readable name
- `common_variable_names`: Alternative column/variable names
Expand All @@ -82,6 +96,7 @@ Each standard has a YAML file in `core/standards/<standard_name>.yaml` with:
## Common Tasks

### Build times (with 50% buffer for timeout recommendations)

- Rust build: ~55 seconds → Use 120+ second timeout
- Rust tests: ~25 seconds → Use 60+ second timeout
- CLI install: ~32 seconds → Use 90+ second timeout
Expand All @@ -90,18 +105,22 @@ Each standard has a YAML file in `core/standards/<standard_name>.yaml` with:
- CLI test updates: ~11 seconds → Use 30+ second timeout

### Project Workspace Structure

This is a Cargo workspace with three members:

- `core` (standard_knowledge): Core Rust library
- `cli` (standard_knowledge_cli): CLI application
- `py` (standard_knowledge_py): Python bindings

### Common CI/CD commands that must pass

- All pre-commit hooks (rustfmt, clippy, ruff, codespell, actionlint)
- `cargo build --verbose`
- `cargo test --verbose`
- `cd py && uv run pytest`

### Example repository root listing

```
.
├── .github/
Expand All @@ -117,7 +136,9 @@ This is a Cargo workspace with three members:
```

### Standards Library Usage Patterns

**Rust:**

```rust
use standard_knowledge::StandardsLibrary;
let mut library = StandardsLibrary::new();
Expand All @@ -126,28 +147,34 @@ library.load_knowledge();
```

**Python:**

```python
import standard_knowledge

library = standard_knowledge.StandardsLibrary()
library.load_cf_standards()
library.load_knowledge()
```

**CLI:**

```bash
standard_knowledge get <standard_name>
standard_knowledge filter --var <variable_name>
standard_knowledge qc config <standard> <region> <params>
```

### Adding new standards knowledge

1. Create or edit YAML file in `core/standards/<standard_name>.yaml`
2. Run tests to validate: `cargo test`
3. Update CLI tests if needed: `TRYCMD=overwrite cargo test --package standard_knowledge_cli --test cmd`
4. Always run linting before committing: `cargo fmt --all && cargo clippy`
1. Run tests to validate: `cargo test`
1. Update CLI tests if needed: `TRYCMD=overwrite cargo test --package standard_knowledge_cli --test cmd`
1. Always run linting before committing: `cargo fmt --all && cargo clippy`

### Building Pyodide/WASM wheels locally

Pyodide wheels require specific Rust and Pyodide versions due to Emscripten compatibility:

```bash
# Install the compatible Rust version
rustup install nightly-2025-01-20
Expand All @@ -156,11 +183,14 @@ rustup target add wasm32-unknown-emscripten --toolchain nightly-2025-01-20
# Build pyodide wheels with pinned Rust toolchain and Pyodide 0.27.7
RUSTUP_TOOLCHAIN=nightly-2025-01-20 CIBW_PYODIDE_VERSION=0.27.7 uvx cibuildwheel --platform pyodide py
```

**Why these version pins?**

1. **Rust nightly-2025-01-20**: Rust 1.87+ generates WASM code requiring `--enable-bulk-memory-opt` which older wasm-opt (bundled with Emscripten 3.1.58) doesn't support. See: https://github.com/PyO3/maturin/issues/2549
2. **Pyodide 0.27.7**: Pyodide 0.28+ uses Emscripten 4.0.9 which has a bug with Rust-mangled symbol exports. The fix is in Emscripten 4.0.22+ but Pyodide hasn't updated yet. See: https://github.com/emscripten-core/emscripten/issues/24825
1. **Pyodide 0.27.7**: Pyodide 0.28+ uses Emscripten 4.0.9 which has a bug with Rust-mangled symbol exports. The fix is in Emscripten 4.0.22+ but Pyodide hasn't updated yet. See: https://github.com/emscripten-core/emscripten/issues/24825

### Important Notes

- Python bindings automatically rebuild when Rust code changes
- CLI tests use trycmd for snapshot testing - update with `TRYCMD=overwrite`
- Standards data is embedded into binaries at compile time via `build.rs`
Expand Down
4 changes: 0 additions & 4 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,6 @@ updates:
- "/py"
schedule:
interval: "weekly"
groups:
cargo:
patterns:
- "*"
cooldown:
default-days: 7

Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/javascript.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ on:
branches: [ main ]
workflow_dispatch:

env:
FORCE_COLOR: 3

permissions:
contents: read

Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ on:
pull_request:
workflow_dispatch:

env:
FORCE_COLOR: 3

permissions:
contents: read

Expand Down Expand Up @@ -114,6 +117,11 @@ jobs:
if: matrix.os == 'windows-intel'
run: rustup target add i686-pc-windows-msvc

- name: Install uv
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0
with:
enable-cache: true

- name: Build wheels
uses: pypa/cibuildwheel@a0a973acdc9e7b7f8b04ac5c80e6883a5a102615 # v4.1.0
env:
Expand Down
43 changes: 43 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,49 @@ repos:
exclude: docs/mkdocs.yml
- id: debug-statements
- id: mixed-line-ending

- repo: https://github.com/abravalheri/validate-pyproject
rev: 4b2e70d08cb2ccd26d1fba73588de41c7a5d50b7 # frozen: v0.25
hooks:
- id: validate-pyproject
# The package lives in py/, not at the repo root
files: 'pyproject\.toml$'
additional_dependencies: ["validate-pyproject-schema-store[all]"]

- repo: https://github.com/python-jsonschema/check-jsonschema
rev: f805888065fdb6162e1f800e50bb9460cbd223d6 # frozen: 0.37.2
hooks:
- id: check-dependabot
- id: check-github-workflows

- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: '3b3f7c3f57fe9925356faf5fe6230835138be230' # frozen: v0.15.17
hooks:
- id: ruff-format
- id: ruff-check
args: ["--fix", "--show-fixes"]

- repo: https://github.com/rhysd/actionlint
rev: 914e7df21a07ef503a81201c76d2b11c789d3fca # frozen: v1.7.12
hooks:
- id: actionlint
files: ".github/workflows/"
args: ["-ignore", "SC1090", "-ignore", "SC2046", "-ignore", "SC2086", "-ignore", "SC2129", "-ignore", "SC2155"]

- repo: https://github.com/zizmorcore/zizmor-pre-commit
rev: 9257c6050c0261b8c57e712f632dc4a8010109a9 # frozen: v1.25.2
hooks:
- id: zizmor
files: "^\\.github"
args: [--persona=pedantic]

- repo: https://github.com/codespell-project/codespell
# Configuration for codespell is in .codespellrc
rev: 2ccb47ff45ad361a21071a7eedda4c37e6ae8c5a # frozen: v2.4.2
hooks:
- id: codespell

- repo: local
hooks:
- id: rustfmt
Expand All @@ -47,8 +66,32 @@ repos:
language: system
pass_filenames: false

- repo: https://github.com/adamchainz/blacken-docs
rev: "fda77690955e9b63c6687d8806bafd56a526e45f" # frozen: 1.20.0
hooks:
- id: blacken-docs
additional_dependencies: [black==24.*]

- repo: https://github.com/hukkin/mdformat
rev: 82912cdaea4fb830f751504486a7879c70526547 # frozen: 1.0.0
hooks:
- id: mdformat

- repo: https://github.com/biomejs/pre-commit
rev: "5e270eb5db26a98324ef2f4b96135fc342c5d197" # frozen: v2.4.16
hooks:
- id: biome-check
additional_dependencies: ["@biomejs/biome@2.1.1"]

- repo: https://github.com/scientific-python/cookie
rev: "8f0581cc50b14023b14a9c77e9b358738ba637e7" # frozen: 2026.06.18
hooks:
- id: sp-repo-review
name: repo-review (root)
entry: repo-review --ignore PY001,PY004,PY005,PP301,GH212,PC140,PC170
- id: sp-repo-review
name: repo-review (py)
entry: repo-review py

auto_update:
cooldown_days: 7
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# standard_knowledge

Programmatically augmenting CF Standards with IOOS operational knowledge.

```py
Expand Down Expand Up @@ -237,9 +238,10 @@ Best run with `./noxfile -s test_js`, but individual tests are listed in the [Ja
Unless a session is specified, Nox will run all but `update_standards`, `install_cli`, `release`, and `test_wasm_env` sessions.

Sessions (use `-s`):

- `update_standards` - Updates the CF standards.
- `install_cli` - Install the Rust CLI.
- `release` -- <patch|minor|major> - Bumps the version of the all the packages for a release.
- `release` -- \<patch|minor|major> - Bumps the version of the all the packages for a release.
- `py_test-<version>` - Test a Python version as specified in the Github actions matrix.
- `py_wheel` - Build Linux wheels for currently supported Python versions.
- `py_wheel_wasm` - Build 3.14 WASM wheel.
Expand Down
2 changes: 1 addition & 1 deletion py/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name = "standard_knowledge_py"
version.workspace = true
license.workspace = true
repository.workspace = true
readme = "Readme.md"
readme = "README.md"
description.workspace = true
edition.workspace = true
rust-version.workspace = true
Expand Down
2 changes: 2 additions & 0 deletions py/Readme.md → py/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,9 @@ Note: [cibuildwheel only uses official builds](https://github.com/pypa/cibuildwh
`../noxfile.py -s wheel` should run that as well.

### Building Pyodide/WASM wheels locally

Pyodide wheels require specific Rust and Pyodide versions due to Emscripten compatibility:

```bash
# Install the compatible Rust version
rustup install nightly-2025-01-20
Expand Down
Loading
Loading