feat: sev_verify test harness base#245
Conversation
There was a problem hiding this comment.
Pull request overview
Introduces an initial sev_verify Python package intended as a host-side SEV-SNP certification test harness, centered around TOML manifest discovery/parsing and a small set of starter data models and example test assets.
Changes:
- Added
sev_verifypackage entry points (__main__, CLI) and a first-pass manifest loader/discovery flow. - Introduced dataclass models for manifest definitions (tests/steps) and runtime results.
- Added a sample test module (
snphost_ok) and a starter certification manifest, plus Python-related.gitignoreentries.
Reviewed changes
Copilot reviewed 7 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| sev_verify/README.md | Documents intended usage/architecture and repository layout for the harness. |
| sev_verify/models.py | Adds dataclasses for steps/tests/certifications and runtime result objects. |
| sev_verify/cli.py | Implements argument parsing, manifest discovery, manifest loading, and basic printing. |
| sev_verify/cert_tests/common/snphost_ok.py | Adds an example host-side test module returning Step definitions. |
| sev_verify/cert_tests/common/init.py | Marks common as a Python package for test modules. |
| sev_verify/cert_tests/cert_3_0/manifest.toml | Adds a starter manifest for certification level 3.0. |
| sev_verify/cert_tests/cert_3_0/init.py | Marks cert_3_0 as a Python package. |
| sev_verify/cert_tests/init.py | Marks cert_tests as a Python package. |
| sev_verify/main.py | Adds python -m sev_verify entry point wiring to CLI main(). |
| sev_verify/init.py | Adds package docstring. |
| .gitignore | Adds standard Python build/cache ignores. |
Comments suppressed due to low confidence (1)
sev_verify/README.md:42
- The layout section says
results/output is gitignored, but there is noresults/entry in.gitignoreand the current code doesn’t create that directory yet. Either add an ignore pattern for the intended results path or adjust the documentation to avoid implying it’s already handled.
manifest.toml What to run
...
results/ Output (gitignored)
</details>
---
💡 <a href="/AMDEPYC/sev-certify/new/main?filename=.github/instructions/*.instructions.md" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Add Copilot custom instructions</a> for smarter, more guided reviews. <a href="https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Learn how to get started</a>.
| 1. Discover manifests at `cert_tests/*/manifest.toml`. Each manifest declares test entries (name, scope, module path). | ||
|
|
||
| 2. For each test, import its Python module from the same `cert_tests/<level>/` directory and call `steps()` to get the ordered list of `Step` objects. Steps specify a shell command, where it runs (host or guest), what constitutes success, and a timeout. | ||
|
|
||
| 3. Execute steps sequentially. Host steps run locally via subprocess. Guest steps are sent to the VM over a dedicated serial channel (`ttyS1`). For tests with `scope: guest` or `scope: mixed`, a QEMU SNP guest is launched before the first guest step and torn down after the last. | ||
|
|
||
| 4. Write results to `results/`. |
| """A test declared in the TOML manifest.""" | ||
|
|
||
| name: str | ||
| module: str # dotted module path, e.g. "cert_tests.common.snphost_ok" |
| print( | ||
| "Error: no manifest.toml found in cert_tests/*/", | ||
| file=sys.stderr, | ||
| ) |
| version = "3.0" | ||
| description = "SEV 3.0 Tests (AMD EPYC 7003+) - Current Level 3.0.0-0" |
There was a problem hiding this comment.
Are you/we planning to only maintain one certificate definition per generation?
Or what is your idea for the certificate definition structure?
I was looking at your command-line arguments and noticed that we would run certificates per CPU generation. I do like that approach.
My original intent was to go much more granular, something like:
-v 3.0.0-0 -v 3.1.2-0 -v 3.0.1-2
But I feel like that would require users to type too much if they want to run more than one test.
We’ll keep it as you proposed, but I’m curious how you envision the full manifest evolving as we add new versions across the different certificate generations.
There was a problem hiding this comment.
Or I guess, now that I read a little more, I think I would need an example on what a full manifest would look like.
There was a problem hiding this comment.
My original intent was to go much more granular, something like:
-v 3.0.0-0 -v 3.1.2-0 -v 3.0.1-2
This would definitely be useful, maybe even to a single test level. Let me work on adding handling for that. And yeah I'll flesh out some examples - I had the vague idea that each test would be its own python module, but didn't think about the sub-groupings of certificates. So good questions, I need to work with @ajcaldelas to get the certificate structure thought through & how we can populate everything.
DGonzalezVillal
left a comment
DGonzalezVillal
left a comment
There was a problem hiding this comment.
I like the manifest idea. I think I'm still a little unclear on what the manifest would look like and how the project would flow. Would the full cert-definitions be inside the manifest? Like the cert, test and steps would be written there?
First pass at a host-side testing harness for SEV certification. Reads TOML manifests that declare which tests to run, imports per-test Python modules that define executable steps, and orchestrates execution across host and guest environments.