Add UI test workflow for Renovate dependency-update PRs#179
Add UI test workflow for Renovate dependency-update PRs#179
Conversation
Co-authored-by: andre8244 <4612169+andre8244@users.noreply.github.com>
andre8244
force-pushed
the
copilot/add-ci-workflow-ui-tests
branch
from
ec3c3d0 to
af43452
Compare
andre8244
force-pushed
the
copilot/add-ci-workflow-ui-tests
branch
from
af43452 to
ce57756
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds an automated UI end-to-end testing pipeline intended to run on Renovate dependency-update PRs affecting the ui/ directory, using Robot Framework tests executed in a container and run on DigitalOcean-based CI infrastructure.
Changes:
- Added a GitHub Actions workflow to trigger UI tests for Renovate PRs that touch
ui/**. - Added a
ui/test-ui.shrunner script plus Robot Framework test suite and Python dependencies. - Updated
ui/.gitignoreto exclude Robot Framework output artifacts.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/test-renovate-ui.yml |
New workflow to gate execution to Renovate PRs and run UI tests via reusable DO-infra workflow. |
ui/test-ui.sh |
New containerized runner script to execute Robot Framework UI tests and collect outputs. |
ui/tests/test_ui.robot |
New Robot Framework suite to install a module, log into UI, take screenshots, and remove the module. |
ui/tests/pythonreq.txt |
Python dependencies for the Robot Framework test container. |
ui/.gitignore |
Ignore generated test output directory under ui/tests/outputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| podman run -i \ | ||
| --network=host \ | ||
| --volume=.:/home/pwuser/ns8-module:z \ |
There was a problem hiding this comment.
This script installs Python deps on every run but does not use the site-packages volume cache that test-module.sh uses, which will slow down CI and increase external dependency flakiness. Consider adding the same cached volume mount (or another cache mechanism) for pip install artifacts.
| --volume=.:/home/pwuser/ns8-module:z \ | |
| --volume=.:/home/pwuser/ns8-module:z \ | |
| --volume=./.pip-cache:/home/pwuser/.cache/pip:z \ |
| Suite Setup Connect to the node | ||
|
|
||
| *** Variables *** | ||
| ${SSH_KEYFILE} %{HOME}/.ssh/id_ecdsa |
There was a problem hiding this comment.
${SSH_KEYFILE} defaults to %{HOME}/.ssh/id_ecdsa, but ui/test-ui.sh defaults to id_rsa and CI passes SSH_KEYFILE explicitly via -v. To avoid local/CI mismatches, consider removing the hard-coded default here (rely on the passed-in variable) or aligning it with the script default.
| ${SSH_KEYFILE} %{HOME}/.ssh/id_ecdsa | |
| ${SSH_KEYFILE} %{HOME}/.ssh/id_rsa |
| robotframework | ||
| robotframework-sshlibrary | ||
| robotframework-browser |
There was a problem hiding this comment.
The test container installs Python dependencies from pythonreq.txt without pinning versions, meaning each run may pull arbitrary new releases of robotframework, robotframework-sshlibrary, or robotframework-browser from PyPI into an environment that has access to SSH credentials and your cluster. If any of these packages (or their transitive dependencies) are compromised in the future, the attack code would automatically execute in CI with the ability to exfiltrate secrets or tamper with test results. Pin these packages to specific, vetted versions (or hashes) to ensure only known-good artifacts are used in the test pipeline.
| ssh_key="$(< $SSH_KEYFILE)" | ||
|
|
||
| cleanup() { | ||
| set +e | ||
| podman cp rf-core-runner:/home/pwuser/outputs tests/ | ||
| podman stop rf-core-runner | ||
| podman rm rf-core-runner | ||
| } | ||
|
|
||
| trap cleanup EXIT | ||
|
|
||
| podman run -i \ | ||
| --network=host \ | ||
| --volume=.:/home/pwuser/ns8-module:z \ | ||
| --name rf-core-runner ghcr.io/marketsquare/robotframework-browser/rfbrowser-stable:19.11.0 \ | ||
| bash -l -s <<EOF | ||
| set -e | ||
| echo "$ssh_key" > /home/pwuser/ns8-key | ||
| pip install -q -r /home/pwuser/ns8-module/tests/pythonreq.txt |
There was a problem hiding this comment.
This script runs a third-party container image ghcr.io/marketsquare/robotframework-browser/rfbrowser-stable:19.11.0 with --network=host and passes in the private SSH key contents via ssh_key, giving that image direct access to secrets and your test infrastructure. Because the image is pinned only by a mutable tag and comes from an external organization, a compromised or hijacked image could exfiltrate the SSH key or tamper with tests without any integrity check. Pin this image to an immutable digest (and/or vendor it under your own namespace) and restrict secrets exposure so that only trusted, first-party images ever see private keys.
DavidePrincipi
left a comment
DavidePrincipi
left a comment
There was a problem hiding this comment.
Please check
- allowed branch name
- test output directory path
| workflow_dispatch: | ||
| pull_request: | ||
| branches: | ||
| - main |
There was a problem hiding this comment.
Workflow must run on renovate branches. IIRC this is the branch prefix:
| - main | |
| - renovate-* |
There was a problem hiding this comment.
Here we should put the name of the base branch, i.e. the branch we are merging into
Co-authored-by: Davide Principi <davide.principi@nethesis.it>
Co-authored-by: Davide Principi <davide.principi@nethesis.it>
This pull request introduces automated UI testing for Renovate PRs affecting the
uidirectory. It adds a GitHub Actions workflow that triggers on Renovate PRs, a shell script to run Robot Framework tests in a container, and the necessary test files and dependencies. The changes enable end-to-end testing of UI modules in a controlled environment.Automated UI testing workflow:
.github/workflows/test-renovate-ui.ymlto run UI tests automatically on Renovate PRs targeting theuidirectory. This workflow checks the PR author, builds/publishes images, gathers module info, and executes UI tests using DigitalOcean infrastructure.Test execution and environment setup:
ui/test-ui.shscript to run Robot Framework tests in a container, passing SSH credentials and module image information, and collecting test outputs.ui/tests/pythonreq.txtlisting Python dependencies for Robot Framework and its libraries.Test suite and outputs:
ui/tests/test_ui.robotcontaining Robot Framework test cases for module installation, UI login, screenshot capture, and module removal..gitignoreto exclude test output files (tests/outputs).