Skip to content

Commit 541ca8b

Browse files
pablomendezroyoMarketen
pablomendezroyo
and
Marketen
authored
Add GitHub Actions documentation for staker and standard packages (#507)
* Add GitHub Actions documentation for staker and standard packages, including workflows and CI/CD overview * changes to intro (#508) --------- Co-authored-by: Marc Font <36164126+Marketen@users.noreply.github.com>
1 parent 5495f25 commit 541ca8b

File tree

9 files changed

+775
-19
lines changed

9 files changed

+775
-19
lines changed

‎docs/dev/github-actions/overview.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: "GitHub Actions for Dappnode Packages"
3+
sidebar_label: Overview
4+
llm_description: "Overview of GitHub Actions workflows for automating upstream updates, testing, and releases in Dappnode packages."
5+
---
6+
7+
# GitHub Actions for Dappnode Packages
8+
9+
GitHub Actions help automate workflows around your projects on GitHub. For Dappnode packages, we utilize specific GitHub Actions to manage updates, testing, and releases.
10+
11+
## Types of Dappnode Packages
12+
13+
Dappnode packages fall into two main categories, each with different CI/CD requirements:
14+
15+
### Standard Packages
16+
17+
Standard packages (e.g., Rotki, IPFS, etc.) use a simpler CI workflow that includes:
18+
19+
- **Upstream version checking**: Automatically detects new upstream releases
20+
- **Build and pre-release**: Builds packages and creates pre-releases on PRs and pushes
21+
22+
See [Standard Packages CI](./standard-packages.md) for detailed workflow configurations.
23+
24+
### Staker Packages
25+
26+
Staker packages require more complex CI that includes integration testing on real Dappnode hardware. These packages include:
27+
28+
- **Execution clients** (Geth, Reth, Nethermind, Besu, Erigon)
29+
- **Consensus clients** (Lodestar, Teku, Prysm, Nimbus, Lighthouse)
30+
- **Web3Signer**
31+
- **MEV-Boost**
32+
33+
The staker package CI runs on a GitHub self-hosted runner with Dappnode pre-installed and running with `DEV=true`, which enables the WebSocket frontend-backend RPC as an API. This allows the CI to use the same RPC calls a user would make to configure a staker setup.
34+
35+
See [Staker Packages CI](./staker-packages/overview.md) for detailed information about:
36+
37+
- [Execution Client Workflows](./staker-packages/execution-clients.md)
38+
- [Consensus Client Workflows](./staker-packages/consensus-clients.md)
39+
- [Web3Signer Workflows](./staker-packages/web3signer.md)
40+
41+
## Common Requirements
42+
43+
All Dappnode package workflows require:
44+
45+
- **`GITHUB_TOKEN`**: Automatically provided by GitHub for authentication
46+
- **`PINATA_API_KEY`** and **`PINATA_SECRET_API_KEY`**: For interfacing with the IPFS pinning service Pinata (standard packages)
47+
48+
Staker packages additionally require:
49+
50+
- Access to a self-hosted runner with Dappnode installed
51+
- Pre-synced execution client volumes for faster test execution

‎docs/dev/github-actions/staker-packages/consensus-clients.md‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
---
2+
title: "Consensus Client Workflows"
3+
sidebar_label: Consensus Clients
4+
llm_description: "GitHub Actions workflows for Dappnode consensus client packages including sync-test and release workflows."
5+
---
6+
7+
# Consensus Client Workflows
8+
9+
Consensus client packages (Lodestar, Teku, Prysm, Nimbus, Lighthouse) use three GitHub Actions workflows. Unlike execution clients, consensus clients **do not need a `sync.yml` workflow** because they use checkpoint sync, which doesn't require pre-synced volumes.
10+
11+
## Workflow Overview
12+
13+
| Workflow | Trigger | Purpose |
14+
| ---------------- | ---------------------- | ----------------------------- |
15+
| `auto_check.yml` | Scheduled (every 4h) | Check for upstream updates |
16+
| `sync-test.yml` | Pull requests + manual | Test package changes |
17+
| `release.yml` | Push to main + manual | Release with attestation test |
18+
19+
:::note
20+
Consensus clients don't have a `sync.yml` workflow because they use **checkpoint sync**. This allows them to sync quickly from a trusted checkpoint rather than from genesis, eliminating the need for pre-synced volumes.
21+
:::
22+
23+
## 1. Auto Check for Upstream Updates (`auto_check.yml`)
24+
25+
Same as standard packages - checks for upstream updates and creates PRs when new versions are available.
26+
27+
```yaml
28+
name: Bump upstream version
29+
30+
on:
31+
schedule:
32+
- cron: "00 */4 * * *"
33+
push:
34+
branches:
35+
- "master"
36+
37+
jobs:
38+
build:
39+
runs-on: ubuntu-latest
40+
steps:
41+
- uses: actions/checkout@v3
42+
- run: npx @dappnode/dappnodesdk github-action bump-upstream
43+
env:
44+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
45+
PINATA_API_KEY: ${{ secrets.PINATA_API_KEY }}
46+
PINATA_SECRET_API_KEY: ${{ secrets.PINATA_SECRET_API_KEY }}
47+
```
48+
49+
## 2. Sync Test (`sync-test.yml`)
50+
51+
Triggered on PRs to test package changes. Builds the consensus client package, uploads it to the local IPFS node, and runs a sync test paired with an execution client.
52+
53+
### Workflow File
54+
55+
```yaml
56+
name: Consensus Client Sync Test
57+
58+
on:
59+
workflow_dispatch:
60+
inputs:
61+
execution_client:
62+
description: "Execution Client"
63+
required: true
64+
type: choice
65+
options: [geth, reth, nethermind, besu, erigon]
66+
pull_request:
67+
branches:
68+
- "main"
69+
paths-ignore:
70+
- "README.md"
71+
72+
jobs:
73+
build:
74+
runs-on: staking-test-hoodi
75+
name: Build
76+
outputs:
77+
ipfs_hash: ${{ steps.extract_hash.outputs.ipfs_hash }}
78+
steps:
79+
- uses: actions/checkout@v6
80+
- name: Build and upload
81+
run: npx @dappnode/dappnodesdk build --provider=http://$(docker inspect DAppNodeCore-ipfs.dnp.dappnode.eth --format '{{.NetworkSettings.Networks.dncore_network.IPAddress}}'):5001 --variant=hoodi
82+
- name: Extract IPFS hash from releases.json
83+
id: extract_hash
84+
run: |
85+
# Get the last key's hash from releases.json
86+
IPFS_HASH=$(jq -r 'to_entries | last | .value.hash' package_variants/hoodi/releases.json)
87+
echo "ipfs_hash=$IPFS_HASH" >> $GITHUB_OUTPUT
88+
echo "Extracted IPFS hash: $IPFS_HASH"
89+
90+
sync-test:
91+
runs-on: staking-test-hoodi
92+
name: Consensus Client Sync Test
93+
needs: [build]
94+
steps:
95+
- name: Run sync
96+
run: |
97+
docker run --rm --pull=always --network dncore_network \
98+
-v /var/run/docker.sock:/var/run/docker.sock \
99+
-e MODE=sync \
100+
-e CONSENSUS_CLIENT='teku' \
101+
-e IPFS_HASH=${{ needs.build.outputs.ipfs_hash }} \
102+
-e EXECUTION_CLIENT=${{ github.event.inputs.execution_client }} \
103+
-e GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }} \
104+
-e GITHUB_REPOSITORY=${{ github.repository }} \
105+
-e GITHUB_PR_NUMBER=${{ github.event.pull_request.number }} \
106+
-e GITHUB_RUN_ID=${{ github.run_id }} \
107+
-e GITHUB_SERVER_URL=${{ github.server_url }} \
108+
ghcr.io/dappnode/staker-test-util/test-runner:latest
109+
```
110+
111+
### Key Differences from Execution Client Workflows
112+
113+
- **`CONSENSUS_CLIENT`** is set to the package being tested (e.g., `teku`)
114+
- **`EXECUTION_CLIENT`** is the input parameter for selecting which execution client to pair with
115+
- The IPFS hash refers to the consensus client package being tested
116+
117+
### PR Report
118+
119+
After the test completes, a report is posted to the PR including:
120+
121+
- Clients used and versions (before/after install)
122+
- Timing measurements for each operation
123+
- Container error logs (if any)
124+
- Link to full CI logs
125+
126+
## 3. Release (`release.yml`)
127+
128+
Triggered on push to main. Runs a full attestation test before releasing the package.
129+
130+
### Workflow File
131+
132+
```yaml
133+
name: "Release"
134+
135+
on:
136+
workflow_dispatch:
137+
inputs:
138+
execution_client:
139+
description: "Execution Client"
140+
required: true
141+
type: choice
142+
options: [geth, reth, nethermind, besu, erigon]
143+
push:
144+
branches:
145+
- "main"
146+
paths-ignore:
147+
- "README.md"
148+
149+
jobs:
150+
build:
151+
runs-on: staking-test-hoodi
152+
name: Build
153+
outputs:
154+
ipfs_hash: ${{ steps.extract_hash.outputs.ipfs_hash }}
155+
steps:
156+
- uses: actions/checkout@v6
157+
- name: Build and upload
158+
run: npx @dappnode/dappnodesdk build --provider=http://$(docker inspect DAppNodeCore-ipfs.dnp.dappnode.eth --format '{{.NetworkSettings.Networks.dncore_network.IPAddress}}'):5001 --variant=hoodi
159+
- name: Extract IPFS hash from releases.json
160+
id: extract_hash
161+
run: |
162+
# Get the last key's hash from releases.json
163+
IPFS_HASH=$(jq -r 'to_entries | last | .value.hash' package_variants/hoodi/releases.json)
164+
echo "ipfs_hash=$IPFS_HASH" >> $GITHUB_OUTPUT
165+
echo "Extracted IPFS hash: $IPFS_HASH"
166+
167+
test:
168+
name: Test
169+
runs-on: staking-test-hoodi
170+
needs: [build]
171+
steps:
172+
- uses: actions/checkout@v6
173+
- name: Run staker test runner
174+
run: |
175+
docker run --rm --pull=always \
176+
--network dncore_network \
177+
-v /var/run/docker.sock:/var/run/docker.sock \
178+
-e MODE=test \
179+
-e IPFS_HASH=${{ needs.build.outputs.ipfs_hash }} \
180+
-e CONSENSUS_CLIENT='teku' \
181+
-e EXECUTION_CLIENT=${{ github.event.inputs.execution_client }} \
182+
-e GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }} \
183+
-e GITHUB_REPOSITORY=${{ github.repository }} \
184+
-e GITHUB_PR_NUMBER=${{ github.event.pull_request.number }} \
185+
-e GITHUB_RUN_ID=${{ github.run_id }} \
186+
-e GITHUB_SERVER_URL=${{ github.server_url }} \
187+
ghcr.io/dappnode/staker-test-util/test-runner:latest
188+
189+
release:
190+
name: Release
191+
runs-on: ipfs-dev-gateway
192+
needs: [test]
193+
steps:
194+
- uses: actions/checkout@v6
195+
- uses: actions/setup-node@v6
196+
with:
197+
node-version: "22"
198+
- name: Publish
199+
run: npx @dappnode/dappnodesdk publish patch --content_provider=http://10.200.200.7:5001 --eth_provider=https://web3.dappnode.net --timeout 2h --all-variants
200+
env:
201+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
202+
DEVELOPER_ADDRESS: "0xf35960302a07022aba880dffaec2fdd64d5bf1c1"
203+
```
204+
205+
### Features
206+
207+
- **Proof of Attestation Test**: The test job uses `MODE=test` which:
208+
1. Syncs execution and consensus clients (consensus uses checkpoint sync)
209+
2. Imports validators into web3signer
210+
3. Waits for validators to become live on the beacon chain
211+
4. Confirms successful attestation
212+
- **Three-stage pipeline**: Build → Test → Release
213+
- **Different runners**: Test runs on `staking-test-hoodi`, release on `ipfs-dev-gateway`
214+
215+
## Checkpoint Sync
216+
217+
Consensus clients benefit from **checkpoint sync**, which allows them to:
218+
219+
1. Start syncing from a recent finalized checkpoint instead of genesis
220+
2. Sync in minutes rather than hours or days
221+
3. Avoid the need for pre-synced volumes on the self-hosted runner
222+
223+
This is why consensus client packages don't need a dedicated `sync.yml` workflow to maintain a synced state.
224+
225+
## Environment Variables
226+
227+
| Variable | Description |
228+
| ------------------- | ------------------------------------------------------------------- |
229+
| `MODE` | `sync` for sync only, `test` for full attestation test |
230+
| `CONSENSUS_CLIENT` | The consensus client name being tested (e.g., `teku`, `lighthouse`) |
231+
| `EXECUTION_CLIENT` | The execution client to pair with |
232+
| `IPFS_HASH` | The IPFS hash of the built package to test |
233+
| `NETWORK` | The network to use (e.g., `hoodi`) |
234+
| `GITHUB_TOKEN` | For PR commenting and GitHub API access |
235+
| `GITHUB_REPOSITORY` | Repository name for PR comments |
236+
| `GITHUB_PR_NUMBER` | PR number for posting reports |
237+
| `GITHUB_RUN_ID` | Run ID for linking to CI logs |
238+
| `GITHUB_SERVER_URL` | GitHub server URL for links |

0 commit comments

Comments
 (0)