Investigate and implement an optional auto-update mechanism for keypers

[ylembachar]

Description:
Explore and implement an optional, GitHub-aware auto-update mechanism for keypers. This would help node operators stay up-to-date with the latest releases, especially when releases include both image updates and configuration or script changes that cannot be handled by tools like Watchtower alone.

Goals

• Investigate how best to implement a lightweight, GitHub-aware auto-updater
• Select and implement the most practical approach for deployments using Docker Compose
• Ensure the solution is opt-in, safe, and easy to integrate

Context
Tools like Watchtower are useful for automatically pulling new Docker images, but they fall short for our release model because:

• Releases are GitHub-based, not solely Docker image–based
• Changes may include updates to .env, configuration scripts, or deployment logic
• Human intervention may still be needed for certain migrations

Proposed Solution
Implement a custom auto-update script that:

• Periodically checks the latest GitHub release in the shutter keyper deployment repo
• Compares it to the last applied version (stored locally)
• If new:
  • Logs or notifies the operator
  • Optionally runs the update instructions
  • Stores the updated version tag to prevent re-application

Deliverables

• A production-ready update script
• Update instructions for keyper operators who want to enable it
• Fallback instructions for manual override

[jannikluhn]

I suggest the following solution: An update script located in the deployment directory that

• checks if there are is a new tag pushed to GitHub,
• checks it can safely pull it without overriding any local changes,
• checks it isn't marked as non-auto-update (e.g., v0.1.2-manual),
• if so fetches it, and
• restarts the docker containers.

In addition, the script has CLI flags to register (and deregister) itself as a cronjob, so that it will run automatically, say, once a day.

The script is part of the repo, so that it would update itself. It would be configurable (via CLI flags or environment variables) to enable log pushing and to delay the update by a random amount of time (to prevent the whole network from updating at the same time).

Changes may include updates to .env, configuration scripts, or deployment logic

I don't think it's a good idea to modify .env files. I would rather make sure that most newly introduced config values are optional. If necessary, updates can be marked as manual via the tag so that users can perform the necessary modifications themselves.

I'm not sure what updated configuration scripts and deployment logic would entail. It's probably not possible to have an automatic but maximally generic update mechanism. We could add migration scripts that would be called by the updater script and can apply arbitrary changes for specific version jumps. However, I think it would be better to keep auto-updating simple and safe and rely on manual updates for more complicated changes.

[ulope]

@ylembachar, @jannikluhn and I had a call about this.

The outcome was that we shouldn't implement the solution as outlined above due to security / centralization concerns.

The main problem is that if an attacker gains access to our Github infrastructure (e.g. via a compromised user account that has permission, coerced employee, etc.) and enough Keypers have the auto update system enabled, the attacker will (relatively) instantly gain access to a threshold of Keypers and could perform any number of malicious actions (e.g. private key extraction)

We discussed various ways to mitigate this issue including:

• Utilizing the Github "Code Owners" feature to require multiple approvals for releases
• Signing releases
• Possibly adopting a system similar to DAppNode

Thos options need further research.

[ulope]

I just discovered this: https://slsa.dev

Could be useful...

[jannikluhn]

I looked into a few options a bit. GitHub doesn't have a multisig option: They allow a single admin to do pretty much anything, so we need another system. We can use it for publishing releases, but not to authenticate them. The same seems to be true for most other services that come to mind (like CI systems). Therefore, using cryptography and signing releases locally seems the way to go. Two options have been suggested: Ethereum keys and GPG keys.

GPG keys are standard for signing git commits. Using Ethereum keys for this purpose is a bit unusual (albeit used by DappNode), but would have the advantage that users might take more care of them. Also, addresses could be verified against some on-chain identity.

Irrespective of which keys we use, we need to define how to publish the signatures and how to distribute the public keys against which to verify the signatures. The public keys could either be included in a file in the repo or they could be manually managed by the user. In the former case, new releases could update the list which is convenient but might constitute a security risk. The latter is more secure, but makes it difficult for us the change keys.

For publishing the signatures, if we use GPG keys, we could use Git itself to publish the signatures, as commits and tags can be signed, however only by one key. So we would have to create multiple tags pointing to the same commit to authenticate a release. Alternatively, we could manually collect the signatures (Ethereum or GPG) in a file an put them in the repository.

[jannikluhn]

@ulope and I had a meeting about this and concluded that locally signing with GPG, collecting the signatures in a JSON file and uploading it as part of the release is the way to go. What we weren't sure of is what exactly needs to be signed: Definitely the release itself, but also potentially transitive dependencies like Rolling Shutter. Figuring this out should be the next step to proceed on this issue.

[pepae]

To mitigate centralization/control issues, we decided we want to implement a delay to the auto-update watchtower, giving keypers time to opt out.

[ylembachar]

Secure the release process in a following issue (to create)