Add ApptainerWorkspace implementation for rootless container support #892
Conversation
This commit implements ApptainerWorkspace, a container-based workspace that uses Apptainer (formerly Singularity) instead of Docker. This addresses the need for rootless container execution in HPC and shared computing environments where Docker may not be available or permitted. Key features: - No root privileges required for container execution - Converts Docker images to Apptainer SIF format with caching - Full RemoteWorkspace API compatibility - Automatic port management and health checking - Support for directory mounting and environment forwarding - Comprehensive documentation and examples Files added: - openhands-workspace/openhands/workspace/apptainer/workspace.py (implementation) - openhands-workspace/openhands/workspace/apptainer/__init__.py (module init) - openhands-workspace/openhands/workspace/apptainer/README.md (documentation) - examples/02_remote_agent_server/05_convo_with_apptainer_sandboxed_server.py (usage example) - tests/workspace/test_apptainer_workspace.py (test suite) - APPTAINER_WORKSPACE_TEST_LOG.md (test results and validation) Files modified: - openhands-workspace/openhands/workspace/__init__.py (export ApptainerWorkspace) Closes #891 Co-authored-by: openhands <openhands@all-hands.dev>
The ApptainerWorkspace implementation could not be tested end-to-end in the development environment because Apptainer is not installed. This commit adds transparency about testing limitations and provides clear guidance for users who want to test the implementation themselves. Changes: - Updated APPTAINER_WORKSPACE_TEST_LOG.md to explicitly state testing limitations - Added clear distinction between what was tested (code structure, types, API) and what requires Apptainer (runtime execution) - Added testing instructions to README.md for users with Apptainer installed - Clarified that validation focused on code correctness rather than runtime behavior This ensures users understand the implementation is structurally sound and type-correct, but requires Apptainer installation for full validation. Co-authored-by: openhands <openhands@all-hands.dev>
- Remove Docker dependency from _prepare_sif_image() - Use 'apptainer pull docker://image' instead of 'apptainer build ... docker-daemon://image' - This eliminates the need for Docker daemon, which is the main value of Apptainer - Remove unused imports (build, BuildOptions) - Add comprehensive test demonstrating Apptainer functionality - Successfully tested image pull and container execution - Document testing results and limitations Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
- Switch ApptainerWorkspace from instance mode to exec mode for better compatibility - Fix RemoteWorkspace to include API key in default HTTP client headers - Add authentication support via SESSION_API_KEY environment variable - Include demo log showing successful Apptainer workspace operation Co-authored-by: openhands <openhands@all-hands.dev>
Coverage Report •
|
||||||||||||||||||||||||||||||
Keep only the essential implementation and demo log as requested in issue. Co-authored-by: openhands <openhands@all-hands.dev>
ℹ️ Note on
|
Co-authored-by: openhands <openhands@all-hands.dev>
- Fix missing dependency that caused import errors for openhands.agent_server modules - Add assertion for cache_dir to help type checking - This allows ApptainerWorkspace to correctly import BuildOptions and related classes Co-authored-by: openhands <openhands@all-hands.dev>
|
[Automatic Post]: It has been a while since there was any activity on this PR. @neubig, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up. |
2 similar comments
|
[Automatic Post]: It has been a while since there was any activity on this PR. @neubig, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up. |
|
[Automatic Post]: It has been a while since there was any activity on this PR. @neubig, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up. |
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Add comprehensive documentation for ApptainerWorkspace, showing how to run agent servers in rootless Apptainer containers for HPC and shared computing environments. Includes: - When to use Apptainer vs Docker - Configuration options (pre-built image, base image, SIF file) - Key features and differences from Docker - Troubleshooting guide Relates to OpenHands/software-agent-sdk#892
|
The |
|
Fixed the The issue was that the docs branch name ( I've:
The check-examples workflow should now pass once GitHub Actions picks up the new branch. You may need to trigger a re-run of the workflow. |
Just to clarify, the docs check is not required for CI to pass. It’s just for humans or agents, to remind us 😅 |
|
@OpenHands We have a workflow which runs in CI, but it’s not required by CI for merge. It’s check-examples workflow. Find it and rename its human-facing title to “[Optional] Docs example / check-examples”. I mean, we want the visible name in CI on GitHub to signal clearly that it’s not a required job (for PR merge). Open a new branch from main and a new PR for this specific task, don’t mess with this PR. |
|
I'm on it! enyst can track my progress at all-hands.dev |
|
Summary of work completed What I changed
Branch and PR
Quality checks
Checklist against request
No behavioral or logic changes—purely a visible name update to clarify the job is optional. |
|
@OpenHands set up apptainer in CI and iterate until you have a test that demonstrates that this example passes: https://github.com/marketplace/actions/setup-apptainer |
Include both ApptainerWorkspace and OpenHandsCloudWorkspace imports. Co-authored-by: openhands <openhands@all-hands.dev>
|
@adityasoni9998 you were interested in this, could you try installing apptainer and running the example and see if it works properly? I think it should. |
- Fix test_apptainer_workspace_no_build_import to handle logging output - Guard __del__ and cleanup against interpreter shutdown Co-authored-by: openhands <openhands@all-hands.dev>
|
I tested this locally and it seems to work well! |
|
I have been able to successfully run the example A small problem I observe is that when I run this example, the TUI shows that the python interpreter is from my local machine though that should not be the case since apptainer has its own python env I guess? |
|
Hmm, OK, let me investigate. UPDATE: this seems like a legit bug that is also present in docker workspace. |
|
There is another super minor issue that may not be a bug but looks different than how docker runtimes used to behave. It seems that the owner of the files inside the remote apptainer workspace is copied from the username of the local machine I guess? For example, when I run ls -la inside the workspace using 
|
|
@OpenHands install apptainer locally and confirm+reproduce both of the issues pointed out by adityasoni9998. Write the logs demonstrating this issue to disk and tell me the location. Once you have done this, implement a fix. |
|
I'm on it! neubig can track my progress at all-hands.dev |
- Add --cleanenv to prevent host environment variables from leaking into container - Add --no-home to prevent mounting host home directory - Add --fakeroot option (configurable via use_fakeroot field) for consistent file ownership - Explicitly set PATH to container paths to ensure container's python is found first These changes address two issues reported by adityasoni9998: 1. TUI showing local python interpreter path instead of container's 2. File ownership showing local username instead of container user Co-authored-by: openhands <openhands@all-hands.dev>
SummaryI've analyzed and fixed the two issues reported by adityasoni9998 on PR #892 for ApptainerWorkspace. Issues Identified
Fix ImplementedModified
Log File LocationAnalysis log written to: Checklist
NoteThe |
Co-authored-by: openhands <openhands@all-hands.dev>
The tests were failing when port 8000 was in use because the fixture didn't mock the check_port_available function that's called during model_post_init. Co-authored-by: openhands <openhands@all-hands.dev>
|
I confirm that I have tested this PR by running this example with the default openhands-agent-server image |
There was a problem hiding this comment.
Can we merge this workflow as a job to tests.yml where we will run the normal unit tests?
There was a problem hiding this comment.
Related to
https://github.com/OpenHands/software-agent-sdk/pull/892/changes#r2644139374
Instead of keeping the test, we should add setup-apptainer to test-examples.yml, so we can run apptainer example when we tag a PR with test-examples
There was a problem hiding this comment.
ircc, we already have example 6, so this should be example 07
xingyaoww
left a comment
xingyaoww
left a comment
There was a problem hiding this comment.
Mostly LGTM! Happy to merge this PR once we can run this example script with test-examples tag
| # Report cost (must be before conversation.close()) | ||
| conversation.state._cached_state = ( | ||
| None # Invalidate cache to fetch latest stats | ||
| ) |
There was a problem hiding this comment.
This should not be necessary any more?
| # Report cost (must be before conversation.close()) | |
| conversation.state._cached_state = ( | |
| None # Invalidate cache to fetch latest stats | |
| ) |
There was a problem hiding this comment.
Please also create a documentation in OpenHands/docs repo, similar to our existing docs for DockerWorkspace
| def check_port_available(port: int) -> bool: | ||
| """Check if a port is available for binding.""" | ||
| import socket | ||
|
|
||
| sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) | ||
| try: | ||
| sock.bind(("0.0.0.0", port)) | ||
| return True | ||
| except OSError: | ||
| time.sleep(0.1) | ||
| return False | ||
| finally: | ||
| sock.close() | ||
|
|
||
|
|
||
| def find_available_tcp_port( | ||
| min_port: int = 30000, max_port: int = 39999, max_attempts: int = 50 | ||
| ) -> int: | ||
| """Find an available TCP port in a specified range.""" | ||
| import random | ||
|
|
||
| rng = random.SystemRandom() | ||
| ports = list(range(min_port, max_port + 1)) | ||
| rng.shuffle(ports) | ||
|
|
||
| for port in ports[:max_attempts]: | ||
| if check_port_available(port): | ||
| return port | ||
| return -1 | ||
|
|
There was a problem hiding this comment.
I believe we already have util function like this in this repo, please re-use those instead of repeatly define it. @OpenHands can you help find the relavant function name and how to import it? Do not edit the code yet
There was a problem hiding this comment.
I'm on it! xingyaoww can track my progress at all-hands.dev
There was a problem hiding this comment.
Looks like this example was not launching anything real on apptainer, rather we are mocking everything? Maybe let's remove it?
There was a problem hiding this comment.
Related to
https://github.com/OpenHands/software-agent-sdk/pull/892/changes#r2644139374
Instead of keeping the test, we should add setup-apptainer to test-examples.yml, so we can run apptainer example when we tag a PR with test-examples
|
Looks like there are a few issues preventing this PR from being merged!
If you'd like me to help, just leave a comment, like Feel free to include any additional details that might help me get this PR into a better state. You can manage your notification settings |
|
@OpenHands take a look at xingyaoww's comments and take a first pass at resolving them |
|
I'm on it! neubig can track my progress at all-hands.dev |
HUMAN: this has been tested
Description
This PR implements
ApptainerWorkspace, a container-based workspace that uses Apptainer (formerly Singularity) instead of Docker. This addresses the need for rootless container execution in HPC and shared computing environments where Docker may not be available or permitted.✨ Critical Bug Fix (2025-10-24): Discovered and fixed a bug where the initial implementation incorrectly used
apptainer build ... docker-daemon://image, which required Docker to be running. This defeated the entire purpose of Apptainer! The fix changes toapptainer pull docker://imagewhich pulls directly from Docker registries without needing Docker daemon. This is the key feature that makes Apptainer valuable.✨ Additional Fixes (2025-10-24):
apptainer execfor better compatibility in environments without systemd/FUSESESSION_API_KEYfrom environment and passes it to RemoteWorkspaceFixes #891
Key Features
apptainer pullImplementation Details
Files Added
openhands-workspace/openhands/workspace/apptainer/workspace.py(378 lines)ApptainerWorkspaceclass implementationapptainer pull(no Docker required!)apptainer execopenhands-workspace/openhands/workspace/apptainer/__init__.pyopenhands-workspace/openhands/workspace/apptainer/README.mdexamples/02_remote_agent_server/05_convo_with_apptainer_sandboxed_server.pytests/workspace/test_apptainer_workspace.pyapptainer_workspace_demo.logFiles Modified
openhands-workspace/openhands/workspace/__init__.pyApptainerWorkspaceto exportsopenhands-workspace/openhands/workspace/apptainer/workspace.pyapptainer pullinstead of Docker daemonapptainer execinstead of instance mode for better compatibilityopenhands-sdk/openhands/sdk/workspace/remote/base.pyUsage
Option 1: Pre-built Server Image (Recommended for HPC)
Option 2: Build from Base Image (Requires Docker for initial build)
Option 3: Use Existing SIF File
Testing
All tests pass successfully:
$ uv run pytest tests/workspace/test_apptainer_workspace.py -v tests/workspace/test_apptainer_workspace.py::test_apptainer_workspace_import PASSED [ 33%] tests/workspace/test_apptainer_workspace.py::test_apptainer_workspace_inheritance PASSED [ 66%] tests/workspace/test_apptainer_workspace.py::test_apptainer_workspace_field_definitions PASSED [100%] ============================== 3 passed in 0.13s ===============================End-to-End Testing with Actual Apptainer
Successfully tested the complete example with Apptainer 1.3.5. See
apptainer_workspace_demo.logfor full details:✅ Image Preparation
/root/.apptainer_cache/ghcr.io_openhands_agent-server_main-python.sif✅ Container Execution
apptainer execmode✅ Command Execution
✅ Authentication
✅ API Endpoints
/healthendpoint: ✅ Working/api/bash/start_bash_command: ✅ Working/api/conversations: ✅ Working (with auth)/api/conversations/{id}/run: ✅ Working (with auth)All pre-commit hooks pass:
Comparison: ApptainerWorkspace vs DockerWorkspace
Prerequisites
Users need to install Apptainer: https://apptainer.org/docs/user/main/quick_start.html
On Ubuntu/Debian:
Or build from source:
Why Apptainer?
As mentioned in issue #891, Docker requires root access which is often not available or permitted in:
Apptainer was specifically designed for these use cases and provides:
Technical Implementation Notes
Exec Mode vs Instance Mode
Initially implemented using Apptainer instance mode (
apptainer instance start), but discovered this requires systemd and/or FUSE which may not be available in all environments. Switched to direct execution mode (apptainer exec) which:Authentication Flow
ApptainerWorkspace discovers SESSION_API_KEY from environment and passes it to RemoteWorkspace, which now properly includes it in the HTTP client's default headers. This ensures all API requests (including conversation creation) are properly authenticated.
Demo Log
See
apptainer_workspace_demo.logfor the complete end-to-end test output showing:Checklist
DockerWorkspaceNext Steps
After merging, users can:
ApptainerWorkspaceas a drop-in replacement forDockerWorkspaceAgent Server images for this PR
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdknikolaik/python-nodejs:python3.12-nodejs22golang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:ccbf1d7-pythonRun
All tags pushed for this build
About Multi-Architecture Support
ccbf1d7-python) is a multi-arch manifest supporting both amd64 and arm64ccbf1d7-python-amd64) are also available if needed