[WIP] composefs-splitfdstream: new crate #228
Conversation
giuseppe
force-pushed
the
splitfdstream
branch
from
f59c77c to
eb7f1ef
Compare
| //! This module implements the JSON-RPC 2.0 protocol types used to communicate | ||
| //! with the containers-storage splitfdstream server. The protocol is line-delimited | ||
| //! JSON over Unix sockets with file descriptor passing via SCM_RIGHTS. |
There was a problem hiding this comment.
WDYT about https://github.com/cgwalters/jsonrpc-fdpass ? I spent a bit of time on that. I was thinking about transferring it either to github.com/composefs or github.com/bootc-dev.
| /// - `i64 LE >= 0`: external FD reference at index `value` | ||
| #[derive(Debug)] | ||
| pub struct SplitFDStreamReader { | ||
| cursor: Cursor<Vec<u8>>, |
There was a problem hiding this comment.
why wouldn't this accept any impl Read (or BufRead) per above
| @@ -0,0 +1,185 @@ | |||
| //! Binary stream format parser for splitfdstream. | |||
There was a problem hiding this comment.
I did a whole implementation of this over in https://github.com/containers/composefs-rs/pull/218/changes#diff-69fe9e0f1388b28cd8d358d8f76589aaf790016b57fca196c2d22e0f93012b9b - any reason not to unify?
There was a problem hiding this comment.
the version here is mostly vibe coded to get it working with the implementation in the storage library.
I guess we can get the other version work too with the c/storage version. Do you mind splitting the splitfdstream implementation from the rest of the PR?
There was a problem hiding this comment.
Do you mind splitting the splitfdstream implementation from the rest of the PR?
➡️ #229
crates/cfsctl/src/main.rs
Outdated
| digest: String, | ||
| /// Optional reference name for the layer | ||
| name: Option<String>, | ||
| /// Import from splitfdstream server socket instead of stdin |
There was a problem hiding this comment.
This one feels different enough to be a new verb
Add a new crate implementing the splitfdstream binary format, which enables streaming tar archives with external file references. The format uses an 8-byte LE signed prefix followed by optional data: - Negative prefix: inline data of size |prefix| - Non-negative prefix: external file descriptor reference by index Key types: - SplitfdstreamWriter: Build splitfdstreams with inline/external chunks - SplitfdstreamReader: Parse splitfdstreams back into chunks - SplitfdstreamTarReader: Read adapter that reconstructs byte streams This enables zero-copy tar reconstruction from containers-storage's tar-split metadata by streaming headers inline while referencing file content via fd. Assisted-by: OpenCode (Opus 4.5) Signed-off-by: Colin Walters <walters@verbum.org>
giuseppe
force-pushed
the
splitfdstream
branch
from
eb7f1ef to
4f80ec6
Compare
| /// | ||
| /// Uses libc recvmsg directly because rustix 1.x has a bug where | ||
| /// msg_controllen is set to 0, preventing SCM_RIGHTS reception. | ||
| #[allow(unsafe_code)] |
There was a problem hiding this comment.
Hmm really? Are you sure? We were successfully using rustix in https://github.com/cgwalters/jsonrpc-fdpass ...
(Also why are we not using that crate? I can publish it formally...ok just transferred it to the bootc-dev org along with the Go version too)
If there's fixes needed in rustix the maintainer is responsive.
We're actually very close to #[forbid(unsafe_code)] in this project, it's just #123
| let mut remaining = size; | ||
|
|
||
| while remaining > 0 { | ||
| let n = rustix::fs::copy_file_range( |
There was a problem hiding this comment.
I believe this optimization is already in std https://github.com/rust-lang/rust/blob/7057231bd78d6c7893f905ea1832365d4c5efe17/library/std/src/io/copy.rs#L54 so we can just use std::io::copy
| repo.finalize_object_tmpfile(File::from(tmpfile), size) | ||
| } | ||
|
|
||
| fn read_fd_contents(fd: &OwnedFd, size: u64) -> Result<Vec<u8>> { |
There was a problem hiding this comment.
Why load the whole thing into memory instead of consistently using the O_TMPFILE path?
| } | ||
| let request = serde_json::json!({ | ||
| "jsonrpc": "2.0", | ||
| "method": "GetSplitFDStream", |
There was a problem hiding this comment.
I think this is fine for PoC however bigger picture I think what we need to do is add this method to https://github.com/containers/skopeo/blob/main/cmd/skopeo/proxy.go and then wrap it with https://github.com/bootc-dev/containers-image-proxy-rs/ right?
(And maybe a side quest in there is "port proxy to jsonrpc-fdpass" as new API)
There was a problem hiding this comment.
Or in other words... look at what I did in #218 the key thing I think here is fast-pathing the containers-storage: transport using this.
I think in order to do that we really do need to have not just an "import a single layer" as a CLI verb but do the whole "fetch manifest+config + layers" right?
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Add a client library for communicating with the containers-storage splitfdstream server. This enables importing OCI container layers via UNIX socket with file descriptor passing. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
giuseppe
force-pushed
the
splitfdstream
branch
from
4f80ec6 to
9d250d2
Compare
2 participants
Add a client library for communicating with the containers-storage splitfdstream server. This enables importing OCI container layers via UNIX socket with file descriptor passing.
Needs: containers/container-libs#651