Design proposal: S3 as optional primary storage backend (future work, server-ng-targeted)

[jayakasadev]

Status

This is a design proposal for future work targeting core/server-ng once cluster + consensus have landed. Not a request to implement against core/server.

Filed at the suggestion of @hubcio in #3228: "a feature this size should land as a GitHub Discussion before an issue + stacked PRs." PRs #3226 / #3227 are closed; this Discussion is the design-level reset.

Goal

Allow iggy to run with an S3-compatible object store as the only persistence medium — no local data files. Useful for ephemeral / scale-to-zero compute, durable-by-default archive, and deployments where local NVMe provisioning is the bottleneck. Local-disk mode remains the default; S3 mode is opt-in via config and behind a default-off cargo feature.

Why now (and why "future work")

core/server is being replaced by core/server-ng (VSR-based). Landing a 10-phase persistence rewrite into core/server ships into actively-deprecated code. So this proposal targets the post-server-ng codebase, after consensus + cluster have merged. Filing the design now gives the core team time to weigh in on shape so implementation isn't bottlenecked on design when server-ng is ready.

High-level shape

A small ObjectStorage trait abstracts the persistence subsystem:

#[async_trait(?Send)]
pub trait ObjectStorage: Debug {
    async fn put(&self, key: &str, bytes: Bytes) -> Result<(), IggyError>;
    async fn put_if_absent(&self, key: &str, bytes: Bytes) -> Result<(), IggyError>;
    async fn put_multipart(&self, key: &str) -> Result<Box<dyn MultipartHandle>, IggyError>;
    async fn get_range(&self, key: &str, range: Range<u64>) -> Result<Bytes, IggyError>;
    async fn head(&self, key: &str) -> Result<ObjectMeta, IggyError>;
    async fn list_prefix(&self, prefix: &str) -> Result<Vec<ObjectMeta>, IggyError>;
    async fn delete(&self, key: &str) -> Result<(), IggyError>;
    async fn delete_many(&self, keys: &[&str]) -> Result<(), IggyError> { /* default per-key */ }
}

Three backends:

• CompioFsStorage — passthrough to compio fs (current behavior)
• S3Storage — built on rusty-s3 (sans-IO SigV4) + cyper (compio HTTP, rustls TLS), behind the object-storage cargo feature
• InMemoryStorage — cfg(test), HashMap-backed (replaces what opendal::services::Memory would have given us)

No tokio. No opendal. Compio-native end-to-end so it doesn't undo the work in #2020 / #2247.

Phase 0 spike (already run, against real AWS S3)

A throwaway feasibility spike validated the rusty-s3 + cyper + compio + rustls combination against real AWS S3 in an ephemeral bucket (us-east-1, 1-day lifecycle backstop, torn down on exit). 5/5 scenarios passed:

Scenario	Latency
PUT 1 KiB	108 ms
Range-GET 256 B	33 ms
Multipart 12 MiB upload (3 parts)	1555 ms
Full GET + byte-compare 12 MiB	919 ms
Conditional PUT race (If-None-Match: *)	62 ms; loser fenced cleanly with HTTP 412

Three correctness findings:

1. rustls::crypto::ring::default_provider().install_default() is required at boot when cyper is configured default-features = false.
2. AWS ETags arrive wrapped in quotes; rusty-s3's complete_multipart_upload re-wraps them, so callers must .trim_matches('"')-strip before passing back. Otherwise: 400 InvalidPart.
3. Multipart minimum part size is 5 MiB except the final part — typical iggy sub-MiB flushes need a buffering layer (BufferedMultipartWriter) to coalesce them into legal parts.

Open questions for server-ng integration

The trait + S3 backend + buffering + credential chain are backend-agnostic. The questions are about where they plug in:

1. Persistence model. Does server-ng keep a per-shard local log + segments + state log, or does VSR's replicated log replace the state log entirely?
2. State log. The proposal had a journal+snapshot model on object storage (since S3 has no append). With VSR, is the state log even a per-replica concept? Or does the consensus layer own it?
3. Segment writes / reads. Where does the MessagesWriter / MessagesReader shape live in server-ng? Is the multipart-during-write model still the right fit, or does VSR's commit model change the durability story?
4. Manifest model. The proposal had a per-partition versioned manifest for fast boot. With VSR, the consensus state may already enumerate segments — manifest redundant?
5. Fencing. Original Phase 8 used S3 conditional PUT for split-brain safety. VSR view-changes provide single-writer-per-view by construction. Probably retires entirely in a server-ng world. Does that match your view? (Side benefit: also retires the GCS-S3-compat limitation since there's no If-None-Match requirement anymore.)
6. ack_after_upload semantics. Producer ack waits for S3 PUT success. VSR adds a quorum-ack on top. Which is the user-visible commit point?
7. Trait ownership. Is Rc<dyn ObjectStorage> (per-shard) the right model for server-ng, or does it want Arc<dyn ObjectStorage> shared more broadly?

AWS credential chain (already designed)

Subset of the standard provider chain, no AWS SDK dep, all on cyper:

1. Inline config ([system.storage.object] access_key_id + secret_access_key)
2. Standard env vars (AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY + optional AWS_SESSION_TOKEN)
3. Container credentials (AWS_CONTAINER_CREDENTIALS_FULL_URI / RELATIVE_URI — covers ECS task roles + EKS Pod Identity)
4. IRSA via STS AssumeRoleWithWebIdentity (covers EKS IRSA)

Credential refresh + IMDSv2 deferred to a later "hardening" phase.

What I'd like from this Discussion

• Direction on which of the open questions above matter most
• Any shape constraints from the in-progress server-ng / VSR / cluster work that should inform the trait surface
• Whether the design as proposed is broadly the one you'd want, or whether server-ng wants a different shape
• A signal on when this work could realistically resume (i.e., when server-ng is the live target)

Happy to revise based on any of the above.

---

References to closed artifacts: PRs #3226 / #3227 and issue #3228 contain the original (server-targeted) design and Phase 0 spike findings.