add support for saving and restoring caches

Cargo.toml

@@ -45,6 +45,7 @@ futures = "0.3"
 futures-core = "0.3"
 getopts = "0.2"
 glob = "0.3"
+hex = "0.4.3"
 hiercmd = { git = "https://github.com/jclulow/hiercmd" }
 hmac-sha256 = "1"
 html-escape = "0.2"
@@ -84,6 +85,7 @@ slog-term = "2.7"
 smf = { git = "https://github.com/illumos/smf-rs.git" }
 strip-ansi-escapes = "0.2"
 strum = { version = "0.27", features = [ "derive" ] }
+tar = "0.4"
 tempfile = "3.3"
 thiserror = "2"
 tlvc = { git = "https://github.com/oxidecomputer/tlvc", version = "0.3.1" }
@@ -95,3 +97,4 @@ toml = "0.8"
 usdt = "0.6"
 uuid = { version = "1", features = [ "v4" ] }
 zone = { version = "0.3", features = [ "async" ], default-features = false }
+zstd = "0.13"

README.md

@@ -659,6 +659,78 @@ Configuration properties supported for basic jobs include:
     environment running in an ephemeral virtual machine, with a reasonable set
     of build tools.  32GB of RAM and 200GB of disk should be available.
 
+## Caching
+
+Buildomat has native support for caching intermediate artifacts across separate
+CI jobs, with both high-level integrations and a low-level manual CLI.
+
+Caches are identified by a "cache key", a string identifying the content being
+cached.  Buildomat only restores caches when the cache key requested by the job
+is the exact match of the key of an existing cache, and doesn't provide a
+way to restore a cache whose key is only a partial match.  High-level
+integrations will pick the most appropriate cache key automatically, while it's
+your responsibility to pick the correct cache key when using the manual CLI.
+
+Each Buildomat account or GitHub repository has its own isolated caching
+storage, to prevent unintentional cross-pollination.  While a cache with a given
+key exists on the Buildomat servers it won't be possible to upload a new cache
+with the same key.  Once the cache is removed for the server it will be possible
+to upload a cache with that key again.
+
+### Rust dependencies caching
+
+Buildomat includes native support for caching Rust dependencies, as part of the
+`bmat` CLI (pre-installed in every worker).  It saves and restores Cargo's
+`target/` directory, making sure to only cache third-party dependencies (as
+caching workspace members often has diminishing returns) and choosing the
+correct cache key.
+
+You can use the `bmat cache rust restore` and `bmat cache rust save` commands in
+your CI script:
+
+```bash
+#!/bin/bash
+bmat cache rust restore
+cargo build --locked
+bmat cache rust save
+```
+
+Both commands optionally accept the path of the `Cargo.toml` corresponding to
+the *workspace* to cache.  This is only needed if multiple separate workspaces
+are present in the same repository.  If no `Cargo.toml` is provided `bmat` will
+look in the current working directory for one, and fail if it's missing.
+
+### Manual caching
+
+If the files you need to cache are not covered by one of Buildomat's native
+integrations, you can use the `bmat` CLI (pre-installed in every worker) to
+manually save and restore caches. When using manual caching, it's your
+responsibility to pick the right cache key, and to choose which files should be
+included as part of the cache.
+
+The `bmat cache restore` command restores an existing cache, and requires the
+cache key as its first argument.
+
+The `bmat cache save` command stores files in the cache: it requires the cache
+key as its first argument, and the list of files to cache to the standard input
+(for example by piping `find -type f` into it).
+
+```bash
+#!/bin/bash
+#
+# This is a worse implementation of `bmat cache rust`.
+#
+
+rust_version="$(rustc --version | cut -d ' ' -f 2)"
+rust_host="$(rustc --print=host-tuple)"
+cargo_lock="$(sha256sum | cut -d ' ' -f 1)"
+cache_key="rust-$rust_version-$rust_host-$cargo_lock"
+
+bmat cache restore "$cache_key"
+cargo build --locked
+find target/ -type f | bmat cache save "$cache_key"
+```
+
 ## Licence
 
 Unless otherwise noted, all components are licenced under the [Mozilla Public

agent/Cargo.toml

@@ -18,14 +18,19 @@ bytes = { workspace = true }
 chrono = { workspace = true }
 futures = { workspace = true }
 glob = { workspace = true }
+hex = { workspace = true }
 hiercmd = { workspace = true }
 ipnet = { workspace = true }
 libc = { workspace = true }
+reqwest = { workspace = true }
 rusty_ulid = { workspace = true }
 serde = { workspace = true }
 serde_json = { workspace = true }
 slog = { workspace = true }
+tar = { workspace = true }
+tempfile = { workspace = true }
 tokio = { workspace = true }
+zstd = { workspace = true }
 #
 # I believe it is necessary to pull this in here, so that we can demand the
 # static linking of the vendored OpenSSL.  We don't use it directly, but the