diff --git a/doc/manual/source/SUMMARY.md.in b/doc/manual/source/SUMMARY.md.in index b1bf1c7da391..c75cbbb9a762 100644 --- a/doc/manual/source/SUMMARY.md.in +++ b/doc/manual/source/SUMMARY.md.in @@ -129,6 +129,7 @@ - [Store Path Specification](protocols/store-path.md) - [Nix Archive (NAR) Format](protocols/nix-archive/index.md) - [Nix Cache Info Format](protocols/nix-cache-info.md) + - [Binary Cache Bloom Filter Format](protocols/binary-cache-bloom-filter.md) - [Derivation "ATerm" file format](protocols/derivation-aterm.md) - [Nix32 Encoding](protocols/nix32.md) - [`builtins.wasm` Host Interface](protocols/wasm.md) diff --git a/doc/manual/source/protocols/binary-cache-bloom-filter.md b/doc/manual/source/protocols/binary-cache-bloom-filter.md new file mode 100644 index 000000000000..b27265c16bd2 --- /dev/null +++ b/doc/manual/source/protocols/binary-cache-bloom-filter.md @@ -0,0 +1,76 @@ +# Binary Cache Bloom Filter Format + +A [binary cache](@docroot@/package-management/binary-cache-substituter.md) may publish a Bloom filter of all store paths it contains. +The filter's URL is announced through the [`BloomFilter`](@docroot@/protocols/nix-cache-info.md#bloomfilter) field of the cache's [`nix-cache-info`](@docroot@/protocols/nix-cache-info.md) file — either as an absolute URL or as a path relative to the cache root. +A cache that does not advertise the field does not provide a Bloom filter; clients must not probe for one at a default path. + +A Bloom filter lets a client decide that a store path is **definitely not** in the cache without issuing a `.narinfo` request. +Membership tests are one-sided: a "not present" answer is authoritative, while a "possibly present" answer must still be confirmed by fetching the `.narinfo`. +False positives occur at a configurable rate; false negatives do not. + +MIME type: `application/octet-stream` + +## Format + +The response is binary, little-endian, with a fixed 32-byte header followed by the raw bit array: + +| Offset | Size | Field | Description | +|-------:|-----------:|-----------|----------------------------------------------------------| +| 0 | 8 | `magic` | ASCII bytes `NixBloom` (no terminating NUL). | +| 8 | 8 | `version` | `uint64` format version. Currently `1`. | +| 16 | 8 | `k` | `uint64` number of hash functions. | +| 24 | 8 | `m` | `uint64` size of the bit array, in bits. Multiple of 8. | +| 32 | `m / 8` | `bits` | The bit array. Bit at position `p` is `bits[p / 8] >> (p % 8)` masked with `1`. | + +The total response size is `32 + m / 8` bytes. + +## Membership test + +A client tests whether a store path *might* be in the cache as follows: + +1. Take the path's [hash part](@docroot@/protocols/store-path.md) — the first 32 [Nix32](@docroot@/protocols/nix32.md) characters of its base name. +2. Decode it into a 20-byte (160-bit) sequence using Nix32 decoding. +3. Read two 64-bit unsigned values from the decoded bytes, little-endian: + - `h1` from bytes `0..8` + - `h2` from bytes `8..16` + (The trailing 4 bytes are unused.) +4. For each `i` in `0, 1, …, k − 1`, compute the bit position + ``` + pos = ((h1 + i * h2) mod 2^64) mod m + ``` + The intermediate addition and multiplication wrap modulo 2^64 (standard unsigned 64-bit overflow) before the modulo by `m`. +5. If every `bits[pos / 8] >> (pos % 8)` has its low bit set, the path is *possibly* present; otherwise it is *definitely not* present. + +This is the standard Kirsch-Mitzenmacher double-hashing scheme. +Because a store path's hash part is already a cryptographic hash, no further hashing is required. + +## Server-side construction + +The server populates the filter by performing the same membership procedure for every valid store path and OR-ing in the resulting bits. + +Parameters are chosen from the count `n` of valid paths and a target false-positive rate `p`: + +``` +m = ceil(-n * ln(p) / (ln 2)^2), rounded up to a multiple of 8 +k = max(1, round((m / n) * ln 2)) +``` + +If `n` is zero, the server may emit a minimal filter (e.g., `m = 8`, `k = 1`, all bits zero), which correctly reports every query as "not present". + +The choice of `p` is server-defined and not advertised separately: a client can infer the asymptotic FPR from `m` and the number of paths in the cache, but does not need to in order to use the filter. + +## Caching + +The Bloom filter changes whenever the cache's path set changes. +Clients should refetch periodically; an HTTP cache lifetime on the order of minutes-to-hours is typically appropriate. + +## Example + +A cache containing roughly 500 000 paths, with a 1% target false-positive rate, produces a filter with `k = 7` and `m ≈ 4.7 × 10^6` bits — roughly 590 KB on the wire including the header. + +## See Also + +- [Nix Cache Info Format](@docroot@/protocols/nix-cache-info.md) +- [Store Path Specification](@docroot@/protocols/store-path.md) +- [Nix32 Encoding](@docroot@/protocols/nix32.md) +- [HTTP Binary Cache Store](@docroot@/store/types/http-binary-cache-store.md) diff --git a/doc/manual/source/protocols/nix-cache-info.md b/doc/manual/source/protocols/nix-cache-info.md index e8351e1cebe8..60ed0bfc9842 100644 --- a/doc/manual/source/protocols/nix-cache-info.md +++ b/doc/manual/source/protocols/nix-cache-info.md @@ -36,12 +36,27 @@ error: binary cache 'https://example.com' is for Nix stores with prefix '/nix/st Integer. Sets the default for [`priority`](@docroot@/store/types/http-binary-cache-store.md#store-http-binary-cache-store-priority). +### `BloomFilter` + +URL of a [Bloom filter](@docroot@/protocols/binary-cache-bloom-filter.md) that enumerates the store paths held by this cache. +Clients may use it to skip `.narinfo` requests for paths the filter rules out. + +The value is either an absolute URL or a path relative to the cache root: + +``` +BloomFilter: /bloom-filter +BloomFilter: https://filters.example.com/cache-abc.bloom +``` + +If absent, the cache does not publish a Bloom filter and clients must not assume one is available at any default location. + ## Example ``` StoreDir: /nix/store WantMassQuery: 1 Priority: 30 +BloomFilter: /bloom-filter ``` ## Caching Behavior diff --git a/src/libstore-tests/nar-info-disk-cache.cc b/src/libstore-tests/nar-info-disk-cache.cc index aebefc775675..7612250c661d 100644 --- a/src/libstore-tests/nar-info-disk-cache.cc +++ b/src/libstore-tests/nar-info-disk-cache.cc @@ -30,15 +30,16 @@ TEST(NarInfoDiskCacheImpl, create_and_read) // Set up "background noise" and check that different caches receive different ids { - auto bc1 = cache->createCache("https://bar", "/nix/storedir", wantMassQuery, prio); - auto bc2 = cache->createCache("https://xyz", "/nix/storedir", false, 12); + auto bc1 = + cache->createCache("https://bar", "/nix/storedir", {.wantMassQuery = wantMassQuery, .priority = prio}); + auto bc2 = cache->createCache("https://xyz", "/nix/storedir", {.priority = 12}); ASSERT_NE(bc1, bc2); barId = bc1; } // Check that the fields are saved and returned correctly. This does not test // the select statement yet, because of in-memory caching. - savedId = cache->createCache("http://foo", "/nix/storedir", wantMassQuery, prio); + savedId = cache->createCache("http://foo", "/nix/storedir", {.wantMassQuery = wantMassQuery, .priority = prio}); ; { auto r = cache->upToDateCacheExists("http://foo"); @@ -84,7 +85,7 @@ TEST(NarInfoDiskCacheImpl, create_and_read) } // "Update", same data, check that the id number is reused - cache2->createCache("http://foo", "/nix/storedir", wantMassQuery, prio); + cache2->createCache("http://foo", "/nix/storedir", {.wantMassQuery = wantMassQuery, .priority = prio}); { auto r = cache2->upToDateCacheExists("http://foo"); @@ -107,7 +108,8 @@ TEST(NarInfoDiskCacheImpl, create_and_read) auto r0 = cache2->upToDateCacheExists("https://bar"); ASSERT_FALSE(r0); - cache2->createCache("https://bar", "/nix/storedir", !wantMassQuery, prio + 10); + cache2->createCache( + "https://bar", "/nix/storedir", {.wantMassQuery = !wantMassQuery, .priority = prio + 10}); auto r = cache2->upToDateCacheExists("https://bar"); ASSERT_EQ(r->wantMassQuery, !wantMassQuery); ASSERT_EQ(r->priority, prio + 10); diff --git a/src/libstore/binary-cache-store.cc b/src/libstore/binary-cache-store.cc index a7f636f12311..7917a8604a79 100644 --- a/src/libstore/binary-cache-store.cc +++ b/src/libstore/binary-cache-store.cc @@ -14,11 +14,16 @@ #include "nix/util/signals.hh" #include "nix/util/archive.hh" #include "nix/util/util.hh" +#include "nix/util/users.hh" +#include "nix/store/bloom-filter.hh" +#include "nix/store/pathlocks.hh" #include +#include #include #include #include +#include #include #include @@ -68,11 +73,118 @@ void BinaryCacheStore::init() config.wantMassQuery.setDefault(value == "1"); } else if (name == "Priority") { config.priority.setDefault(std::stoi(value)); + } else if (name == "BloomFilter") { + bloomFilterUrl = value; } } } } +BinaryCacheStore::ConditionalGetResult +BinaryCacheStore::getFileConditional(const std::string & path, const std::string & /*expectedETag*/) +{ + /* Default: no ETag support; just do an ordinary fetch. */ + auto data = getFile(path); + return ConditionalGetResult{.data = std::move(data), .etag = "", .notModified = false}; +} + +bool BinaryCacheStore::fetchBloomFilter(const std::string & uri) +{ + /* Disable the Bloom filter for this cache for a short cooldown, so an + unavailable/broken filter doesn't cause a fetch on every query. */ + auto disable = [&] { + auto state(bloomState.lock()); + if (state->enabled) { + int t = 60; + debug("disabling Bloom filter for cache '%s' for %d seconds", uri, t); + state->enabled = false; + state->disabledUntil = std::chrono::steady_clock::now() + std::chrono::seconds(t); + } + return false; + }; + + auto expectedETag = diskCache->getBloomFilterETag(uri).value_or(""); + + /* `*bloomFilterUrl` can be a full (absolute) URL or a path relative to + the cache root; either way the resolution is done by `getFile()` / + `makeRequest()`, the same as for NAR URLs in `.narinfo` files. */ + ConditionalGetResult res; + try { + res = getFileConditional(*bloomFilterUrl, expectedETag); + } catch (Error & e) { + warn("failed to fetch Bloom filter from cache '%s': %s; disabling for now", uri, e.message()); + return disable(); + } + + if (res.notModified) { + debug("Bloom filter for '%s' unchanged (304 Not Modified)", uri); + diskCache->touchBloomFilter(uri, res.etag.empty() ? expectedETag : res.etag); + return true; + } + + if (!res.data) { + warn("Bloom filter at '%s' returned 404; disabling for now", uri); + return disable(); + } + + const auto & body = *res.data; + auto params = parseBloomFilterHeader(body); + if (!params || body.size() != bloomFilterHeaderLen + params->mBits / 8) { + warn("Bloom filter from cache '%s' is malformed; disabling for now", uri); + return disable(); + } + + diskCache->upsertBloomFilter(uri, res.etag, {reinterpret_cast(body.data()), body.size()}); + return true; +} + +bool BinaryCacheStore::isDefinitelyMissing(const StorePath & storePath) +{ + if (!diskCache || !bloomFilterUrl || !config.useBloomFilter) + return false; + + const auto uri = config.getReference().render(/*withParams=*/false); + + /* Per-process cooldown after a failed fetch, so an unavailable filter + doesn't cause a fetch on every query. */ + { + auto state(bloomState.lock()); + if (!state->enabled) { + if (std::chrono::steady_clock::now() < state->disabledUntil) + return false; + state->enabled = true; // cooldown elapsed; try again + } + } + + auto r = diskCache->probeBloomFilter(uri, storePath); + + if (!r) { + /* No fresh filter cached. Acquire a cross-process file lock so + concurrent first-probers don't all hit the network, then + re-check and fetch. */ + auto lockDir = getCacheDir() / "bloom-filter-locks"; + std::filesystem::create_directories(lockDir); + auto lockFile = + lockDir / hashString(HashAlgorithm::SHA256, uri).to_string(HashFormat::Base16, /*includePrefix=*/false); + PathLocks fetchLock( + {lockFile.string()}, fmt("waiting for another Nix process to fetch Bloom filter for '%s'...", uri)); + + r = diskCache->probeBloomFilter(uri, storePath); + if (!r) { + if (!fetchBloomFilter(uri)) + return false; + r = diskCache->probeBloomFilter(uri, storePath); + } + } + + if (!r) + return false; + + if (!*r) + debug("Bloom filter for '%s' ruled out '%s'", uri, printStorePath(storePath)); + return !*r; +} + std::optional BinaryCacheStore::getNixCacheInfo() { return getFile(cacheInfoFile); @@ -527,6 +639,8 @@ StorePath BinaryCacheStore::addToStoreFromDump( bool BinaryCacheStore::isValidPathUncached(const StorePath & storePath) { + if (isDefinitelyMissing(storePath)) + return false; // FIXME: this only checks whether a .narinfo with a matching hash // part exists. So ‘f4kb...-foo’ matches ‘f4kb...-bar’, even // though they shouldn't. Not easily fixed. @@ -580,6 +694,9 @@ void BinaryCacheStore::queryPathInfoUncached( auto callbackPtr = std::make_shared(std::move(callback)); try { + if (isDefinitelyMissing(storePath)) + return (*callbackPtr)({}); + auto uri = config.getReference().render(/*FIXME withParams=*/false); auto storePathS = printStorePath(storePath); auto act = std::make_shared( diff --git a/src/libstore/bloom-filter.cc b/src/libstore/bloom-filter.cc new file mode 100644 index 000000000000..90b4f1e6e279 --- /dev/null +++ b/src/libstore/bloom-filter.cc @@ -0,0 +1,67 @@ +#include "nix/store/bloom-filter.hh" +#include "nix/util/serialise.hh" + +#include + +namespace nix { + +std::optional parseBloomFilterHeader(std::string_view header) +{ + using namespace std::string_view_literals; + if (header.size() < bloomFilterHeaderLen || header.substr(0, 8) != "NixBloom"sv) + return std::nullopt; + + StringSource source(header.substr(8)); + uint64_t version; + uint32_t k; + uint64_t mBits; + try { + source >> version >> k >> mBits; + } catch (SerialisationError &) { + return std::nullopt; + } + + if (version != 1 || mBits == 0 || mBits % 8 != 0) + return std::nullopt; + + return BloomFilterParams{.k = k, .mBits = mBits}; +} + +std::string buildBloomFilter(const StorePathSet & paths, double falsePositiveRate) +{ + /* Rejects NaN as well, because all comparisons with NaN are false. */ + if (!(falsePositiveRate > 0 && falsePositiveRate < 1)) + throw Error("Bloom filter false positive rate must be between 0 and 1, got %f", falsePositiveRate); + + size_t n = paths.size(); + + uint64_t mBits = 8; + uint32_t k = 1; + if (n) { + constexpr double ln2 = 0.6931471805599453; + double mF = -double(n) * std::log(falsePositiveRate) / (ln2 * ln2); + /* `falsePositiveRate` very close to 1 makes `mF` round down to zero; + keep the floor of 8 bits so we never modulo by zero later. */ + mBits = std::max(8, ((uint64_t(std::ceil(mF)) + 7) / 8) * 8); + long kL = std::lround((double(mBits) / double(n)) * ln2); + k = uint32_t(std::max(1, kL)); + } + + StringSink sink(bloomFilterHeaderLen + mBits / 8); + + using namespace std::string_view_literals; + sink("NixBloom"sv); + sink << 1; // version + sink << k; + sink << mBits; + assert(sink.s.size() == bloomFilterHeaderLen); + + sink.s.resize(bloomFilterHeaderLen + mBits / 8); + char * bits = sink.s.data() + bloomFilterHeaderLen; + for (auto & path : paths) + forEachBloomBitPosition(path, k, mBits, [&](uint64_t pos) { bits[pos / 8] |= uint8_t(1) << (pos % 8); }); + + return std::move(sink.s); +} + +} // namespace nix diff --git a/src/libstore/http-binary-cache-store.cc b/src/libstore/http-binary-cache-store.cc index b3678ae4fdf1..8322b26f4ab7 100644 --- a/src/libstore/http-binary-cache-store.cc +++ b/src/libstore/http-binary-cache-store.cc @@ -69,13 +69,17 @@ void HttpBinaryCacheStore::init() if (auto cacheInfo = diskCache->upToDateCacheExists(cacheKey)) { config->wantMassQuery.setDefault(cacheInfo->wantMassQuery); config->priority.setDefault(cacheInfo->priority); + bloomFilterUrl = cacheInfo->bloomFilterUrl; } else { try { BinaryCacheStore::init(); } catch (UploadToHTTP &) { throw Error("'%s' does not appear to be a binary cache", config->cacheUri.to_string()); } - diskCache->createCache(cacheKey, config->storeDir, config->wantMassQuery, config->priority); + diskCache->createCache( + cacheKey, + config->storeDir, + {.wantMassQuery = config->wantMassQuery, .priority = config->priority, .bloomFilterUrl = bloomFilterUrl}); } } @@ -259,6 +263,28 @@ void HttpBinaryCacheStore::getFile(const std::string & path, Callbackdownload(request); + return ConditionalGetResult{ + .data = result.cached ? std::optional(std::string{}) + : std::optional(std::move(result.data)), + .etag = std::move(result.etag), + .notModified = result.cached, + }; + } catch (FileTransferError & e) { + if (e.error == FileTransfer::NotFound || e.error == FileTransfer::Forbidden) + return ConditionalGetResult{.data = std::nullopt, .etag = "", .notModified = false}; + maybeDisable(); + throw; + } +} + std::optional HttpBinaryCacheStore::getNixCacheInfo() { try { diff --git a/src/libstore/include/nix/store/binary-cache-store.hh b/src/libstore/include/nix/store/binary-cache-store.hh index 6a66e901a883..da22d58f549f 100644 --- a/src/libstore/include/nix/store/binary-cache-store.hh +++ b/src/libstore/include/nix/store/binary-cache-store.hh @@ -6,8 +6,10 @@ #include "nix/store/log-store.hh" #include "nix/util/pool.hh" +#include "nix/util/sync.hh" #include +#include namespace nix { @@ -66,6 +68,17 @@ struct BinaryCacheStoreConfig : virtual StoreConfig The meaning and accepted values depend on the compression method selected. `-1` specifies that the default compression level should be used. )"}; + + Setting useBloomFilter{ + this, + true, + "use-bloom-filter", + R"( + Whether to use the Bloom filter advertised by this binary cache (if + any) to avoid querying `.narinfo` files for store paths that are + definitely not in the cache. Set to `false` to disable this + optimization. + )"}; }; /** @@ -84,9 +97,39 @@ struct alignas(8) /* Work around ASAN failures on i686-linux. */ */ Config & config; + /** + * URL of the Bloom filter advertised by this cache (from the + * `BloomFilter:` field in `nix-cache-info`), as written by the server. + * Absolute URL or path relative to the cache root. `nullopt` if the + * cache doesn't advertise a Bloom filter. Populated by `init()` on + * the cold path or restored from the disk-cache by subclasses on the + * warm path. + */ + std::optional bloomFilterUrl; + private: std::vector> signers; + /** + * Per-process cooldown that suppresses Bloom filter use after a failed + * fetch, so we don't re-hit an unavailable filter on every query. Mirrors + * `HttpBinaryCacheStore::maybeDisable()`. + */ + struct BloomState + { + bool enabled = true; + std::chrono::steady_clock::time_point disabledUntil; + }; + + Sync bloomState; + + /** + * Fetch (with a conditional GET), validate, and store the Bloom filter in + * the disk cache. Returns false if the filter is unavailable/invalid (and + * disables it for a cooldown). Caller must hold the fetch lock. + */ + bool fetchBloomFilter(const std::string & uri); + protected: /** @@ -153,10 +196,43 @@ public: std::optional getFile(const std::string & path); + /** + * Result of a conditional HTTP-style GET. Returned by + * `BinaryCacheStore::getFileConditional`. + */ + struct ConditionalGetResult + { + /** Response body. Empty if `notModified`. `nullopt` if the file does not exist (404). */ + std::optional data; + + /** ETag returned by the server. Empty if no ETag was sent. */ + std::string etag; + + /** True if the server replied 304 Not Modified to our If-None-Match. */ + bool notModified = false; + }; + + /** + * Fetch a file with an HTTP-style conditional GET. The default + * implementation just forwards to `getFile()` (no ETag support). + * `HttpBinaryCacheStore` overrides this to use `If-None-Match` and + * to surface 304 responses. + */ + virtual ConditionalGetResult getFileConditional(const std::string & path, const std::string & expectedETag); + public: virtual void init() override; + /** + * Return true if this cache definitely does not contain `storePath`. + * Consults the Bloom filter advertised by the cache; lazily fetches + * and caches the filter on first call. Returns false in every other + * case (no filter advertised, filter disabled after a failure, + * filter says "possibly present"). Never throws. + */ + bool isDefinitelyMissing(const StorePath & storePath); + private: std::string narMagic; diff --git a/src/libstore/include/nix/store/bloom-filter.hh b/src/libstore/include/nix/store/bloom-filter.hh new file mode 100644 index 000000000000..4d3251bedc33 --- /dev/null +++ b/src/libstore/include/nix/store/bloom-filter.hh @@ -0,0 +1,69 @@ +#pragma once +///@file + +#include "nix/store/path.hh" +#include "nix/util/base-nix-32.hh" +#include "nix/util/util.hh" + +#include +#include +#include +#include +#include + +namespace nix { + +/** + * Size of the Bloom filter blob header: magic(8) + version(8) + k(8) + mBits(8). + * See `doc/manual/source/protocols/binary-cache-bloom-filter.md`. + */ +constexpr size_t bloomFilterHeaderLen = 8 + 8 + 8 + 8; + +/** + * The parameters of a Bloom filter, as encoded in its header. + */ +struct BloomFilterParams +{ + uint32_t k; + uint64_t mBits; +}; + +/** + * Parse and validate the `bloomFilterHeaderLen`-byte header at the start + * of a Bloom filter blob: magic `NixBloom`, version 1, `mBits != 0` and a + * multiple of 8. Returns `std::nullopt` if the header is too short or + * invalid. Does *not* check that the total body length matches `mBits`; + * the caller does that when it has the whole body. + */ +std::optional parseBloomFilterHeader(std::string_view header); + +/** + * Build a bloom-filter blob (`bloomFilterHeaderLen`-byte header + raw bit + * array, see `doc/manual/source/protocols/binary-cache-bloom-filter.md`) + * from a set of store paths. + */ +std::string buildBloomFilter(const StorePathSet & paths, double falsePositiveRate); + +/** + * Invoke `f(uint64_t pos)` for each of the `k` bit positions in an + * `mBits`-sized Bloom filter that correspond to `path`. + * + * Kirsch-Mitzenmacher double hashing over the 160 bits of the path's + * `hashPart`; intermediate arithmetic wraps modulo 2^64 before the + * final modulo by `mBits`. See + * `doc/manual/source/protocols/binary-cache-bloom-filter.md` for the + * full specification. + */ +template +void forEachBloomBitPosition(const StorePath & path, uint32_t k, uint64_t mBits, F && f) +{ + auto raw = BaseNix32::decode(std::string(path.hashPart())); + assert(raw.size() == 20); + auto * b = reinterpret_cast(raw.data()); + uint64_t h1 = readLittleEndian(b); + uint64_t h2 = readLittleEndian(b + 8); + for (uint32_t i = 0; i < k; ++i) + f((h1 + uint64_t(i) * h2) % mBits); +} + +} // namespace nix diff --git a/src/libstore/include/nix/store/http-binary-cache-store.hh b/src/libstore/include/nix/store/http-binary-cache-store.hh index 765eb6dd5135..e91d48a0a315 100644 --- a/src/libstore/include/nix/store/http-binary-cache-store.hh +++ b/src/libstore/include/nix/store/http-binary-cache-store.hh @@ -121,6 +121,8 @@ protected: void getFile(const std::string & path, Callback> callback) noexcept override; + ConditionalGetResult getFileConditional(const std::string & path, const std::string & expectedETag) override; + std::optional getNixCacheInfo() override; std::optional isTrustedClient() override; diff --git a/src/libstore/include/nix/store/meson.build b/src/libstore/include/nix/store/meson.build index 9900e64c67a0..d59fc3a5ce43 100644 --- a/src/libstore/include/nix/store/meson.build +++ b/src/libstore/include/nix/store/meson.build @@ -14,6 +14,7 @@ headers = [ config_pub_h ] + files( 'async-path-writer.hh', 'aws-creds.hh', 'binary-cache-store.hh', + 'bloom-filter.hh', 'build-result.hh', 'build/build-log.hh', 'build/derivation-builder.hh', diff --git a/src/libstore/include/nix/store/nar-info-disk-cache.hh b/src/libstore/include/nix/store/nar-info-disk-cache.hh index a30c5a553b96..505ec25792c7 100644 --- a/src/libstore/include/nix/store/nar-info-disk-cache.hh +++ b/src/libstore/include/nix/store/nar-info-disk-cache.hh @@ -5,6 +5,8 @@ #include "nix/store/nar-info.hh" #include "nix/store/realisation.hh" +#include + namespace nix { struct SQLiteSettings; @@ -25,16 +27,20 @@ struct NarInfoDiskCache virtual ~NarInfoDiskCache() {} - virtual int - createCache(const std::string & uri, const std::string & storeDir, bool wantMassQuery, int priority) = 0; - struct CacheInfo { - int id; - bool wantMassQuery; - int priority; + int id = 0; + bool wantMassQuery = false; + int priority = 0; + std::optional bloomFilterUrl; }; + /** + * Create or update the cached nix-cache-info for the binary cache at `uri`. + * Note that `info.id` is ignored. This function returns the id of the cache entry. + */ + virtual int createCache(const std::string & uri, const std::string & storeDir, const CacheInfo & info) = 0; + virtual std::optional upToDateCacheExists(const std::string & uri) = 0; virtual std::pair> @@ -48,6 +54,40 @@ struct NarInfoDiskCache virtual std::pair> lookupRealisation(const std::string & uri, const DrvOutput & id) = 0; + /** + * Probe `path` against the cached Bloom filter for `uri`. + * + * Returns `std::nullopt` if there is no Bloom filter cached for this + * cache, or the cached one is stale (older than the negative TTL) — the + * caller should (re)fetch and try again. Otherwise returns whether the + * filter says the path is *possibly present* (`true`) or *definitely not + * present* (`false`). + * + * The filter parameters (`k`, `mBits`) and the bits are read from the + * same stored blob in a single transaction, so they cannot drift. + */ + virtual std::optional probeBloomFilter(const std::string & uri, const StorePath & path) = 0; + + /** + * Store a freshly fetched Bloom filter blob (the full response body: + * header + bit array). + */ + virtual void + upsertBloomFilter(const std::string & uri, const std::string & etag, std::span blob) = 0; + + /** + * Refresh the timestamp (and optionally the etag) of an existing Bloom filter + * after a successful conditional GET returned 304 Not Modified. + */ + virtual void touchBloomFilter(const std::string & uri, const std::string & etag) = 0; + + /** + * Return the etag of the currently cached Bloom filter for `uri` + * (regardless of its age), or nullopt if none is cached or it has no + * etag. Used to send `If-None-Match` when refetching. + */ + virtual std::optional getBloomFilterETag(const std::string & uri) = 0; + /** * Return a singleton cache object that can be used concurrently by * multiple threads. diff --git a/src/libstore/meson.build b/src/libstore/meson.build index 47c7d20d59c4..10131ab495c5 100644 --- a/src/libstore/meson.build +++ b/src/libstore/meson.build @@ -295,6 +295,7 @@ sources = files( 'active-builds.cc', 'async-path-writer.cc', 'binary-cache-store.cc', + 'bloom-filter.cc', 'build-result.cc', 'build/build-log.cc', 'build/derivation-builder.cc', diff --git a/src/libstore/nar-info-disk-cache.cc b/src/libstore/nar-info-disk-cache.cc index 5c69561bebf0..5f73f8c55e52 100644 --- a/src/libstore/nar-info-disk-cache.cc +++ b/src/libstore/nar-info-disk-cache.cc @@ -1,6 +1,8 @@ #include "nix/store/nar-info-disk-cache.hh" +#include "nix/store/bloom-filter.hh" #include "nix/util/users.hh" #include "nix/util/sync.hh" +#include "nix/util/finally.hh" #include "nix/store/sqlite.hh" #include "nix/store/globals.hh" #include "nix/store/provenance.hh" @@ -20,7 +22,16 @@ create table if not exists BinaryCaches ( timestamp integer not null, storeDir text not null, wantMassQuery integer not null, - priority integer not null + priority integer not null, + bloomFilterUrl text -- NULL if the cache doesn't advertise a Bloom filter +); + +create table if not exists BloomFilters ( + cache integer primary key not null, + timestamp integer not null, + etag text, + blob blob not null, -- full filter body (header + bit array) + foreign key (cache) references BinaryCaches(id) on delete cascade ); create table if not exists NARs ( @@ -67,17 +78,16 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache struct Cache { - int id; std::string storeDir; - bool wantMassQuery; - int priority; + CacheInfo info; }; struct State { SQLite db; SQLiteStmt insertCache, queryCache, insertNAR, insertMissingNAR, queryNAR, insertRealisation, - insertMissingRealisation, queryRealisation, purgeCache; + insertMissingRealisation, queryRealisation, purgeCache, queryBloomFilterETag, insertBloomFilter, + touchBloomFilter, queryFreshBloomFilter; std::map caches; }; @@ -86,7 +96,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache NarInfoDiskCacheImpl( const Settings & settings, SQLiteSettings sqliteSettings, - std::filesystem::path dbPath = getCacheDir() / "binary-cache-detsys-v1.sqlite") + std::filesystem::path dbPath = getCacheDir() / "binary-cache-detsys-v2.sqlite") : NarInfoDiskCache{settings} { auto state(_state.lock()); @@ -101,11 +111,23 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache state->insertCache.create( state->db, - "insert into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority) values (?1, ?2, ?3, ?4, ?5) on conflict (url) do update set timestamp = ?2, storeDir = ?3, wantMassQuery = ?4, priority = ?5 returning id;"); + "insert into BinaryCaches(url, timestamp, storeDir, wantMassQuery, priority, bloomFilterUrl) values (?1, ?2, ?3, ?4, ?5, ?6) on conflict (url) do update set timestamp = ?2, storeDir = ?3, wantMassQuery = ?4, priority = ?5, bloomFilterUrl = ?6 returning id;"); state->queryCache.create( state->db, - "select id, storeDir, wantMassQuery, priority from BinaryCaches where url = ? and timestamp > ?"); + "select id, storeDir, wantMassQuery, priority, bloomFilterUrl from BinaryCaches where url = ? and timestamp > ?"); + + state->queryBloomFilterETag.create(state->db, "select etag from BloomFilters where cache = ?"); + + /* `>=` (not `>`) so a filter (re)fetched and stamped at the probe's + reference time still counts as fresh; see `probeBloomFilter`. */ + state->queryFreshBloomFilter.create( + state->db, "select rowid from BloomFilters where cache = ? and timestamp >= ?"); + + state->insertBloomFilter.create( + state->db, "insert or replace into BloomFilters(cache, timestamp, etag, blob) values (?, ?, ?, ?)"); + + state->touchBloomFilter.create(state->db, "update BloomFilters set timestamp = ?, etag = ? where cache = ?"); state->insertNAR.create( state->db, @@ -192,18 +214,20 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache if (!queryCache.next()) return std::nullopt; auto cache = Cache{ - .id = (int) queryCache.getInt(0), .storeDir = queryCache.getStr(1), - .wantMassQuery = queryCache.getInt(2) != 0, - .priority = (int) queryCache.getInt(3), - }; + .info = { + .id = (int) queryCache.getInt(0), + .wantMassQuery = queryCache.getInt(2) != 0, + .priority = (int) queryCache.getInt(3), + .bloomFilterUrl = queryCache.isNull(4) ? std::nullopt : std::optional(queryCache.getStr(4)), + }}; state.caches.emplace(uri, cache); } return getCache(state, uri); } public: - int createCache(const std::string & uri, const std::string & storeDir, bool wantMassQuery, int priority) override + int createCache(const std::string & uri, const std::string & storeDir, const CacheInfo & info) override { return retrySQLite([&]() { auto state(_state.lock()); @@ -214,32 +238,28 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache auto cache(queryCacheRaw(*state, uri)); if (cache) - return cache->id; + return cache->info.id; - Cache ret{ - .id = -1, // set below - .storeDir = storeDir, - .wantMassQuery = wantMassQuery, - .priority = priority, - }; + Cache ret{.storeDir = storeDir, .info = info}; { auto r(state->insertCache.use() .apply(uri) .apply(time(nullptr)) .apply(storeDir) - .apply(wantMassQuery) - .apply(priority)); + .apply(info.wantMassQuery) + .apply(info.priority) + .apply(info.bloomFilterUrl.value_or(""), info.bloomFilterUrl.has_value())); if (!r.next()) { unreachable(); } - ret.id = (int) r.getInt(0); + ret.info.id = (int) r.getInt(0); } state->caches[uri] = ret; txn.commit(); - return ret.id; + return ret.info.id; }); } @@ -250,7 +270,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache auto cache(queryCacheRaw(*state, uri)); if (!cache) return std::nullopt; - return CacheInfo{.id = cache->id, .wantMassQuery = cache->wantMassQuery, .priority = cache->priority}; + return cache->info; }); } @@ -266,7 +286,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache auto now = time(nullptr); auto queryNAR(state->queryNAR.use() - .apply(cache.id) + .apply(cache.info.id) .apply(hashPart) .apply(now - settings.ttlNegative) .apply(now - settings.ttlPositive)); @@ -312,7 +332,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache auto now = time(nullptr); auto queryRealisation(state->queryRealisation.use() - .apply(cache.id) + .apply(cache.info.id) .apply(id.to_string()) .apply(now - settings.ttlNegative) .apply(now - settings.ttlPositive)); @@ -350,7 +370,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache // assert(hashPart == storePathToHash(info->path)); state->insertNAR.use() - .apply(cache.id) + .apply(cache.info.id) .apply(hashPart) .apply(std::string(info->path.name())) .apply(narInfo ? narInfo->url : "", narInfo != 0) @@ -372,7 +392,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache .exec(); } else { - state->insertMissingNAR.use().apply(cache.id).apply(hashPart).apply(time(nullptr)).exec(); + state->insertMissingNAR.use().apply(cache.info.id).apply(hashPart).apply(time(nullptr)).exec(); } }); } @@ -385,7 +405,7 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache auto & cache(getCache(*state, uri)); state->insertRealisation.use() - .apply(cache.id) + .apply(cache.info.id) .apply(realisation.id.to_string()) .apply(static_cast(realisation).dump()) .apply(time(nullptr)) @@ -399,7 +419,100 @@ struct NarInfoDiskCacheImpl : NarInfoDiskCache auto state(_state.lock()); auto & cache(getCache(*state, uri)); - state->insertMissingRealisation.use().apply(cache.id).apply(id.to_string()).apply(time(nullptr)).exec(); + state->insertMissingRealisation.use() + .apply(cache.info.id) + .apply(id.to_string()) + .apply(time(nullptr)) + .exec(); + }); + } + + std::optional probeBloomFilter(const std::string & uri, const StorePath & path) override + { + return retrySQLite>([&]() -> std::optional { + auto state(_state.lock()); + auto & cache(getCache(*state, uri)); + + /* Use a fixed reference time (captured at the first probe in + this process) rather than the moving wall clock. Otherwise a + filter we (re)fetched and stamped a moment ago could already + read as "stale" — especially under `--refresh`, which sets + `ttlNegative` to 0 — and we'd re-fetch the shared filter on + every query. With a fixed `startTime`, a filter stamped at or + after `startTime` stays fresh for the rest of the process. */ + static auto startTime = time(nullptr); + + int64_t rowid; + { + auto q(state->queryFreshBloomFilter.use().apply(cache.info.id).apply(startTime - settings.ttlNegative)); + if (!q.next()) + return std::nullopt; // no filter cached, or stale + rowid = q.getInt(0); + } + + sqlite3_blob * blob = nullptr; + if (sqlite3_blob_open(state->db, "main", "BloomFilters", "blob", rowid, /*write=*/0, &blob) != SQLITE_OK) + SQLiteError::throw_(state->db, "opening bloom-filter blob"); + Finally _closeBlob([&] { + if (blob) + sqlite3_blob_close(blob); + }); + + /* Read and parse the header to get the filter parameters. */ + char header[bloomFilterHeaderLen]; + if (sqlite3_blob_bytes(blob) < (int) bloomFilterHeaderLen + || sqlite3_blob_read(blob, header, bloomFilterHeaderLen, 0) != SQLITE_OK) + return std::nullopt; // corrupt; treat as absent so we refetch + auto params = parseBloomFilterHeader({header, bloomFilterHeaderLen}); + if (!params) + return std::nullopt; + + bool allSet = true; + forEachBloomBitPosition(path, params->k, params->mBits, [&](uint64_t pos) { + if (!allSet) + return; + unsigned char byte = 0; + if (sqlite3_blob_read(blob, &byte, 1, (int) (bloomFilterHeaderLen + pos / 8)) != SQLITE_OK) + SQLiteError::throw_(state->db, "reading bloom-filter blob"); + if (!((byte >> (pos % 8)) & 1)) + allSet = false; + }); + return allSet; + }); + } + + void upsertBloomFilter(const std::string & uri, const std::string & etag, std::span blob) override + { + retrySQLite([&]() { + auto state(_state.lock()); + auto & cache(getCache(*state, uri)); + state->insertBloomFilter.use() + .apply(cache.info.id) + .apply(time(nullptr)) + .apply(etag, !etag.empty()) + .apply(reinterpret_cast(blob.data()), blob.size()) + .exec(); + }); + } + + void touchBloomFilter(const std::string & uri, const std::string & etag) override + { + retrySQLite([&]() { + auto state(_state.lock()); + auto & cache(getCache(*state, uri)); + state->touchBloomFilter.use().apply(time(nullptr)).apply(etag, !etag.empty()).apply(cache.info.id).exec(); + }); + } + + std::optional getBloomFilterETag(const std::string & uri) override + { + return retrySQLite>([&]() -> std::optional { + auto state(_state.lock()); + auto & cache(getCache(*state, uri)); + auto q(state->queryBloomFilterETag.use().apply(cache.info.id)); + if (!q.next() || q.isNull(0)) + return std::nullopt; + return q.getStr(0); }); } }; diff --git a/src/nix/generate-bloom-filter.cc b/src/nix/generate-bloom-filter.cc new file mode 100644 index 000000000000..813f3c59d898 --- /dev/null +++ b/src/nix/generate-bloom-filter.cc @@ -0,0 +1,76 @@ +#include "nix/cmd/command.hh" +#include "nix/store/bloom-filter.hh" +#include "nix/store/store-api.hh" +#include "nix/util/file-system.hh" +#include "nix/util/serialise.hh" +#include "nix/util/strings.hh" + +#include + +using namespace nix; + +struct CmdGenerateBloomFilter : StoreCommand +{ + std::optional fromFile; + double falsePositiveRate = 0.01; + + CmdGenerateBloomFilter() + { + addFlag({ + .longName = "from-file", + .description = "Read newline-separated store paths from *file* instead of " + "enumerating every valid path in the store.", + .labels = {"file"}, + .handler = {[this](std::string s) { fromFile = s; }}, + }); + addFlag({ + .longName = "false-positive-rate", + .description = "Target false-positive rate (default: 0.01).", + .labels = {"rate"}, + .handler = {[this](std::string s) { falsePositiveRate = std::stod(s); }}, + }); + } + + std::string description() override + { + return "build a Bloom filter from the store's valid paths"; + } + + Category category() override + { + return catUndocumented; + } + + void run(ref store) override + { + auto fd = getStandardOutput(); + if (isatty(fd)) + throw UsageError("refusing to write Bloom filter to a terminal"); + + StorePathSet paths; + if (fromFile) { + for (auto & line : tokenizeString(readFile(*fromFile), "\n")) { + auto trimmed = trim(line); + if (trimmed.empty()) + continue; + paths.insert(store->parseStorePath(trimmed)); + } + } else { + paths = store->queryAllValidPaths(); + } + + auto blob = buildBloomFilter(paths, falsePositiveRate); + + FdSink sink(std::move(fd)); + sink(blob); + sink.flush(); + + notice( + "Wrote Bloom filter (%d bytes) for %d store paths (%f false positive rate).", + blob.size(), + paths.size(), + falsePositiveRate); + } +}; + +static auto rCmdGenerateBloomFilter = registerCommand2({"store", "generate-bloom-filter"}); diff --git a/src/nix/meson.build b/src/nix/meson.build index b7ddcc8eec44..5b452f809fa7 100644 --- a/src/nix/meson.build +++ b/src/nix/meson.build @@ -96,6 +96,7 @@ nix_sources = [ config_priv_h ] + files( 'flake-prefetch-inputs.cc', 'flake.cc', 'formatter.cc', + 'generate-bloom-filter.cc', 'hash.cc', 'log.cc', 'ls.cc', diff --git a/src/nix/serve.cc b/src/nix/serve.cc index 7206411fd3da..b97463aac3b6 100644 --- a/src/nix/serve.cc +++ b/src/nix/serve.cc @@ -1,10 +1,12 @@ #include "nix/cmd/command.hh" #include "nix/util/file-system.hh" +#include "nix/util/hash.hh" #include "nix/util/serialise.hh" #include "nix/util/signals.hh" #include "nix/util/deleter.hh" #include "nix/store/nar-info.hh" #include "nix/store/binary-cache-store.hh" +#include "nix/store/bloom-filter.hh" #include "nix/store/log-store.hh" #include "nix/util/environment-variables.hh" @@ -24,6 +26,7 @@ struct CmdServe : StoreCommand std::string listenAddress = "127.0.0.1"; std::optional priority; std::optional portFile; + double bloomFalsePositiveRate = 0.01; CmdServe() { @@ -53,6 +56,13 @@ struct CmdServe : StoreCommand .labels = {"priority"}, .handler = {[this](std::string s) { priority = std::stoi(s); }}, }); + addFlag({ + .longName = "false-positive-rate", + .description = "Target false-positive rate for the Bloom filter " + "served at `/bloom-filter` (default: 0.01).", + .labels = {"rate"}, + .handler = {[this](std::string s) { bloomFalsePositiveRate = std::stod(s); }}, + }); } std::string description() override @@ -110,7 +120,8 @@ struct CmdServe : StoreCommand auto body = std::make_unique( "StoreDir: " + store.storeDir + "\n" "WantMassQuery: " + (store.config.wantMassQuery ? "1" : "0") + "\n" - "Priority: " + std::to_string(priority.value_or(store.config.priority)) + "\n"); + "Priority: " + std::to_string(priority.value_or(store.config.priority)) + "\n" + "BloomFilter: /bloom-filter\n"); response.reset(MHD_create_response_from_buffer(body->size(), body->data(), MHD_RESPMEM_MUST_COPY)); MHD_add_response_header(response.get(), "Content-Type", "text/x-nix-cache-info"); @@ -211,6 +222,25 @@ struct CmdServe : StoreCommand response.reset(MHD_create_response_from_buffer(log->size(), log->data(), MHD_RESPMEM_MUST_COPY)); MHD_add_response_header(response.get(), "Content-Type", "text/plain; charset=utf-8"); + } + + else if (url == "/bloom-filter") { + auto body = + std::make_unique(buildBloomFilter(store.queryAllValidPaths(), bloomFalsePositiveRate)); + auto etag = + "\"" + hashString(HashAlgorithm::SHA512, *body).to_string(HashFormat::Base16, /*includePrefix=*/false) + + "\""; + + if (auto * inm = MHD_lookup_connection_value(connection, MHD_HEADER_KIND, "If-None-Match"); + inm && etag == inm) { + response.reset(MHD_create_response_from_buffer(0, (void *) "", MHD_RESPMEM_PERSISTENT)); + MHD_add_response_header(response.get(), "ETag", etag.c_str()); + return MHD_queue_response(connection, MHD_HTTP_NOT_MODIFIED, response.get()); + } + + response.reset(MHD_create_response_from_buffer(body->size(), body->data(), MHD_RESPMEM_MUST_COPY)); + MHD_add_response_header(response.get(), "Content-Type", "application/octet-stream"); + MHD_add_response_header(response.get(), "ETag", etag.c_str()); } else return notFound(); diff --git a/tests/functional/binary-cache.sh b/tests/functional/binary-cache.sh index ee6185a07563..e6adaa436b11 100755 --- a/tests/functional/binary-cache.sh +++ b/tests/functional/binary-cache.sh @@ -63,7 +63,9 @@ stopNixServe() { startNixServe() { local portFile="$TEST_ROOT/nix-serve-port" rm -f "$portFile" - nix serve --port 0 --port-file "$portFile" "$@" & + nixServeLog="$TEST_ROOT/nix-serve.log" + rm -f "$nixServeLog" + nix serve --port 0 --port-file "$portFile" "$@" > "$nixServeLog" 2>&1 & nixServePid="$!" while [[ ! -e "$portFile" ]]; do if ! kill -0 "$nixServePid" 2>/dev/null; then @@ -145,6 +147,45 @@ nix path-info -vvvv --store "$httpBinaryCacheUrl" "$bigFile" 2> "$TEST_ROOT/log" [[ $(grep -c "downloading.*narinfo'" "$TEST_ROOT/log") -eq 1 ]] +# Bloom filter advertised by `nix serve` should rule out random store paths. +# Any fixed hashpart can hit a false positive, so loop generating fakes +# until one is ruled out. The counter sits in the last 6 chars of the +# hashpart because Nix32 decode reverse-iterates: low-order chars feed the +# bytes that drive `h1` in the double-hashing scheme; varying the leading +# chars wouldn't change `h1` or `h2` at all. +clearCacheCache +restartNixServe +ruledOut=0 +for n in $(seq 0 100); do + fake="$NIX_STORE_DIR/00000000000000000000000000$(printf '%06d' "$n")-fake-not-in-cache" + nix path-info --debug --store "$httpBinaryCacheUrl" "$fake" 2> "$TEST_ROOT/bloom-log" || true + if grep -q "Bloom filter for.*ruled out.*$fake" "$TEST_ROOT/bloom-log"; then + ruledOut=1 + break + fi +done +[[ $ruledOut -eq 1 ]] + + +# The Bloom filter should have been fetched exactly once across all the +# loop iterations, proving the disk-cache reuse path works. +[[ $(grep -c "url=/bloom-filter" "$nixServeLog") -eq 1 ]] + + +# `--refresh` should force the cached filter to be treated as stale; the +# client must re-fetch with `If-None-Match` and the server should reply 304 +# Not Modified instead of resending the body. +# Sleep so the cached entry is stamped strictly before this --refresh +# process starts (freshness is at 1-second resolution). +sleep 1 +prev=$(grep -c "url=/bloom-filter" "$nixServeLog") +nix path-info --debug --refresh --store "$httpBinaryCacheUrl" "$fake" 2> "$TEST_ROOT/bloom-log3" || true +# One additional /bloom-filter request was made. +[[ $(grep -c "url=/bloom-filter" "$nixServeLog") -eq $((prev + 1)) ]] +# And the client logged the 304 Not Modified branch. +grepQuiet "Bloom filter for.*unchanged.*304 Not Modified" "$TEST_ROOT/bloom-log3" + + # Test that multiple concurrent substitutions do only one download. clearStore nix-store --init # needed because concurrent creation of the store can give SQLite errors