add Chunk + Fields API with selective decode#2
Open
georgeboot wants to merge 3 commits into
Open
Conversation
a6b0a9b to
be13e32
Compare
Replace DecompressedChunk + Vec<las::Point> with a Chunk wrapping las::PointCloud (from gadomski/las-rs#143) and a Fields bitflag that drives laz::DecompressionSelection for layer-level selective decode. Two-tier reader API: fetch_chunk/query_chunks take a Fields mask and return Chunks with zero-copy column access; fetch_points/query_points stay as thin wrappers. copc-temporal mirrors the split. Field accessors return Option so skipped fields can't read as zero. Bumps version to 0.2.0-dev to reflect the breaking changes (DecompressedChunk and the old read_points methods are gone).
be13e32 to
9a1f5b7
Compare
Two additive APIs that share the same internal pipeline: - `CopcStreamingReader::fetch_chunks(keys, fields)`: coalesces N chunk reads into a single `ByteSource::read_ranges` call. HTTP byte sources that override `read_ranges` (parallel requests / range merging) issue far fewer round-trips than calling `fetch_chunk` in a loop. The internal `decompress_chunk` is extracted from `fetch_and_decompress` so both paths share the same decode logic. - `Chunk::decompress(compressed, entry, laz_vlr, header, fields)`: public sync entry point that takes pre-fetched bytes. Useful when a caller wants to issue range fetches on one executor (e.g. browser main thread) and run the CPU-bound LAZ decompression on another (e.g. a Rayon worker pool) — or otherwise drive the I/O step independently. `copc-temporal::TemporalCache::query_chunks` / `query_points` switch to `fetch_chunks` so temporal queries get the same batching benefit. No breaking changes: `fetch_chunk`, `query_chunks`, and the simple `query_points` API keep their existing signatures.
Adds `HierarchyCache::pending_page_keys` (and a forwarder on `CopcStreamingReader`) that iterates the `VoxelKey`s anchoring the currently-pending sub-pages. Lets consumers surface unloaded sub-trees as proxy nodes in their own data structure and trigger `load_pages_for_bounds` only when traversal needs them, rather than having to either eagerly call `load_all_hierarchy` (expensive on large files) or never realise an unloaded sub-tree exists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the old
DecompressedChunk+Vec<las::Point>reader path with a column-orientedChunkwrappinglas::PointCloud(from gadomski/las-rs#143) and aFieldsbitflag that driveslaz::DecompressionSelectionfor layer-level selective decode. The point is to stop decoding layers nobody reads and stop materializing points we're just going to throw away.Two-tier reader API:
fetch_points/query_points/query_points_to_levelreturnVec<las::Point>. Unchanged shape for scripts and tests.fetch_chunk/query_chunks/query_chunks_to_leveltake aFieldsmask and returnChunks with zero-copy column access. Field accessors returnOptionso skipped fields can't silently read as zero.copc-temporalmirrors the same split withquery_chunks/query_chunks_by_timealongside the existingquery_pointsvariants.Depends on the updated
jacovdbergh/las-rspoint-cloud-apibranch (pinned to833cf7a). Bumps the workspace version to0.2.0-devsinceDecompressedChunkand the oldread_points*methods are gone.Before / after
Fetching only xyz + rgb for a bounding box:
Temporal query that intersects a spatial and time window:
Test plan
cargo build --allcargo clippy --all --all-targets -- -D warningscargo test --all— 42 unit tests + 1 doctest passcargo doc --no-deps --all— no warnings, no broken linkscargo semver-checks check-release -p copc-streaming -p copc-temporal— passes against the 0.1.5 baseline