Skip to content

Conversation

@jswelling
Copy link
Collaborator

@jswelling jswelling commented Jan 21, 2026

This PR introduces CellTable and CellTable3D, more compact representations of the long list of tiny cell-footprint arrays SPRM accesses via the ROI_coords list-of-lists. Iterators are provided to walk efficiently through the new tables.

Images also get a new "cache" functionality whereby a value can be associated with a name and then retrieved later via cache_set() and cache_get() methods. This is used because the existing code keeps a large data structure around so that a small summary can be extracted later. Using these methods, the summaries are extracted early and the larger dataset is freed.

The gist of the representation is that the input masks are sorted by cell ID, leading to a representation in which all coordinates associated with a given cell are grouped together. These are also sorted by Z, Y, and X coordinates so that the coordinates for a given cell appear in the expected order. This sorting is done in-place. The end result is a faster, much more compact representation. A great deal of memory is saved by eliminating the array description information associated with each small cell array. A second array is allocated listing the number of consecutive entries in each cell to facilitate traversal.

I have some worries about the 3D case, because the cell iterators return voxels in ZYX order as per "normal" arrays while some existing code reformats things into YXZ order to simplify the general case logic. Since I don't have a good 3D dataset for testing I have been unable to validate this case.

This version includes yet more diagnostic output, which will have to be cleaned up in a later PR.

The PR would benefit from a minor rebase, I think.

@jswelling jswelling marked this pull request as draft January 21, 2026 22:46
@jswelling jswelling requested review from mruffalo and tCeZ January 22, 2026 19:01
@jswelling jswelling marked this pull request as ready for review January 22, 2026 19:02
@jswelling jswelling requested a review from SFD5311 January 22, 2026 19:05
@mruffalo mruffalo force-pushed the welling/performance_opt_2_rebase branch from 0b6fe20 to b3e00b9 Compare January 22, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants