feat: add theta sketch (part 2) #59

ZENOTME · 2026-01-01T07:48:43Z

add binomial_bounds to support calculate lower_bound&&upper_bound
add get_lower_bound&&get_upper_bound in ThetaSketch

cc @notfilippo @Xuanwo @PsiACE @tisonkun

datasketches/src/theta/sketch.rs

leerho · 2026-01-04T23:24:30Z

Waiting on requested change.

tisonkun · 2026-01-05T00:07:01Z

datasketches/src/theta/sketch.rs

+    pub fn lower_bound(&self, num_std_devs: u32) -> Result<f64, Error> {
+        if !self.is_estimation_mode() {
+            return Ok(self.num_retained() as f64);
+        }
+        binomial_bounds::lower_bound(self.num_retained() as u64, self.theta(), num_std_devs)
+    }


We already have a NumStdDev enum:

datasketches-rust/datasketches/src/hll/estimator.rs

Lines 327 to 341 in 9ade42d

/// Number of standard deviations for confidence bounds

///

/// This enum specifies the number of standard deviations to use when computing

/// upper and lower bounds for cardinality estimates. Higher values provide wider

/// confidence intervals with greater certainty that the true cardinality falls

/// within the bounds.

#[repr(u8)]

pub enum NumStdDev {

/// One standard deviation (\~68% confidence interval)

One = 1,

/// Two standard deviations (\~95% confidence interval)

Two = 2,

/// Three standard deviations (\~99.7% confidence interval)

Three = 3,

}

You can reuse it by moving the struct to a shared module level and thus we would never error here.

Besides, we may consider where to place these shared structs, including the ResizeFactor. datasketches::{NumStdDev, ResizeFactor} is fair enough now, but perhaps pollute the top-level namespace too much.

tisonkun · 2026-01-05T00:08:21Z

datasketches/src/binomial_bounds.rs

+        // Comment for this larger test like Java
+        // for ci in 1..=3 {
+        //     let arr = run_test_aux(2000, ci, 1e-7);
+        //     for j in 0..5 {
+        //         let ratio = arr[j] / STD[i][j];
+        //         assert!(
+        //             (ratio - 1.0).abs() < TOL,
+        //             "ci={}, j={}: expected {}, got {}, ratio={}",
+        //             ci,
+        //             j,
+        //             STD[i][j],
+        //             arr[j],
+        //             ratio
+        //         );
+        //     }
+        //     i += 1;
+        // }


How much time it costs? I'd prefer not to comment out tests.

I can't seem to find the source for this commented out test in either C++ or Java . Could you give me a full link?

https://github.com/apache/datasketches-java/blob/6acf50c346a1b5d79b19895d855540995419250d/src/test/java/org/apache/datasketches/thetacommon/BinomialBoundsNTest.java#L96

How much time it costs? I'd prefer not to comment out tests.

0.56s(comment out) vs 3.19s in mac m1 pro

0.56s(comment out) vs 3.19s in mac m1 pro

This can be fair enough to include.

tisonkun

The rest should be OK.

https://github.com/apache/datasketches-rust/pull/59/changes#r2660026906 is significant.

* add binomial_bounds to support calculate lower_bound&&upper_bound * add get_lower_bound&&get_upper_bound in ThetaSketch

tisonkun · 2026-01-05T15:55:17Z

I'll appreciate it if you can, the next time, avoid force push after reviewers claim in. Then reviewers can easily get the changeset from the last review.

tisonkun · 2026-01-05T15:56:09Z

datasketches/src/lib.rs

+mod binomial_bounds;
 mod codec;
 mod hash;
+pub mod num_std_dev;


Suggested change

pub mod num_std_dev;

mod num_std_dev;

Why pub?

I remove it later

datasketches/src/lib.rs

ZENOTME · 2026-01-05T16:33:56Z

I'll appreciate it if you can, the next time, avoid force push after reviewers claim in. Then reviewers can easily get the changeset from the last review.

I’ll be more careful next time. If I need to rebase, should I use merge in situations like this instead?

leerho

This Looks OK to me. However, there are 2 unresolved requested changes, and the last CI produced errors.

ZENOTME · 2026-01-06T14:05:24Z

This Looks OK to me. However, there are 2 unresolved requested changes, and the last CI produced errors.

Looks like ci fail at ./tools/generate_serialization_test_data.py, should we re run to try again. Test locally pass

leerho · 2026-01-06T18:04:27Z

Ran the tests again, still failed.

leerho · 2026-01-06T18:13:46Z

I think it would be a good idea to have the workflow_dispatch: trigger on any workflows. It simplifies troubleshooting.

tisonkun · 2026-01-07T01:43:40Z

Error: Expected generated files directory not found at /home/runner/work/datasketches-rust/datasketches-rust/tmp_datasketches_java/serialization_test_data/java_generated_files

Look like some Java side logic changed so that directory has changed as well.

I'll take a look after managing the release vote to the next stage. And perhaps #10 should help in this issue.

I think it would be a good idea to have the workflow_dispatch: trigger on any workflows. It simplifies troubleshooting.

Good point.

PsiACE · 2026-01-07T05:34:13Z

Ran the tests again, still failed.

I included a temporary fix in #62 (Density Sketch) to ensure CI passes.

tisonkun · 2026-01-08T14:19:31Z

CI fixed. I'll give another look tomorrow and try to merge it to cut 0.2.0-rc.2/

Signed-off-by: tison <wander4096@gmail.com>

tisonkun

Push a new commit for reorg common types.

Now this looks good to me.

tisonkun · 2026-01-08T16:09:44Z

cc @leerho @PsiACE you may take another look then.

PsiACE

LGTM. And looking forward to the new release.

PsiACE suggested changes Jan 1, 2026

View reviewed changes

datasketches/src/theta/sketch.rs Outdated Show resolved Hide resolved

datasketches/src/theta/sketch.rs Outdated Show resolved Hide resolved

ZENOTME force-pushed the binomial_bounds branch from cf1c7b5 to c4e321a Compare January 1, 2026 16:06

tisonkun reviewed Jan 5, 2026

View reviewed changes

tisonkun requested changes Jan 5, 2026

View reviewed changes

ZENOTME added 2 commits January 5, 2026 21:44

feat: add theta sketch (part 2)

45019bd

* add binomial_bounds to support calculate lower_bound&&upper_bound * add get_lower_bound&&get_upper_bound in ThetaSketch

refine NumStdDev and some interface

3bfa390

ZENOTME force-pushed the binomial_bounds branch from c4e321a to 3bfa390 Compare January 5, 2026 14:54

ZENOTME requested review from PsiACE, leerho and tisonkun January 5, 2026 14:54

tisonkun reviewed Jan 5, 2026

View reviewed changes

datasketches/src/lib.rs Outdated Show resolved Hide resolved

refine code: remove unnecessary pub

911891e

leerho requested changes Jan 5, 2026

View reviewed changes

tisonkun mentioned this pull request Jan 7, 2026

Release v0.2.0 #56

Open

Merge branch 'main' into binomial_bounds

244ebbf

better common layout

fb13cef

Signed-off-by: tison <wander4096@gmail.com>

tisonkun approved these changes Jan 8, 2026

View reviewed changes

tisonkun requested a review from leerho January 8, 2026 16:09

PsiACE approved these changes Jan 8, 2026

View reviewed changes

leerho approved these changes Jan 8, 2026

View reviewed changes

leerho merged commit fdb38b8 into apache:main Jan 8, 2026
9 checks passed

	/// Number of standard deviations for confidence bounds
	///
	/// This enum specifies the number of standard deviations to use when computing
	/// upper and lower bounds for cardinality estimates. Higher values provide wider
	/// confidence intervals with greater certainty that the true cardinality falls
	/// within the bounds.
	#[repr(u8)]
	pub enum NumStdDev {
	/// One standard deviation (\~68% confidence interval)
	One = 1,
	/// Two standard deviations (\~95% confidence interval)
	Two = 2,
	/// Three standard deviations (\~99.7% confidence interval)
	Three = 3,
	}

feat: add theta sketch (part 2) #59

feat: add theta sketch (part 2) #59

Uh oh!

Conversation

ZENOTME commented Jan 1, 2026

Uh oh!

Uh oh!

Uh oh!

leerho commented Jan 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tisonkun left a comment

Choose a reason for hiding this comment

Uh oh!

tisonkun commented Jan 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZENOTME commented Jan 5, 2026

Uh oh!

leerho left a comment

Choose a reason for hiding this comment

Uh oh!

ZENOTME commented Jan 6, 2026

Uh oh!

leerho commented Jan 6, 2026

Uh oh!

leerho commented Jan 6, 2026

Uh oh!

tisonkun commented Jan 7, 2026

Uh oh!

PsiACE commented Jan 7, 2026

Uh oh!

tisonkun commented Jan 8, 2026

Uh oh!

tisonkun left a comment

Choose a reason for hiding this comment

Uh oh!

tisonkun commented Jan 8, 2026

Uh oh!

PsiACE left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants