[QDP] Unify Null Value Handling Strategy between Batch and Streaming Modes

[ryankert01]

Description

There is currently an inconsistency in how our system handles null values depending on the execution mode. To ensure predictable behavior and data integrity, we should align these strategies.

Current Behavior

• Batch Mode: Null values are automatically coerced/set to 0.
• Streaming Mode: Null values trigger a runtime error.

Proposed Change

We need to decide on a unified strategy for handling null values across both modes. Potential directions include:

1. Strict Error Handling: Both modes return an error to prevent silent data corruption.
2. Default Coercion: Both modes default to 0 (or a type-appropriate default).
3. Configurable Policy: Allow users to define the on_null behavior (e.g., ignore, fill_zero, or raise_error).

Context

#753 (comment)

To-Do / Discussion Points

• Research performance implications of adding null-checks in the streaming hot path.
• Determine if "Default to 0" is safe for all supported data types.
• Reach a consensus on the unified default behavior.

[manvendra5345]

Hi @ryankert01 ,

I’m interested in working on this issue as part of GSoC. I’ve gone through the issue details and related code, and I’d love to take ownership and start contributing.

If it’s available, could you please assign this issue to me?

Thanks!

[ryankert01]

@manvendra5345 Sweet! Feel free to take over!

[manvendra5345]

Sure sir I will strat working on it right away.

[manvendra5345]

Hi @ryankert01 , I'm a new open-source contributor and I'm interested in taking on this issue. I’ve reviewed the description regarding the inconsistency between Batch and Streaming modes. Could you point me toward the specific modules or files where the null-handling logic for these modes is defined? I want to make sure I'm looking at the right place before I start drafting a fix. Thanks!

[400Ping]

Btw I am not very sure if the qdp part is included in the GSoC, it is a PoC that we are working on and it requires knowledge of cuda, rust and GPUs.

[manvendra5345]

No problem @400Ping sir as I told above my goal is to learn open contribution as I a, just strating so I have read the issue discription and undestood the changes but i am not able to locate how can I start. If you can help me.

[400Ping]

You can start by looking into the code since it's already pretty straightforward. Also you need a Linux system and a RTX 30s GPU or above to run and test the code base.

[ryankert01]

Good luck along with the journey! Btw, if you don't have device needed, you can use colab to run tests! We have a reference of how to install in /benchmarks/notebooks

[manvendra5345]

No sir I have device which fulfil the requirements