MAHOUT-725: [QDP] PyTorch Tensor Detection and CPU Path by CheyuWu · Pull Request #763 · apache/mahout

CheyuWu · 2025-12-27T14:02:34Z

Purpose of PR

This issue asks the ability to detect and handle PyTorch tensors, focusing on cpu tensors.

add PyTorch tensor detection utilities
add tensor validation
add cpu tensor path in encode_tensor()
add error handling
add basic tests

Related Issues or PRs

Closes #725

Changes Made

Breaking Changes

Yes
No

Checklist

Added or updated unit tests for all changes
Added or updated documentation for all changes
Successfully built and ran all unit tests or manual tests locally
PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
Code follows ASF guidelines

CheyuWu · 2025-12-27T14:08:25Z

Manual Testing

(mahout) user@DESKTOP-UDTSV2K:~/code/mahout$ cd ./qdp/qdp-python/
(mahout) user@DESKTOP-UDTSV2K:~/code/mahout/qdp/qdp-python$ cp ../target/debug/libmahout_qdp.so mahout_qdp.so && PYTHONPATH=. pytest .
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.13, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/user/code/mahout/qdp/qdp-python
configfile: pyproject.toml
collected 21 items                                                                                                                                                      

tests/test_bindings.py ...s.....                                                                                                                                  [ 42%]
tests/test_high_fidelity.py ............                                                                                                                          [100%]

===================================================================== 20 passed, 1 skipped in 3.30s =====================================================================

400Ping · 2025-12-29T13:37:00Z

please fix the ci error

CheyuWu · 2025-12-29T16:07:00Z

Hi @400Ping, I have fixed the problem PTAL

400Ping · 2025-12-29T16:39:39Z

I think overall is good but we are going to merge #753 into dev-qdp, I think it is best to wait for it to merge and fix the conflicts for this pr.

400Ping · 2025-12-30T12:21:09Z

Its merged, please taken account how other inputs are written and refactor this pr.

CheyuWu · 2025-12-30T15:43:05Z

Its merged, please taken account how other inputs are written and refactor this pr.

OK, I will address this tomorrow

rich7420 · 2025-12-31T08:07:43Z

plz fix precommit errors

CheyuWu · 2025-12-31T10:36:04Z

Hi @rich7420 @400Ping , I’ve addressed the issue.
BTW, the numpy package was missing from pyproject.toml, so I added it.

Manual Testing

(qdp-python) user@DESKTOP-UDTSV2K:~/code/mahout/qdp/qdp-python$ PYTHONPATH=. python -m pytest .
========================================================================== test session starts ==========================================================================
platform linux -- Python 3.11.13, pytest-9.0.1, pluggy-1.6.0
rootdir: /home/user/code/mahout/qdp/qdp-python
configfile: pyproject.toml
collected 21 items                                                                                                                                                      

tests/test_bindings.py ...s.....                                                                                                                                  [ 42%]
tests/test_high_fidelity.py ............                                                                                                                          [100%]

===================================================================== 20 passed, 1 skipped in 2.63s =====================================================================

400Ping · 2025-12-31T16:51:42Z

@CheyuWu see this #752 (comment)

ryankert01 · 2025-12-31T17:04:23Z

@CheyuWu feel free to help review this related PR #777 ~ It's kind of complex to add a input data format.

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

400Ping · 2026-01-03T10:54:05Z

cc @guan404ming @rich7420 @ryankert01 to review
I think we can merge this first and I will open a pr for handling PyTorch input format.

rich7420 · 2026-01-03T13:47:49Z

@CheyuWu plz fix this pre-commit error

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

CheyuWu · 2026-01-03T13:55:02Z

Let me check what is going on

rich7420 · 2026-01-03T14:03:49Z

+        let data: Vec<f64> = tensor
+            .call_method0("flatten")?
+            .call_method0("tolist")?
+            .extract()?;


It seems here do twice copy.

PyTorch tensor → Python list Python list → Rust Vec<f64>

We should improve this in follow-up PR use like numpy() or PinnedHostBuffer way to decrease memory copy time.
Please comments in this part.

OK no problem
I have added the comment PTAL

rich7420 · 2026-01-03T14:04:47Z

I think overall LGTM
It will be merged when all green.

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

CheyuWu · 2026-01-03T14:19:26Z

This is modified by ruff format

I think this is a unrelated format error. It should be reverted.
cc @ryankert01

I have reverted the code PTAL

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

rich7420 · 2026-01-03T15:03:30Z

@400Ping do you want to take another look?

400Ping

LGTM

ryankert01

Lg

rich7420 · 2026-01-04T09:07:54Z

@CheyuWu thanks for the update!
@400Ping , @ryankert01 thanks for the review!

* feat: [QDP] PyTorch Tensor Detection and CPU Path Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * style: fix linter issue Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * style: linter err Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * doc: add comment for followup pr Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> * revert: ipynb linter fix Signed-off-by: Cheyu Wu <cheyu1220@gmail.com> --------- Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

CheyuWu force-pushed the feat/i/725 branch from 02af635 to 538829b Compare December 27, 2025 14:03

CheyuWu changed the title ~~[QDP] PyTorch Tensor Detection and CPU Path~~ MAHOUT-725 [QDP] PyTorch Tensor Detection and CPU Path Dec 27, 2025

CheyuWu changed the title ~~MAHOUT-725 [QDP] PyTorch Tensor Detection and CPU Path~~ MAHOUT-725: [QDP] PyTorch Tensor Detection and CPU Path Dec 27, 2025

CheyuWu changed the base branch from main to dev-qdp December 27, 2025 14:07

rich7420 added the qdp label Dec 29, 2025

CheyuWu force-pushed the feat/i/725 branch from 538829b to edd1d18 Compare December 29, 2025 15:31

CheyuWu force-pushed the feat/i/725 branch from edd1d18 to 4d99723 Compare December 31, 2025 10:19

feat: [QDP] PyTorch Tensor Detection and CPU Path

41643f9

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

CheyuWu force-pushed the feat/i/725 branch from a8c4575 to 41643f9 Compare January 3, 2026 11:19

style: fix linter issue

9683362

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

rich7420 reviewed Jan 3, 2026

View reviewed changes

CheyuWu added 2 commits January 3, 2026 22:14

style: linter err

648ccf7

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

doc: add comment for followup pr

03b698c

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

CheyuWu commented Jan 3, 2026

View reviewed changes

revert: ipynb linter fix

cfd2abb

Signed-off-by: Cheyu Wu <cheyu1220@gmail.com>

400Ping approved these changes Jan 4, 2026

View reviewed changes

ryankert01 approved these changes Jan 4, 2026

View reviewed changes

rich7420 approved these changes Jan 4, 2026

View reviewed changes

rich7420 merged commit 6bb8382 into apache:dev-qdp Jan 4, 2026
2 checks passed

guan404ming mentioned this pull request Jan 8, 2026

[QDP] PyTorch Tensor Detection and CPU Path #725

Closed

guan404ming added this to the Qumat 0.5.0 milestone Jan 20, 2026

This was referenced Jan 28, 2026

Status of testing Apache Mahout Qumat 0.5.0rc1 #963

Closed

Status of testing Apache Mahout Qumat 0.5.0rc2 #986

Closed

Conversation

CheyuWu commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of PR

Related Issues or PRs

Changes Made

Breaking Changes

Checklist

Uh oh!

CheyuWu commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Manual Testing

Uh oh!

400Ping commented Dec 29, 2025

Uh oh!

CheyuWu commented Dec 29, 2025

Uh oh!

400Ping commented Dec 29, 2025

Uh oh!

400Ping commented Dec 30, 2025

Uh oh!

CheyuWu commented Dec 30, 2025

Uh oh!

rich7420 commented Dec 31, 2025

Uh oh!

CheyuWu commented Dec 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Manual Testing

Uh oh!

400Ping commented Dec 31, 2025

Uh oh!

ryankert01 commented Dec 31, 2025

Uh oh!

400Ping commented Jan 3, 2026

Uh oh!

rich7420 commented Jan 3, 2026

Uh oh!

CheyuWu commented Jan 3, 2026

Uh oh!

rich7420 Jan 3, 2026

Choose a reason for hiding this comment

Uh oh!

CheyuWu Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rich7420 commented Jan 3, 2026

Uh oh!

CheyuWu Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rich7420 Jan 3, 2026

Choose a reason for hiding this comment

Uh oh!

CheyuWu Jan 3, 2026

Choose a reason for hiding this comment

Uh oh!

rich7420 commented Jan 3, 2026

Uh oh!

400Ping left a comment

Choose a reason for hiding this comment

Uh oh!

ryankert01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rich7420 commented Jan 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

CheyuWu commented Dec 27, 2025 •

edited

Loading

CheyuWu commented Dec 27, 2025 •

edited

Loading

CheyuWu commented Dec 31, 2025 •

edited

Loading

CheyuWu Jan 3, 2026 •

edited

Loading

CheyuWu Jan 3, 2026 •

edited

Loading