Consolidate open PRs: MIM inference, Hessian solver, gradient accumulation, and AMP by Copilot · Pull Request #10 · thinksyncs/YOLOZU

Copilot · 2026-02-06T04:29:29Z

Merges PRs #6, #7, #8, and #9 to prevent conflicts. Total delta: 18 files, +2,888 lines.

Changes

PR #6: Masked reconstruction branch (1,270 lines)

Adds RenderTeacher and DecoderMIM modules for geometry-aligned masked image modeling in inference
New loss functions: mim_reconstruction_loss() and entropy_loss() for test-time adaptation
RTDETRPose.forward() accepts geom_input (mask + normalized depth), feature_mask, and return_mim flag

model = RTDETRPose(enable_mim=True)
geom = create_geom_input(mask, depth_obj)
outputs = model(image, geom_input=geom, feature_mask=mask_feat, return_mim=True)
loss = mim_reconstruction_loss(outputs["mim"]["recon_feat"], outputs["mim"]["teacher_feat"])

PR #7: Hessian solver (1,319 lines)

Gauss-Newton with Levenberg-Marquardt damping for per-detection depth/rotation/offset refinement
HessianSolverConfig and refine_predictions_hessian() API
CLI tool: tools/refine_predictions_hessian.py

PR #8: Gradient accumulation + AMP (252 lines)

--gradient-accumulation-steps N scales loss by 1/N, defers optimizer step to accumulation boundaries
--use-amp enables torch.cuda.amp with proper gradient unscaling before clipping
Fixed lint error: removed unused json import in test file

PR #9: Lint documentation (50 lines)

Documents lint fixes needed in other PRs (already applied during merge)

Implementation notes

Merge strategy: patch-based application to main branch to avoid unrelated history conflicts
All features backward compatible with existing code (disabled by default)
MIM operates on P5 features (1/16 to 1/32 scale) with ~100K additional parameters
Hessian solver typical convergence: 3-5 iterations, ~1-2ms per detection on CPU

Original prompt

今上がっているPRを全てまとめて一つのPRにして欲しい。
for prevents conflicts

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

…entropy loss and geometric consistency

…inement

…with lint fix)

Copilot

Pull request overview

This pull request consolidates four separate PRs (#6, #7, #8, #9) into a single comprehensive update that adds significant new functionality to the YOLOZU RT-DETR pose estimation system. The consolidation strategy prevents merge conflicts by applying changes in a coordinated manner.

Changes:

MIM Reconstruction Branch (PR #6): Adds geometry-aligned masked image modeling for test-time adaptation with ~100K additional parameters, including RenderTeacher and DecoderMIM modules
Hessian Solver (PR #7): Implements Gauss-Newton optimization with Levenberg-Marquardt damping for per-detection refinement of depth, rotation, and offset predictions
Gradient Accumulation & AMP (PR #8): Adds training enhancements with gradient accumulation support and automatic mixed precision (AMP) for efficient large-batch training
Lint Documentation (PR #9): Documents and fixes lint errors from other PRs (unused import removed)

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`yolozu/calibration/hessian_solver.py`	New Hessian-based solver for iterative regression refinement using second-order optimization
`yolozu/calibration/__init__.py`	Exports for HessianSolverConfig and refinement functions
`tools/refine_predictions_hessian.py`	CLI tool for batch refinement of predictions with configurable solver parameters
`rtdetr_pose/rtdetr_pose/model.py`	Adds RenderTeacher, DecoderMIM modules and MIM branch to RTDETRPose with backward compatibility
`rtdetr_pose/rtdetr_pose/losses.py`	New loss functions: mim_reconstruction_loss and entropy_loss for geometric consistency
`rtdetr_pose/tools/train_minimal.py`	Gradient accumulation and AMP integration with proper gradient scaling and clipping
`tests/test_hessian_solver.py`	Comprehensive test suite (9 tests) for Hessian solver
`tests/test_mim_reconstruction.py`	Comprehensive test suite (10 tests) for MIM components
`rtdetr_pose/tests/test_train_minimal_integration.py`	Integration tests for gradient accumulation and AMP
`rtdetr_pose/tests/test_train_minimal_grad_accum_amp.py`	Unit tests for new training arguments
`tools/example_mim_inference.py`	Example script demonstrating MIM usage with test-time adaptation
`docs/hessian_solver.md`	Documentation for Hessian solver API and usage
`docs/mim_inference.md`	Documentation for MIM branch usage and test-time training
`train_setting.yaml`	Configuration examples for new gradient accumulation and AMP features
`README.md`	Updated feature highlights with Hessian solver reference
`LINT_FIXES_NEEDED.md`	Documents lint fixes applied during merge
`SECURITY_SUMMARY.md`	Security assessment of all changes
`IMPLEMENTATION_SUMMARY.md`	Detailed implementation summary for MIM feature

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-06T06:44:13Z

rtdetr_pose/tools/train_minimal.py

+            # Perform optimizer step only at accumulation boundaries
+            # steps is 0-indexed within each epoch, so we use (steps + 1) for the check
+            if (steps + 1) % accum_steps == 0:
+                if scaler is not None:
+                    # Unscale gradients before clipping
+                    if args.clip_grad_norm and float(args.clip_grad_norm) > 0:
+                        scaler.unscale_(optim)
+                        torch.nn.utils.clip_grad_norm_(model.parameters(), float(args.clip_grad_norm))
+                    scaler.step(optim)
+                    scaler.update()
+                    optim.zero_grad(set_to_none=True)
+                else:
+                    if args.clip_grad_norm and float(args.clip_grad_norm) > 0:
+                        torch.nn.utils.clip_grad_norm_(model.parameters(), float(args.clip_grad_norm))
+                    optim.step()
+                    optim.zero_grad(set_to_none=True)


The optimizer step is only performed when (steps + 1) % accum_steps == 0. However, if the epoch ends before reaching an accumulation boundary, gradients will remain accumulated without being applied. This means the last few batches of an epoch might not contribute to parameter updates if they don't reach an accumulation boundary. Consider adding logic to perform a final optimizer step at the end of each epoch if there are accumulated gradients remaining.

Copilot · 2026-02-06T06:44:13Z

rtdetr_pose/tools/train_minimal.py

+    # Initialize GradScaler for AMP if enabled
+    scaler = None
+    if args.use_amp:
+        if device.startswith("cuda"):
+            scaler = torch.cuda.amp.GradScaler()
+            print("amp_enabled=True device=cuda")
+        else:
+            print("amp_warning: --use-amp requires CUDA device; AMP disabled")
+


When gradient accumulation is used with checkpoint resumption, there's a potential issue: if training is resumed and the checkpoint had accumulated gradients, those gradients might still be present. The code doesn't explicitly zero gradients at the start of training after checkpoint loading. Consider adding optim.zero_grad(set_to_none=True) after initializing the scaler (around line 1718) to ensure a clean gradient state when starting/resuming training with gradient accumulation.

Copilot · 2026-02-06T06:44:14Z

tests/test_mim_reconstruction.py

+        # With mask
+        loss_masked = mim_reconstruction_loss(recon_feat, teacher_feat, mask=mask)
+        self.assertEqual(loss_masked.shape, torch.Size([]))
+        self.assertTrue(loss_masked.item() >= 0.0)


assertTrue(a >= b) cannot provide an informative message. Using assertGreaterEqual(a, b) instead will give more informative messages.

Copilot · 2026-02-06T06:44:14Z

tests/test_mim_reconstruction.py

+        self.assertTrue(loss_masked.item() >= 0.0)
+
+        # Without mask
+        loss_full = mim_reconstruction_loss(recon_feat, teacher_feat, mask=None)
+        self.assertEqual(loss_full.shape, torch.Size([]))
+        self.assertTrue(loss_full.item() >= 0.0)


assertTrue(a >= b) cannot provide an informative message. Using assertGreaterEqual(a, b) instead will give more informative messages.

Suggested change

self.assertTrue(loss_masked.item() >= 0.0)

# Without mask

loss_full = mim_reconstruction_loss(recon_feat, teacher_feat, mask=None)

self.assertEqual(loss_full.shape, torch.Size([]))

self.assertTrue(loss_full.item() >= 0.0)

self.assertGreaterEqual(loss_masked.item(), 0.0)

# Without mask

loss_full = mim_reconstruction_loss(recon_feat, teacher_feat, mask=None)

self.assertEqual(loss_full.shape, torch.Size([]))

self.assertGreaterEqual(loss_full.item(), 0.0)

Copilot · 2026-02-06T06:44:14Z

tests/test_mim_reconstruction.py

+        loss = entropy_loss(logits)
+
+        self.assertEqual(loss.shape, torch.Size([]))
+        self.assertTrue(loss.item() >= 0.0)


assertTrue(a >= b) cannot provide an informative message. Using assertGreaterEqual(a, b) instead will give more informative messages.

Suggested change

self.assertTrue(loss.item() >= 0.0)

self.assertGreaterEqual(loss.item(), 0.0)

Copilot · 2026-02-06T06:44:14Z

tests/test_mim_reconstruction.py

+        self.assertIn("loss", result)
+        self.assertIn("loss_mim", result)
+        self.assertIn("loss_entropy", result)
+        self.assertTrue(result["loss"].item() > 0.0)


assertTrue(a > b) cannot provide an informative message. Using assertGreater(a, b) instead will give more informative messages.

Suggested change

self.assertTrue(result["loss"].item() > 0.0)

self.assertGreater(result["loss"].item(), 0.0)

Copilot · 2026-02-06T06:44:15Z

yolozu/calibration/hessian_solver.py

+            r_flat = [float(x) for row in gt_rotation for x in row]
+            if len(r_flat) == 9:
+                r_gt_tensor = torch.tensor(r_flat, dtype=dtype, device=device).reshape(3, 3)
+        except (TypeError, ValueError):


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except (TypeError, ValueError):

except (TypeError, ValueError):

# If ground-truth rotation is malformed, silently skip rotation supervision.

# This keeps behavior consistent with treating missing gt_rotation as no constraint.

Copilot · 2026-02-06T06:44:15Z

yolozu/calibration/hessian_solver.py

+                        if isinstance(t_gt, (list, tuple)) and len(t_gt) >= 3:
+                            try:
+                                gt_depth = float(t_gt[2])  # Z component.
+                            except (TypeError, ValueError):


'except' clause does nothing but pass and there is no explanatory comment.

Copilot · 2026-02-06T06:44:15Z

yolozu/calibration/hessian_solver.py

+                        if isinstance(r_gt, (list, tuple)) and len(r_gt) == 3:
+                            try:
+                                gt_rotation = [[float(x) for x in row] for row in r_gt]
+                            except (TypeError, ValueError):


'except' clause does nothing but pass and there is no explanatory comment.

Initial plan

ae12dea

Copilot AI assigned Copilot and thinksyncs Feb 6, 2026

Copilot started work on behalf of thinksyncs February 6, 2026 04:29 View session

Copilot AI added 6 commits February 6, 2026 04:32

Merge PR #6: Add masked reconstruction branch to inference loop with …

3c69c0b

…entropy loss and geometric consistency

Merge PR #7: Add Hessian solver for per-detection regression head ref…

2a6289e

…inement

Merge PR #8: Add gradient accumulation and AMP to training scaffold (…

ac3f6ad

…with lint fix)

Merge PR #9: Document lint errors (already fixed in PR #8 merge)

2f77223

Fix all linting errors from merged PRs

b883836

Update SECURITY_SUMMARY.md to reflect all consolidated PRs

a523e61

Copilot AI changed the title ~~[WIP] Consolidate all open pull requests into one~~ Consolidate open PRs: MIM inference, Hessian solver, gradient accumulation, and AMP Feb 6, 2026

Copilot AI requested a review from thinksyncs February 6, 2026 04:38

Copilot finished work on behalf of thinksyncs February 6, 2026 04:38

thinksyncs approved these changes Feb 6, 2026

View reviewed changes

thinksyncs marked this pull request as ready for review February 6, 2026 06:38

Copilot AI review requested due to automatic review settings February 6, 2026 06:38

Copilot started reviewing on behalf of thinksyncs February 6, 2026 06:38 View session

thinksyncs merged commit 4b11323 into main Feb 6, 2026
8 checks passed

Copilot AI reviewed Feb 6, 2026

View reviewed changes

thinksyncs deleted the copilot/combine-all-prs branch February 6, 2026 06:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate open PRs: MIM inference, Hessian solver, gradient accumulation, and AMP#10

Consolidate open PRs: MIM inference, Hessian solver, gradient accumulation, and AMP#10
thinksyncs merged 7 commits intomainfrom
copilot/combine-all-prs

Copilot AI commented Feb 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	self.assertTrue(loss.item() >= 0.0)
	self.assertGreaterEqual(loss.item(), 0.0)

	self.assertTrue(result["loss"].item() > 0.0)
	self.assertGreater(result["loss"].item(), 0.0)

Conversation

Copilot AI commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

PR #6: Masked reconstruction branch (1,270 lines)

PR #7: Hessian solver (1,319 lines)

PR #8: Gradient accumulation + AMP (252 lines)

PR #9: Lint documentation (50 lines)

Implementation notes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Feb 6, 2026 •

edited

Loading