Fix: Remove device_map to prevent meta tensor errors in table detection

[micmarty-deepsense]

Summary

Fixes NotImplementedError: Cannot copy out of meta tensor error in table detection during multi-threaded processing.

Root Cause: How device_map Creates Meta Tensors

The Problem

When passing device_map to from_pretrained(), HuggingFace Transformers uses a special loading path:

1. Forces low_cpu_mem_usage=True automatically (HF #33326)
2. Creates model on meta device first - placeholder tensors with NO actual data
3. Attempts to load state_dict onto meta tensors without assign=True (HF #37615)
4. Tries to move to target device with .to(device)
5. FAILS because meta tensors cannot be copied/moved (HF #26700)

From HF issue #33326:

"Tensors created on the meta device are meaningless empty tensors, which renders initialization code completely ineffective."

From HF issue #26700:

"When using device_map, the transformers library creates a context manager that sets the default device to 'meta'. During initialization, the code attempts to copy weights from the original modules. However, since the backbone was created on the meta device, the weights are not materialized, causing the copy operation to fail."

Two Different Code Paths in Transformers

With device_map (broken path):

from_pretrained(device_map="cpu")
  → Accelerate's distributed loading
    → Initialize on meta device
      → Load state_dict onto meta tensors
        → Try to move to CPU
          → ERROR: Cannot copy from meta tensor

Without device_map (working path):

from_pretrained()
  → Normal PyTorch loading
    → Initialize on CPU with real tensors
      → Load weights directly into memory
        → .to(device) moves real data
          → SUCCESS

Why device_map Exists

Originally designed for HUGE models (>100B params) that don't fit in memory. Our TableTransformer models (~500MB) don't need this optimization and shouldn't use it.

Changes

• Removed device_map from DetrImageProcessor.from_pretrained() (line 75)
• Removed device_map from TableTransformerForObjectDetection.from_pretrained() (lines 85-88)
• Added explicit .to(self.device) after model loading (line 87)

# BEFORE (broken)
self.model = TableTransformerForObjectDetection.from_pretrained(
    model,
    device_map=self.device,  # Forces meta device path
)

# AFTER (working)
self.model = TableTransformerForObjectDetection.from_pretrained(model)  # Normal loading
self.model.to(self.device)  # Move real tensors to device

Pattern Consistency

This matches the fix pattern used for SentenceTransformer models in core-product:

• c8b175f7: Added device parameter to model constructor
• db636932: Made thread-safe with @threadsafe_lazyproperty

Antonio's PR #446 addressed a different issue (thread-safety race condition), while this PR fixes the underlying meta tensor problem.

Testing

Error observed in in-vpc customer deployment with strategy=fast + table detection during OCR processing.

Stacktrace (from customer deployment)

tables_agent_patch.py:297 → patched_load_agent
  → tables.py:85 → initialize
    → TableTransformerForObjectDetection.from_pretrained()
      → model.to(device)  ← META TENSOR ERROR

References

• HF Issue #26700: device_map causes "Cannot copy out of meta tensor"
• HF Issue #37615: Object Detection models meta tensor warnings
• HF Issue #33326: low_cpu_mem_usage creates meaningless empty tensors