gc.get_objects() in dump_allocated_tensors() can return weakref proxy objects. A proxy passes isinstance(obj, torch.Tensor) (forwarded __class__), but if the referent is collected before .numel() / .element_size() / .shape are accessed, a ReferenceError is raised.
This is called from Profiler.record_allocated_tensors() during training (training.py:502, 544), so a crash here interrupts a running experiment.
Fix: wrap tensor attribute access in try/except ReferenceError.
gc.get_objects()indump_allocated_tensors()can return weakref proxy objects. A proxy passesisinstance(obj, torch.Tensor)(forwarded__class__), but if the referent is collected before.numel()/.element_size()/.shapeare accessed, aReferenceErroris raised.This is called from
Profiler.record_allocated_tensors()during training (training.py:502, 544), so a crash here interrupts a running experiment.Fix: wrap tensor attribute access in
try/except ReferenceError.