GPU inference?

[cleong110]

Is there a way to run this with GPU acceleration? I tried a number of different ways to run the models on GPU, but no matter what I do I still get errors like

RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu

It seems that you need to save a different traced .pth?
https://docs.pytorch.org/docs/stable/jit.html#frequently-asked-questions seems to suggest that one must load the model architecture, "trace" it on a GPU and then save that.

Full traceback:

Traceback of TorchScript, serialized code (most recent call last):                                                                                                                            
  File "code/__torch__/pose_to_segments/src/model.py", line 22, in forward                                                                                                                    
    seq_length = ops.prim.NumToTensor(torch.size(pose_data, 1))                                                                                                                               
    input = torch.reshape(pose_data, [_0, int(seq_length), -1])                                                                                                                               
    _1 = (encoder).forward((pose_projection).forward(input, ), )                                                                                                                              
          ~~~~~~~~~~~~~~~~ <--- HERE                                                                                                                                                          
    _2 = (sign_bio_head).forward(_1, )                                                                                                                                                        
    _3 = (sentence_bio_head).forward(_1, )                                                                                                                                                    
  File "code/__torch__/torch/nn/modules/rnn.py", line 78, in forward                                                                                                                          
    _1 = [hx, hx0]                                                                                                                                                                            
    _2 = [weight_ih_l0, weight_hh_l0, bias_ih_l0, bias_hh_l0, weight_ih_l0_reverse, weight_hh_l0_reverse, bias_ih_l0_reverse, bias_hh_l0_reverse, weight_ih_l1, weight_hh_l1, bias_ih_l1, bias
_hh_l1, weight_ih_l1_reverse, weight_hh_l1_reverse, bias_ih_l1_reverse, bias_hh_l1_reverse, weight_ih_l2, weight_hh_l2, bias_ih_l2, bias_hh_l2, weight_ih_l2_reverse, weight_hh_l2_reverse, bi
as_ih_l2_reverse, bias_hh_l2_reverse, weight_ih_l3, weight_hh_l3, bias_ih_l3, bias_hh_l3, weight_ih_l3_reverse, weight_hh_l3_reverse, bias_ih_l3_reverse, bias_hh_l3_reverse]                 
    input, _3, _4 = torch.lstm(argument_1, _1, _2, True, 4, 0., True, True, True)                                                                                                             
                    ~~~~~~~~~~ <--- HERE                                                                                                                                                      
    return input                                                                                                                                                                              
                                                                                                                                                                                              
Traceback of TorchScript, original code (most recent call last):                                                                                                                              
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/rnn.py(812): forward                                                                                          
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1488): _slow_forward                                                                                
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1501): _call_impl                                                                                   
/home/ubuntu/workspace/transcription/pose_to_segments/src/model.py(140): forward                                                                                                              
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1488): _slow_forward                                                                                
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1501): _call_impl                                                                                   
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/jit/_trace.py(1056): trace_module                                                                                        
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/jit/_trace.py(794): trace                                                                                                /home/ubuntu/workspace/transcription/pose_to_segments/src/train.py(153): <module>                                                                                                             
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/runpy.py(86): _run_code                                                                                                                      
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/runpy.py(196): _run_module_as_main
RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu

[AmitMY]

and we can't model.to('cuda:0')?

[cleong110]

Right, exactly. You'd think so, but then you get problems with "Hidden Tensors".

ERROR:root:Failed processing foo/CBS-105-ker-4-More_Info _ Christ and Church Relationship as Example.pose: The following operation failed in the TorchScript interpreter.                     
Traceback of TorchScript, serialized code (most recent call last):                                                                                                                            
  File "code/__torch__/pose_to_segments/src/model.py", line 22, in forward                                                                                                                    
    seq_length = ops.prim.NumToTensor(torch.size(pose_data, 1))                                                                                                                               
    input = torch.reshape(pose_data, [_0, int(seq_length), -1])                                                                                                                               
    _1 = (encoder).forward((pose_projection).forward(input, ), )                                                                                                                              
          ~~~~~~~~~~~~~~~~ <--- HERE                                                                                                                                                          
    _2 = (sign_bio_head).forward(_1, )                                                                                                                                                        
    _3 = (sentence_bio_head).forward(_1, )                                                                                                                                                    
  File "code/__torch__/torch/nn/modules/rnn.py", line 78, in forward                                                                                                                          
    _1 = [hx, hx0]                                                                                                                                                                            
    _2 = [weight_ih_l0, weight_hh_l0, bias_ih_l0, bias_hh_l0, weight_ih_l0_reverse, weight_hh_l0_reverse, bias_ih_l0_reverse, bias_hh_l0_reverse, weight_ih_l1, weight_hh_l1, bias_ih_l1, bias
_hh_l1, weight_ih_l1_reverse, weight_hh_l1_reverse, bias_ih_l1_reverse, bias_hh_l1_reverse, weight_ih_l2, weight_hh_l2, bias_ih_l2, bias_hh_l2, weight_ih_l2_reverse, weight_hh_l2_reverse, bi
as_ih_l2_reverse, bias_hh_l2_reverse, weight_ih_l3, weight_hh_l3, bias_ih_l3, bias_hh_l3, weight_ih_l3_reverse, weight_hh_l3_reverse, bias_ih_l3_reverse, bias_hh_l3_reverse]                 
    input, _3, _4 = torch.lstm(argument_1, _1, _2, True, 4, 0., True, True, True)                                                                                                             
                    ~~~~~~~~~~ <--- HERE                                                                                                                                                      
    return input                                                                                                                                                                              
                                                                                                                                                                                              
Traceback of TorchScript, original code (most recent call last):                                                                                                                              
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/rnn.py(812): forward                                                                                          
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1488): _slow_forward                                                                                
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1501): _call_impl                                                                                   
/home/ubuntu/workspace/transcription/pose_to_segments/src/model.py(140): forward              
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/nn/modules/module.py(1501): _call_impl
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/jit/_trace.py(1056): trace_module
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/site-packages/torch/jit/_trace.py(794): trace                                                                                                
/home/ubuntu/workspace/transcription/pose_to_segments/src/train.py(153): <module>
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/runpy.py(86): _run_code                                                                                                                      
/home/ubuntu/anaconda3/envs/trans/lib/python3.10/runpy.py(196): _run_module_as_main                                                                                                           
RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu

I wasn't able to fix this

[cleong110]

Example: I modify these two in bin.py..

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")


@lru_cache(maxsize=1)
def load_model(model_path: str):
    model = torch.jit.load(model_path, map_location=DEVICE)
    model.eval()
    model.to(DEVICE)
    print(f"Loaded model {model_path} on {DEVICE}")
    return model


def predict(model, pose: Pose):
    with torch.no_grad():
        torch_body = pose.body.torch()
        pose_data = torch_body.data.tensor[:, 0, :, :].unsqueeze(0).to(DEVICE)
        return model(pose_data)

plus set verbose to true, and get:

Loading model model_E1s-1.pth ...
Loaded model /opt/home/cleong/projects/semantic_and_visual_similarity/segmentation/sign_language_segmentation/dist/model_E1s-1.pth on cuda
Estimating segments, pose is 424 frames ...
Traceback (most recent call last):
  File "/opt/home/cleong/projects/semantic_and_visual_similarity/segmentation/sign_language_segmentation/bin.py", line 221, in <module>
    main()
    ~~~~^^
  File "/opt/home/cleong/projects/semantic_and_visual_similarity/segmentation/sign_language_segmentation/bin.py", line 186, in main
    eaf, tiers = segment_pose(pose, model=args.model, verbose=args.verbose)
                 ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/home/cleong/projects/semantic_and_visual_similarity/segmentation/sign_language_segmentation/bin.py", line 149, in segment_pose
    probs = predict(model, pose)
  File "/opt/home/cleong/projects/semantic_and_visual_similarity/segmentation/sign_language_segmentation/bin.py", line 74, in predict
    return model(pose_data)
  File "/opt/home/cleong/envs/pose_segmentation_src/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/home/cleong/envs/pose_segmentation_src/lib/python3.13/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/pose_to_segments/src/model.py", line 22, in forward
    seq_length = ops.prim.NumToTensor(torch.size(pose_data, 1))
    input = torch.reshape(pose_data, [_0, int(seq_length), -1])
    _1 = (encoder).forward((pose_projection).forward(input, ), )
          ~~~~~~~~~~~~~~~~ <--- HERE
    _2 = (sign_bio_head).forward(_1, )
    _3 = (sentence_bio_head).forward(_1, )
  File "code/__torch__/torch/nn/modules/rnn.py", line 78, in forward
    _1 = [hx, hx0]
    _2 = [weight_ih_l0, weight_hh_l0, bias_ih_l0, bias_hh_l0, weight_ih_l0_reverse, weight_hh_l0_reverse, bias_ih_l0_reverse, bias_hh_l0_reverse, weight_ih_l1, weight_hh_l1, bias_ih_l1, bias_hh_l1, weight_ih_l1_reverse, weight_hh_l1_reverse, bias_ih_l1_reverse, bias_hh_l1_reverse, weight_ih_l2, weight_hh_l2, bias_ih_l2, bias_hh_l2, weight_ih_l2_reverse, weight_hh_l2_reverse, bias_ih_l2_reverse, bias_hh_l2_reverse, weight_ih_l3, weight_hh_l3, bias_ih_l3, bias_hh_l3, weight_ih_l3_reverse, weight_hh_l3_reverse, bias_ih_l3_reverse, bias_hh_l3_reverse]
    input, _3, _4 = torch.lstm(argument_1, _1, _2, True, 4, 0., True, True, True)
                    ~~~~~~~~~~ <--- HERE
    return input

Traceback of TorchScript, original code (most recent call last):
/home/zifjia/data/conda/envs/trans/lib/python3.9/site-packages/torch/nn/modules/rnn.py(774): forward
/home/zifjia/data/conda/envs/trans/lib/python3.9/site-packages/torch/nn/modules/module.py(1182): _slow_forward
/home/zifjia/data/conda/envs/trans/lib/python3.9/site-packages/torch/nn/modules/module.py(1194): _call_impl
/home/zifjia/transcription/pose_to_segments/src/model.py(140): forward
/home/zifjia/data/conda/envs/trans/lib/python3.9/site-packages/torch/nn/modules/module.py(1182): _slow_forward
/home/zifjia/data/conda/envs/trans/lib/python3.9/site-packages/torch/nn/modules/module.py(1194): _call_impl
/home/zifjia/data/conda/envs/trans/lib/python3.9/site-packages/torch/jit/_trace.py(976): trace_module
/home/zifjia/data/conda/envs/trans/lib/python3.9/site-packages/torch/jit/_trace.py(759): trace
/home/zifjia/transcription/pose_to_segments/src/train.py(153): <module>
/home/zifjia/data/conda/envs/trans/lib/python3.9/runpy.py(87): _run_code
/home/zifjia/data/conda/envs/trans/lib/python3.9/runpy.py(197): _run_module_as_main
RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu

I also tried "cuda:0" instead of "cuda" to no avail, same error

[cleong110]

segmentation_pip_list.txt
Relevant pip versions

[cleong110]

OK, tried on another compute node with no GPU, just cpu, grabbed 22 sign-bible pose files at random, here's the length of them extracted by pose_info -i|grep Duration

Duration (seconds): 388.12
Duration (seconds): 188.12
Duration (seconds): 46.504790402679575
Duration (seconds): 593.8
Duration (seconds): 14.147466732328102
Duration (seconds): 409.94286856929966
Duration (seconds): 25.458766784826278
Duration (seconds): 601.1338694566582
Duration (seconds): 1.8685316771697926
Duration (seconds): 0.33366550028505093
Duration (seconds): 194.54

On the CPU node with "AMD EPYC 7702 64-Core Processor" cpus

2025-07-21 14:46:32,044 INFO Total time: 287.82 seconds
2025-07-21 14:46:32,044 INFO Processed 22 files, skipped 0    
2025-07-21 14:46:32,044 INFO Per-model timings:                   
2025-07-21 14:46:32,044 INFO   model_E4s-1.pth: 149.54 seconds      
2025-07-21 14:46:32,044 INFO   model_E1s-1.pth: 138.28 seconds   

During this process, CPU usage jumped up to about 77 CPUs

On the GPU node with 16 Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz cpus, it maxes out every single one and is still running.

So it seems the bottleneck is CPUs actually