Crash when no matches are found

[tsattler]

Hi Qunjie,

Thanks for releasing the code!

I observed a test-time crash that happens when there are no matches found. The trace is

  File "../localize.py", line 385, in main
    matches, _, _, _ = matcher(img1_name, img2_name)
  File "../localize.py", line 314, in <lambda>
    matcher = lambda im1, im2: model.match_pairs(im1, im2)
  File "/local/datasets/aachen_day_night/temp/image-matching-toolbox/immatch/modules/patch2pix.py", line 86, in match_pairs
    io_thres=self.match_threshold)        
  File "/local/datasets/aachen_day_night/temp/image-matching-toolbox/immatch/modules/../../third_party/patch2pix/networks/patch2pix.py", line 296, in refine_matches
    regressor=self.regress_mid)
  File "/local/datasets/aachen_day_night/temp/image-matching-toolbox/immatch/modules/../../third_party/patch2pix/networks/patch2pix.py", line 215, in forward_fine_match
    psize, ptype, regressor)
  File "/local/datasets/aachen_day_night/temp/image-matching-toolbox/immatch/modules/../../third_party/patch2pix/networks/patch2pix.py", line 177, in forward_fine_match_mini_batch
    f1s = f1s.view(-1, N, psize, psize).permute(1, 0, 2, 3)
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0, 16, 16] because the unspecified dimension size -1 can be any value and is ambiguous

Here, I am using your image matching toolbox to use patch2pix with superglue features to match two images.
Do you know what the best fix would be?

[GrumpyZhou]

Hi Torsten,

I pushed some bugfix code, can you try to see whether the problem is fixed now?

[tsattler]

Hi Qunjie,

Many thanks. I can run Patch2Pix both with and without SuperGlue in the aachen_v1.1 configuration. If I run Patch2Pix with SuperGlue in the default configuration, I get the error that

File "/local/datasets/aachen_day_night/temp/image-matching-toolbox/immatch/modules/patch2pix.py", line 62, in __init__
    cargs = args['coarse']
KeyError: 'coarse'

Another question: I observed for Patch2Pix (without SuperGlue) that sometimes the x- or y-coordinate of a feature can be beyond the image size (e.g., I have an image with 800 pixels in the x-direction and get a feature with a x-coordinate of 800). Is this a potential bug? Or an indexing (1-based vs. 0-based) convention?

[GrumpyZhou]

Hi Torsten,

So in patch2pix_superglue.yml, there are default and coarse_default configs:

default: &default
    class: 'Patch2PixRefined'
    ckpt: 'pretrained/patch2pix/patch2pix_pretrained.pth'
    imsize: 1024
coarse_default: &coarse_default
    name: 'SuperGlue'
    weights: 'outdoor'    
    sinkhorn_iterations: 50
    max_keypoints: 4096
    nms_radius: 4
    imsize: 1024

And they define the followings:

• default defines the default setting of patch2pix
• coarse_default defines the coarse model and in this case it is superglue

So if you want to use patch2pix with superglue on another dataset, you will need to define a config for that dataset.
Take the format of aachen_v1.1:

$dataset_name$:
    # Basic settings about models
    <<: *default
    match_threshold: 0.*
    coarse:
        <<: *coarse_default
        match_threshold: 0.1  
    # Then some dataset-specific settings
    pairs: [...]
    qt_dthres: 4
    qt_psize: 48
    qt_unique: True
    ransac_thres: [20]
    sc_thres: 0.25 # Filtering during quantization

I am not sure this answers your 1st question.
Btw, can you also share this issue to image-matching-toolbox?
I feel this answer might also be interesting to other users there.

Regarding the 2nd question, so the feature should be 0-index based. So if your image is 800x800, the x and y should be <=799. I feel this might be a bug somewhere. Could you provide the sample so that i can reproduce this for debugging?
Thanks for reporting this problem 😁.

[tsattler]

Thanks for the detailed answer. This answers my first question. Maybe this is also something to put on the website?

I will try to share the example per email. If you don't hear back from me in the next two weeks, please ping me.