ZeroDivisionError from f1 function

[willystumblr]

Hi,

I'm continuously facing an issue during an epoch when running train_baselines.py with following arguments:

python train_baselines.py --config-name closed model=binary_dpr_nl seed=1 project=local-closed-1

It says that ZeroDivisionError occurred from model/seq2seq.py, especially when calling f1 function. Below is the error message:

  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/pytorch_lightning/overrides/base.py", line 110, in forward
    return self._forward_module.validation_step(*inputs, **kwargs)
  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/wrangl/learn/supervised.py", line 196, in validation_step
    metrics = self.compute_metrics(pred, gold)
  File "/home/minseo/SemanticFiltering/romqa/model/seq2seq.py", line 81, in compute_metrics
    m = {k: v for k, v in f1(pred, gold, ignore_empty=True).items()}
  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/wrangl/learn/metrics.py", line 78, in __call__
    return {k: sum(v)/len(v) for k, v in metrics.items()}
  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/wrangl/learn/metrics.py", line 78, in <dictcomp>
    return {k: sum(v)/len(v) for k, v in metrics.items()}
ZeroDivisionError: division by zero

I guess it occurs because the score evaluates to zero at some point in the train set, but I wonder why it happens. Would you check this issue? Thank you in advance.

PS: Just in case this might help explain the issue, I slightly increased the value of batch size, gpus from classify.yaml and added strategy='ddp' option to pl.Trainer.

[vzhong]

would you mind helping me pull out the example that is causing this? for example can you pdb this and trace the point of exception and print out the example and paste it here? I was under the impression that examples with no gold answers have been removed during training.

[vzhong]

Actually it seems like the zero is the len(v), which suggests that somehow your batch has no examples in it (as opposed to some score evaluating to zero).

[willystumblr]

It seems that this occurs during the validation steps in the training process.

Epoch 0:  32%|█▉    | 250/780 [03:47<08:01,  1.10it/s, loss=0.582, v_num=59]
Validation DataLoader 0:   0%|            ]

The above is the printed output when the error occurs.
And I figured out that out variable below becomes the list of 0s:

#wrangl/learn/supervised.py
out = self.infer(feat, batch)
pred = self.extract_pred(out, feat, batch) #*******

metrics = self.compute_metrics(pred, gold)
    for k, v in metrics.items():

This is the screenshot of the debugging process:

Because we have limited resources (gpus, etc.), we train the model on the sample training set. I guess that's why the error occurred.

[vzhong]

I don't understand how the debugging trace can result in a divide by zero error. In your original trace:

  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/pytorch_lightning/overrides/base.py", line 110, in forward
    return self._forward_module.validation_step(*inputs, **kwargs)
  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/wrangl/learn/supervised.py", line 196, in validation_step
    metrics = self.compute_metrics(pred, gold)
  File "/home/minseo/SemanticFiltering/romqa/model/seq2seq.py", line 81, in compute_metrics
    m = {k: v for k, v in f1(pred, gold, ignore_empty=True).items()}
  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/wrangl/learn/metrics.py", line 78, in __call__
    return {k: sum(v)/len(v) for k, v in metrics.items()}
  File "/home/minseo/.conda/envs/RoMQA-3.7/lib/python3.7/site-packages/wrangl/learn/metrics.py", line 78, in <dictcomp>
    return {k: sum(v)/len(v) for k, v in metrics.items()}
ZeroDivisionError: division by zero

The division by zero is happening in len(v). I'm not sure out is helpful in debugging this error. In you new trace, what is in metrics? In particular for what k is v empty?