Why does validation epoch end before train epoch?

[march-o]

It seems that validation_epoch_end gets called and only after that train_epoch_end gets called. Why is that?

[akihironitta]

When you run trainer.fit(...), there're two calls to validation_step() followed by a call to validation_epoch_end() before any calls to training_step() by default.

The order of all hooks are documented at: https://pytorch-lightning.readthedocs.io/en/1.7.2/common/lightning_module.html#hooks

# the sanity check runs here <--- mind that here's **SANITY CHECK**

on_train_start()
for epoch in epochs:
    fit_loop()
on_train_end()

You can disable it by specifying Trainer(num_sanity_val_steps=0): https://pytorch-lightning.readthedocs.io/en/1.7.2/common/trainer.html#num-sanity-val-steps

[march-o]

Thanks very much for the reply. I had already disabled the sanity check, and knew about its procedure. But what I did not know, from the documentation, is that the training loop goes like: train_epoch_start() -> valid_epoch_start() -> valid_eopch_end() -> train_epoch_end(). Kinda weird and messes up all of my metric functions if I want to reuse them for train and valid. Also, could you maybe drop a link to some great lightning examples with callback use and torchmetrics use. Like, currently I am returning the prediction(segmentation) in traing_step end, so that I can calculate metrics in callbacks, but that I presume stores the prediction which is bad, because my model is crashing with a ram mem leak.

[akihironitta]

Kinda weird and messes up all of my metric functions if I want to reuse them for train and valid.

There's a trainer property called trainer.state.stage you might find useful to handle your situation.
https://pytorch-lightning.readthedocs.io/en/1.7.2/common/trainer.html#state

if self.trainer.state.stage != "sanity_check":
    # run something you don't want to run during sanity checking.
    do_something()

Also, could you maybe drop a link to some great lightning examples with callback use and torchmetrics use.

Examples are usually located in examples/ directory of each repository. Would you mind opening a new discussion or issue with your thoughts on what's missing and what you'd like to see in our examples?