How to predict on the test dataset using trainer.predict()?

[karndeepsingh]

Hi I have trained the model using trainer and was trying to use trainer.predict() method to predict on the datamodule. But it throws the following error:

MisconfigurationException                 Traceback (most recent call last)
Input In [200], in <cell line: 1>()
----> 1 trainer.predict(dataloaders=datamodule)

File /opt/conda/envs/pytorch-lightning/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1025, in Trainer.predict(self, model, dataloaders, datamodule, return_predictions, ckpt_path)
   1000 r"""
   1001 Run inference on your data.
   1002 This will call the model forward function to compute predictions. Useful to perform distributed
   (...)
   1022     Returns a list of dictionaries, one for each provided dataloader containing their respective predictions.
   1023 """
   1024 self.strategy.model = model or self.lightning_module
-> 1025 return self._call_and_handle_interrupt(
   1026     self._predict_impl, model, dataloaders, datamodule, return_predictions, ckpt_path
   1027 )

File /opt/conda/envs/pytorch-lightning/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:723, in Trainer._call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)
    721         return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
    722     else:
--> 723         return trainer_fn(*args, **kwargs)
    724 # TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7
    725 except KeyboardInterrupt as exception:

File /opt/conda/envs/pytorch-lightning/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1072, in Trainer._predict_impl(self, model, dataloaders, datamodule, return_predictions, ckpt_path)
   1066 self._ckpt_path = self.__set_ckpt_path(
   1067     ckpt_path, model_provided=model_provided, model_connected=self.lightning_module is not None
   1068 )
   1070 self._predicted_ckpt_path = self.ckpt_path  # TODO: remove in v1.8
-> 1072 results = self._run(model, ckpt_path=self.ckpt_path)
   1074 assert self.state.stopped
   1075 self.predicting = False

File /opt/conda/envs/pytorch-lightning/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1160, in Trainer._run(self, model, ckpt_path)
   1157 self._callback_connector._attach_model_callbacks()
   1158 self._callback_connector._attach_model_logging_functions()
-> 1160 verify_loop_configurations(self)
   1162 # hook
   1163 log.detail(f"{self.__class__.__name__}: preparing data")

File /opt/conda/envs/pytorch-lightning/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py:48, in verify_loop_configurations(trainer)
     46     __verify_eval_loop_configuration(trainer, model, "test")
     47 elif trainer.state.fn == TrainerFn.PREDICTING:
---> 48     __verify_eval_loop_configuration(trainer, model, "predict")
     50 __verify_dp_batch_transfer_support(trainer, model)
     51 _check_add_get_queue(model)

File /opt/conda/envs/pytorch-lightning/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py:197, in __verify_eval_loop_configuration(trainer, model, stage)
    193 # -----------------------------------
    194 # verify model has an eval_dataloader
    195 # -----------------------------------
    196 if not has_loader:
--> 197     raise MisconfigurationException(f"No `{loader_name}()` method defined to run `Trainer.{trainer_method}`.")
    199 # predict_step is not required to be overridden
    200 if stage == "predict":

MisconfigurationException: No `predict_dataloader()` method defined to run `Trainer.predict`.

I have the following dataloader:

class ProductDatasetModule(pl.LightningDataModule):
    def __init__(self,train_data,test_data,tokenizer,train_transforms,test_transforms,batch_size):
        super().__init__()
        self.train_data = train_data
        self.test_data = test_data
        self.tokenizer = tokenizer
        self.batch_size= batch_size
        self.train_transforms = train_transforms
        self.test_transforms = test_transforms
        
    def setup(self,stage=None):
        self.train_dataset = ImageTextDataset(self.train_data,tokenizer=self.tokenizer,transforms=self.train_transforms)
        self.test_dataset = ImageTextDataset(self.test_data,tokenizer=self.tokenizer,transforms=self.test_transforms)
        
    def train_dataloader(self):
        return torch.utils.data.DataLoader(
        self.train_dataset,
        batch_size=self.batch_size,
        num_workers=4,
        shuffle=True)
    
    def test_dataloader(self):
        
        return torch.utils.data.DataLoader(
        self.test_dataset,
        batch_size=self.batch_size,
        num_workers=4,
        shuffle=False)

    def val_dataloader(self):
        return torch.utils.data.DataLoader(
        self.test_dataset,
        batch_size=self.batch_size,
        num_workers=4,
        shuffle=False)
        

I have following model defined:

class MLP(pl.LightningModule):
    """
    Creates a MTL model with the encoder from "arch" and with dropout multiplier ps.
    """
    def __init__(self, model,input_embedding_dim,output_dim,learning_rate,batch_size):
        super(MLP,self).__init__()
        self.encoder = model
        for param in model.parameters():
            param.requires_grad = False
        self.batch_size=batch_size
        self.projection = nn.Linear(input_embedding_dim, 256)
        # self.layernorm = nn.LayerNorm()
        self.gelu = nn.ReLU()
        self.fc = nn.Linear(256, 256)
        self.dropout = nn.Dropout(0.15)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        # self.fc2 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(64,output_dim)
        self.criterion = nn.CrossEntropyLoss()
        self.learning_rate = learning_rate
        self.train_acc = torchmetrics.Accuracy()
        self.train_f1 = torchmetrics.F1Score(number_classes=25,
        average="micro")
        self.train_auroc = torchmetrics.AUROC(number_classes=25,
        average="micro")
        self.val_acc = torchmetrics.Accuracy()
        self.val_f1 = torchmetrics.F1Score(number_classes=25,
        average="micro")
        self.val_auroc = torchmetrics.AUROC(number_classes=25,
        average="micro")
        #self.layer_norm = nn.LayerNorm(output_dim)

    def forward(self,x):
        image_features = self.encoder.image_encoder(x["image"].to(CFG.device))
        image_embeddings = self.encoder.image_projection(image_features)
        text_features = self.encoder.text_encoder(
        input_ids=x["input_ids"], attention_mask=x["attention_mask"])
        text_embeddings = self.encoder.text_projection(text_features)
        img_txt_embeddings = torch.cat((image_embeddings,text_embeddings),dim=1)
        projected = self.projection(img_txt_embeddings)
        # z = self.layernorm(z)
        z= self.gelu(projected)
        z= self.fc(z)
        # z=self.layernorm(z)
        z= self.gelu(z)
        z= self.dropout(z)
        z= self.fc2(z)
        # z=self.layernorm(z)
        z= self.gelu(z)
        z= self.dropout(z)
        z= self.fc3(z)
        z= self.gelu(z)
        z= self.fc4(z)
        
        
        loss = self.criterion(z,x["label"])
        return loss,z
    
    def training_step(self, batch, batch_idx):
        y=batch["label"]
        loss,y_pred= self(batch)
        y_pred = y_pred.softmax(dim=-1)
        # accumulate and return metrics for logging
        acc = self.train_acc(y_pred, y)
        f1 = self.train_f1(y_pred, y)
        # just accumulate
        self.train_auroc.update(y_pred, y)
        self.log("train_accuracy", acc,prog_bar=True, on_step=True, on_epoch=True,logger=True)
        self.log("train_f1", f1,prog_bar=True, on_step=True, on_epoch=True,logger=True)
        self.log("train_loss",loss, prog_bar=True, on_step=True, on_epoch=True,logger=True,batch_size=self.batch_size)
        return loss
    def validation_step(self, batch, batch_idx):
        y=batch["label"]
        loss,y_pred= self(batch)
        y_pred = y_pred.softmax(dim=-1)
        self.val_acc.update(y_pred, y)
        self.val_f1.update(y_pred, y)
        self.val_auroc.update(y_pred, y)
        self.log("val_accuracy",self.val_acc,prog_bar=True, on_step=True, on_epoch=True,logger=True)
        self.log("val_f1",self.val_f1,prog_bar=True, on_step=True, on_epoch=True,logger=True)
        self.log("val_loss", loss, prog_bar=True,on_step=True, on_epoch=True, logger=True,batch_size=self.batch_size)
        return loss
    def test_step(self, batch, batch_idx):
        y=batch["label"]
        loss,y_pred= self(batch)
        y_pred = y_pred.softmax(dim=-1)
        self.val_acc.update(y_pred, y)
        self.val_f1.update(y_pred, y)
        self.val_auroc.update(y_pred, y)
        self.log("test_accuracy",self.val_acc,prog_bar=True, on_step=True, on_epoch=True,logger=True)
        self.log("test_f1",self.val_f1,prog_bar=True, on_step=True, on_epoch=True,logger=True)
        self.log("test_loss", loss, prog_bar=True, on_step=True, on_epoch=True,logger=True,batch_size=self.batch_size)
        return loss
    def predict_step(self, batch, batch_idx, dataloader_idx=0):
        y=batch["label"]
        loss,y_pred= self(x)
        return y_pred.softmax(dim=-1)
    def configure_optimizers(self):
        optimizer = torch.optim.AdamW(self.parameters(),lr= self.learning_rate)
        return optimizer

Please, help to predict on the testdata. How I can leverage trainer to predict on the test data and get classification report for the predicted output?

Thanks

[akihironitta]

@karndeepsingh To use Trainer.predict(), You must have predict_dataloader() defined in your LightningModule or LightningDataModule as the error message states:

MisconfigurationException: No `predict_dataloader()` method defined to run `Trainer.predict`.

If you'd like to run inference on your test set, you just need to define predict_dataloader() with your test set:

    def predict_dataloader(self):
        return torch.utils.data.DataLoader(
        self.test_dataset,
        batch_size=self.batch_size,
        num_workers=4,
        shuffle=False)

[karndeepsingh]

@karndeepsingh To use Trainer.predict(), You must have predict_dataloader() defined in your LightningModule or LightningDataModule as the error message states:

MisconfigurationException: No `predict_dataloader()` method defined to run `Trainer.predict`.

If you'd like to run inference on your test set, you just need to define predict_dataloader() with your test set:

    def predict_dataloader(self):
        return torch.utils.data.DataLoader(
        self.test_dataset,
        batch_size=self.batch_size,
        num_workers=4,
        shuffle=False)

@akihironitta Thanks for replying. I also had a doubt on how I can re start the training from last checkpoint? Please guide me on process of re-training the model.

Thanks

[akihironitta]

how I can re start the training from last checkpoint? Please guide me on process of re-training the model.

@karndeepsingh Since it's off-topic to this discussion, could you create another discussion to ask your particular question on resuming training from a checkpoint?