Continuing an FSI simulation from a restart file stalls the solver

[ktbolt]

Description

I'm trying to continue an FSI simulation from a restart file using

 <Continue_previous_simulation> true </Continue_previous_simulation>
 <Restart_file_name> stFile </Restart_file_name>

The restart file appears to be read correctly and the simulation starts. However, the simulation seems to stall after the first iteration

---------------------------------------------------------------------
 Eq     N-i     T       dB  Ri/R1   Ri/R0    R/Ri     lsIt   dB  %t
---------------------------------------------------------------------
 FS 11-1  3.587e+00  [0 1.000e+00 1.778e+00 8.737e-13]  [147 -80 1]
 MS 11-1  3.595e+00  [0 1.000e+00 2.776e+01 8.312e-13]  [102 0 63]

Reproduction

Modify the FSI pipe_3d test problem to restart after 10 time steps.

Expected behavior

An FSI simulation should be able to continue from a restart file.

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines

[ktbolt]

The restart works for 1 and 2 processors, but stalls for 3

---------------------------------------------------------------------
 Eq     N-i     T       dB  Ri/R1   Ri/R0    R/Ri     lsIt   dB  %t
---------------------------------------------------------------------
 FS 11-1  1.834e+00  [0 1.000e+00 8.524e-01 8.759e-05]  [48 -93 1]
 MS 11-1  1.841e+00  [0 1.000e+00 2.862e+01 9.515e-05]  [34 -3 43]
 FS 11-2  1.878e+00  [-18 1.192e-01 1.016e-01 9.713e-05]  [40 -92 22]
 MS 11-2  1.886e+00  [-42 7.524e-03 2.153e-01 9.164e-05]  [34 -3 38]
 FS 11-3  1.923e+00  [-64 5.768e-04 4.916e-04 8.481e-05]  [39 -94 22]
 MS 11-3  1.931e+00  [-90 2.927e-05 8.378e-04 9.867e-05]  [34 -3 37]
 FS 11-4  1.994e+00  [-107 4.166e-06 3.551e-06 9.488e-05]  [41 -93 14]

stalling after step FS 11-4

[ktbolt]

I see that restarting an FSI simulation used to work; tested that using the svFSI-Tests/07-fsi/ale/03-pipe_3D files.

[ktbolt]

The problem is that in fsi_linear_solver::fsils_commuv when doing a data transfer using

MPI_Irecv(rB.col_data(i), lhs.cS[i].n, mpreal, lhs.cS[i].iP, mpi_tag, lhs.commu.comm, &rReq[i]);

and waiting for it to complete using

MPI_Wait(&rReq[i], &stat);

creates a deadlock.