[MIG-663] Migrations should not block on a single bad DVM dependency with multi-PVCs

[eriknelson]

Raised by @alaypatel07 and tracked here: https://issues.redhat.com/browse/MIG-663

I'm opening this issue so a solution can be agreed upon here @alaypatel07 / @pranavgaikwad to open enhancement doc for discussion; a PR can close this issue once implemented.

The issue:

If DVM controller is not able to process the dependency of the rsync transfer on any one PVC the entire DVM CR is blocked in the itinerary and hence the migmgiration

An example from the BZ would be quotas, we recommend 10 namespaces in a single plan on any migration at scale, if any one of the namespaces has quota problems all the 9 namespaces will be blocked

In order to solve this, I propose the following.

• Add a status field to the DVM CR called skippedPVCs containing a list of PVC Ref
• For all places where we create dependency pods if the requests are not server by quota, find that error and add the PVCs in that namespace on the skippedPVCs
• For the next phases in the itinerary just look at the remaining PVC and try to migrate them
• if all the other PVCs complete rsync transfer, make this a partially failed DVM, and migmigration will complete with warnings

This gives the user the ability to migrate the remaining resources, identify the failing PVCs and retry if needed. This also allows us to add more logic in the future(if needed) to make other cases of skipping a PVC.

Resources to identify the error:

apierrors.IsForbidden(err) && strings.Contains(fmt.Sprintf("%#v", err), "quota"), will do the trick see the error code 403 and message in the following debugger snippet.