Tokens may expire during migration causing failure

[narasimha18sv]

Any migration able 1 TB are failing with container panic after 2-3 hours when the migrate command completes pushing the volumes.

[narasimha18sv]

For MigrateKit migrations involving disks larger than 1TB, you should allocate sufficient RAM and CPU to prevent failures. While there is no official documentation specifying exact requirements, here are some best practices based on typical VMware to OpenStack migration scenarios:

Recommended RAM & CPU for MigrateKit
Disk Size Recommended RAM Recommended CPU Cores
Up to 500GB 8GB+ 4 vCPUs
500GB - 1TB 16GB+ 6 vCPUs
1TB - 2TB 32GB+ 8 vCPUs
2TB - 4TB 64GB+ 12+ vCPUs
Above 4TB 128GB+ 16+ vCPUs

I just see these things as recommendations but don't see anywhere as an official documentation.

[narasimha18sv]

`Removing snapshot 100% [===================================] (100/100) [11s:0s]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xd8 pc=0xac5aa1]

goroutine 1 [running]:
github.com/vexxhost/migratekit/internal/target.(OpenStack).WriteChangeID(0xc00019cbe8, {0x1001758, 0xc0004bb710}, 0xc000478000)
/builds/syseng/rhosp/migratekit/internal/target/openstack.go:296 +0x81
github.com/vexxhost/migratekit/internal/vmware_nbdkit.(NbdkitServer).SyncToTarget(0xc000121ed8, {0x1001758, 0xc0004bb710}, {0x1003dd0, 0xc00019cbe8}, 0x0, 0xc000473cb0)
/builds/syseng/rhosp/migratekit/internal/vmware_nbdkit/vmware_nbdkit.go:578 +0x564
github.com/vexxhost/migratekit/internal/vmware_nbdkit.(NbdkitServers).MigrationCycle(0xc000488300, {0x1001758, 0xc0004bb710}, 0x0, 0xc000473cb0)
/builds/syseng/rhosp/migratekit/internal/vmware_nbdkit/vmware_nbdkit.go:363 +0x4bd
main.init.func2(0x164fee0?, {0xead89a?, 0x4?, 0xead89e?})
/builds/syseng/rhosp/migratekit/main.go:217 +0x114
github.com/spf13/cobra.(Command).execute(0x164fee0, {0xc000244f70, 0xd, 0xd})
/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaca
github.com/spf13/cobra.(Command).ExecuteC(0x164fc00)
/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(Command).Execute(...)
/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
main.main()
/builds/syseng/rhosp/migratekit/main.go:358 +0x1a

Error coming while trying for 4TB disks .

Even after making migratekit VM with 64GB of ram and 14 Cpu still getting the same issue

[mnaser]

That looks more like a bug than anything else. It seems that you have your own internal build however, so you will need to check, I suggest using Docker image.

[Thieum27]

@mnaser I have a same error but i'm using a Docker image .

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xd8 pc=0xafa25e]

goroutine 1 [running]:
github.com/vexxhost/migratekit/internal/target.(*OpenStack).WriteChangeID(0xc000558738, {0x104e438, 0xc0003f2750}, 0xc0004b0300)
/src/internal/target/openstack.go:330 +0x7e
github.com/vexxhost/migratekit/internal/vmware_nbdkit.(*NbdkitServer).SyncToTarget(0xc0005c2378, {0x104e438, 0xc0003f2750}, {0x1050970, 0xc000558738}, 0x0)
/src/internal/vmware_nbdkit/vmware_nbdkit.go:384 +0x636
github.com/vexxhost/migratekit/internal/vmware_nbdkit.(*NbdkitServers).MigrationCycle(0xc0003f6360, {0x104e438, 0xc0003f2750}, 0x0)
/src/internal/vmware_nbdkit/vmware_nbdkit.go:197 +0x16f
main.init.func2(0x16bee60?, {0xed9477?, 0x4?, 0xed947b?})
/src/main.go:226 +0xee
github.com/spf13/cobra.(*Command).execute(0x16bee60, {0xc0000fd080, 0xb, 0xb})
/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0x16beb80)
/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
/root/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
main.main()

I can confirm that the issue also occurs when using a configuration with 32 vCPUs, 64 GB of RAM, and a 1TB disk

[mnaser]

@Thieum27 It seems the issue here stems from the following line:

https://github.com/vexxhost/migratekit/blob/main/internal/target/openstack.go#L330

It seems for some reason, the metadata field is coming as nil instead of an empty map, I assume some error handling needs to be done there but we're quite busy with customer work to push a fix. We're happy to review a PR or if this is bubbled up from a customer we can try and act on it.

[Thieum27]

It seems for some reason, the metadata field is coming as nil instead of an empty map, I assume some error handling needs to be done there but we're quite busy with customer work to push a fix. We're happy to review a PR or if this is bubbled up from a customer we can try and act on it.

Hi ,

Thanks for the clarification! I'm also definitely interested in a fix if one becomes available.

[michael-salo]

You can definitely move bigger disks with the over 1TiB, we've moved VMs with disk size over 3TiB.

We have not seen this problem using migratekit, worth noting that we're not running v2v during migration.

--run-v2v=False

Our migrator VM has got only 16GiB of RAM and 8 VPCUs.

[Thieum27]

You can definitely move bigger disks with the over 1TiB, we've moved VMs with disk size over 3TiB.

We have not seen this problem using migratekit, worth noting that we're not running v2v during migration.

--run-v2v=False

Our migrator VM has got only 16GiB of RAM and 8 VPCUs.

Thanks a lot @michael-salo for your quick reply, I really appreciate it.

Just to clarify — V2V is intentionally skipped during the initial migration phase, and only executed at cutover time, correct?

[michael-salo]

@Thieum27 , we don't use V2V, we set it to False as all VMs that we were moving had got required virtio drivers installed. Yes, you're correct, V2V is only executed at cutover time, in our use case it's set to False.

[Thieum27]

@michael-salo I'm getting back to you regarding our migration issue — we finally identified the root cause!
It was a token timeout in OpenStack, which was originally set to 1 hour. We've now increased it to 6 hours to allow enough time for disks larger than 2TB to complete their migration.

I hope this information will be helpful to others as well.

Cheers!

[mnaser]

Cool, thanks for sharing that. I think we'll have to make sure someone would adjust the code to handle the token expiration before requests are made.