Crucible backend needs a mechanism to be reconfigured before restarting after failed LM

[gjcolombo]

Hypothesized repro steps:

1. Launch a VM that connects to a set of Crucible downstairs with Crucible generation 1.
2. Start a migration target that will connect to the same downstairs with Crucible generation 2.
3. Inject a failure into migration immediately after the target activates (note that this can happen even after Migration: creating a migration-target Crucible upstairs prevents (or may prevent) the source from pausing #155 is fixed if there is any way for the target to fail to start after Crucible activates).

Expected: The source will wait to be moved back to the 'Running' state. Before that happens, the control plane will set the source's Crucible generation to 3 and direct it to reactivate.

Current state: There are two problems to solve here:

• Nexus has no way to update the Crucible generation number of an ensured instance.
• The migration state machine automatically resumes the source after migration fails; it should wait until it's told to do so (and should allow configuration changes to be interposed between migration failure and source restart).

[leftwo]

Nexus has no way to update the Crucible generation number of an ensured instance.

I'm working on the Omicron side to enable it to update a volume's generation number each time it is
pulled out of the database. I think this might solve this issue, but what exactly to you want to
happen to the "ensured instance"? Is this "ensured_instance" in propolis, or Omicron?

[gjcolombo]

It's in Propolis. Specifically this code needs to change:

propolis/bin/propolis-server/src/lib/server.rs

Lines 318 to 344 in 70e019a

	// Handle requests to an instance that has already been initialized. Treat
	// the instances as compatible (and return Ok) if they have the same
	// properties and return an appropriate error otherwise.
	//
	// TODO(#205): Consider whether to use this interface to change an
	// instance's devices and backends at runtime.
	if let VmControllerState::Created(existing) =
	&*server_context.services.vm.lock().await
	{
	let existing_properties = existing.properties();
	if existing_properties.id != request.properties.id {
	return Err(HttpError::for_internal_error(format!(
	"Server already initialized with ID {}",
	existing_properties.id
	)));
	}
	if *existing_properties != request.properties {
	return Err(HttpError::for_internal_error(
	"Cannot update running server".to_string(),
	));
	}
	return Ok(HttpResponseCreated(api::InstanceEnsureResponse {
	migrate: None,
	}));
	}

Now that I look at it again, this issue is sort of a duplicate of #205, but that issue doesn't track the migration state machine problem I mentioned, so I'll keep this alive at least for that part of the problem.

[gjcolombo]

For MVP, we're consciously going to allow instances to resume immediately after a failed migration out. This is legal because (if we've implemented everything correctly) nothing in the migration protocol destroys state on the source: the entire protocol has to finish before the target can do anything that would render the source unable to resume. This simplifies things considerably up in the control plane.

I think we could maintain this property even with a more complicated migration protocol (e.g. one where the target resumes while still pulling data in from the source), so long as we had a clear point of no return after which it was definitely not safe for the source to resume on its own.

Anyway, triaging this as Unscheduled since we don't really have any concrete plans that need this work right now.