Fix flaky test: k8s script v1 integration test

[todthomson]

Background

Because of an historically intermittent (flaky) test run failure, which triggers alerts to @OctopusDeploy/team-executions-foundations in test:

Octopus.Tentacle.Kubernetes.Tests.Integration
.KubernetesAgent.KubernetesScriptServiceV1IntegrationTest.RunSimpleScript()

caused by:

Server exception: 
System.IO.EndOfStreamException: Attempted to read past the end of the stream.
at Halibut.Transport.Protocol.MessageSerializer
.ReadMessageAsync[T](RewindableBufferStream stream, CancellationToken cancellationToken)

which results in:

An error occurred communicating with Tentacle.
This action will be retried after 1 seconds.
Retry attempt 1.
Retries will be performed for up to 89 seconds.

which subsequently results in an assertion failure:

Started to be 1L, but found 2L.

caused by the following assertion:

recordedMethodUsages.For(nameof(IAsyncClientKubernetesScriptServiceV1.StartScriptAsync))
.Started.Should().Be(1);
=> 2

but I think that's perfectly reasonable (i.e. it's reasonable to restart when loosing comms with Tentacle).

Results

Thus, I've updated the assertion to be "at least once":

recordedMethodUsages.For(nameof(IAsyncClientKubernetesScriptServiceV1.StartScriptAsync))
.Started.Should().BeGreaterThanOrEqualTo(1);

Before

Flaky test e.g.
https://build.octopushq.com/buildConfiguration/TeamFireAndMotion_OctopusTentacle_TentacleVLatest_net80_ChainFullChainNightly/21130454

After

No more flaky test.

How to review this PR

This is how Octopus behaves when the connection to Tentacle is interrupted.

Do you agree that, therefore, the assertion in the original test "as written" was wrong?

If so, please approve.

If not, please talk to me and explain why (the assertion should remain as it is)?

Pre-requisites

• I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.

Yes, this is a flaky test, I don't think it needs a GitHub issue.

• I have considered informing or consulting the right people, according to the ownership map.

Consultation plan:

1. Talk internally within my team (to see if I'm on the right track).
2. Talk to the original test author @APErebus (to confirm that I'm on the right track).

• I have considered appropriate testing for my change.

My change IS a test, and, it will run in the CI environment.

[julietrb1]

I like it very much, but I've suggested shortening the comment and adding it to this PR's details instead.

[Jtango18]

Approved by MD - the logic behind it makes sense

@APErebus - if you find this abhorent, we can always reverse down the track.