TestISOImage times out (flake)

[medyagh]

Example:
TestISOImage times out

https://storage.googleapis.com/minikube-builds/logs/21368/42582/KVM_Linux.html#fail_TestISOImage%2fPersistentMounts%2f%2fdata

@nirs said

Looks like the global timeout - the iso tests ran last and failed in microseconds:

iso_test.go:97: (dbg) Non-zero exit: out/minikube-linux-amd64 -p guest-202831 ssh "df -t ext4 /data | grep /data": context deadline exceeded (2.786µs)
Replace the global timeout with per test timeout.

[nirs]

More info on the timeout - we use a global timeout for all tests (90 minutes). If some tests were very slow, we may time out when running other tests. The tests will fail with "context deadline exceeded" error which is not related to the test. This make tests that run at the end of the run look flakey and waste developer time debugging the "flaky" runs.

The way to avoid this issue is to remove the global test timeout, and replace it with per-test timeout. This requires more work as every test need different timeout, but ensures that slow tests do not affect other tests.

This also helps to identify very slow tests that need to improve. We have some tests that wait 10 minutes for for operation that take seconds. These tests can fail much faster minimizing the total test time.

[nirs]

Looking at more failures, we see that this is not the global timeout.

This KVM_Linux failed with the known issue when TestISO are run after the context deadline was exceeded:
KVM_Linux
KVM_Linux — Jenkins: completed with 10 / 447 failures in 65.73 minutes.

The interesting part is that this test did not time out after 90 minutes - so this is not an issue with the global test timeout. Maybe this an issue with the context we use in these tests:

=== RUN   TestISOImage/PersistentMounts//data
=== PAUSE TestISOImage/PersistentMounts//data
=== CONT  TestISOImage/PersistentMounts//data
iso_test.go:97: (dbg) Run:  out/minikube-linux-amd64 -p guest-625557 ssh "df -t ext4 /data | grep /data"
iso_test.go:97: (dbg) Non-zero exit: out/minikube-linux-amd64 -p guest-625557 ssh "df -t ext4 /data | grep /data": context deadline exceeded (3.039µs)

[nirs]

/assign