e2e: tolerate transient errors in Eventually

[gman0]

Summary

There are a few e2e tests that fail on errors that may be only transient. This PR adds a helper function TolerateOrFail depending on the supplied predicates and tolerates/fails accordingly. It's being called where appropriate: as far as I could tell, these are the last places that were left after the other Eventually() VS require.* cleanups we had.

What Type of PR Is This?

/kind cleanup

Related Issue(s)

Fixes #3905

Release Notes

NONE

[kcp-ci-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign embik for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gman0]

Also, I think our error handling in tests could use some work. E.g. it seems, as time went on, the different kcptestinghelpers.Eventually, require.Eventually, and require.EventuallyWithT retry-able functions have been used, and it's a bit of a mess. IMHO we should pick one and stick with it.

I'd be really happy if someone more knowledgeable around testing frameworks could give their few cents and say if there are any ideas worth borrowing.