Add loadtesting framework

[SimonTheLeg]

Summary

This PR adds the loadtesting framework, considerations for review:

• testing/example_test.go is a good starting point to see how all different aspects of the framework come together
• Updates the Readme to describe the loadtest itself. Old Readme has been moved to Proposal.md
• For the stats we just wrap around https://github.com/montanaflynn/stats, so we can give stats names for the output. But other than that I found it much better to use a well-tested, zero dependency library instead of us going around implementing our own P99 and so on functions
• The framework is kept as minimal as possible, and takes some inspiration from clusterloader2 kubernetes performance testing https://github.com/kubernetes/perf-tests/tree/master/clusterloader2, especially the overall folder structure and the tuningset terminology. It does modernize some aspects of it by using Go iterators for the tuninsets themselves to control concurrency via yielding, which I think results in a bit cleaner code and error handling
• I went against passing configurable options to the tests via flags and instead went with env variables. The idea behind this was mainly that a) we don't need to have custom TestMains everywhere to ensure centralized flags in setup.go + b) we can run things like go test ./... without all test packages requiring the same flags

What Type of PR Is This?

/kind feature

Release Notes

NONE

[ntnn]

First quick look

[ntnn]

Wouldn't it be short to just require.NotEmpty cfg.FrontProxyKubeconfig?

	if slices.Contains(caps, KCPFrontProxyKubeconfig) {
		cfg.FrontProxyKubeconfig = os.Getenv(string(KCPFrontProxyKubeconfig))
		require.NotEmpty(t, cfg.FrontProxyKubeconfig, "env var must be set", KCPFrontProxyKubeconfig))
	}

This should happen early enough that we don't need to check all settings first before failing.

[SimonTheLeg]

Hmm I really like the behavior that it immediately tells you all the variables that are missing. I expect over time for the list to grow quite large and when you run these tests as a pod somewhere, I think its convenient if the first error message just tells you everything.

What I could offer ot make it a bit more streamlined is something like this. Let me know if you have a strong preference on it over the other one.

	cfg := &Config{}
	var missing []Capability

	setters := map[Capability]func(*rest.Config){
		KCPFrontProxyKubeconfig: func(c *rest.Config) { cfg.FrontProxyKubeconfig = c },
		KCPShardKubeconfig:      func(c *rest.Config) { cfg.ShardKubeconfig = c },
	}

	for _, cap := range caps {
		v := os.Getenv(string(cap))
		if v == "" {
			missing = append(missing, cap)
			continue
		}
		setters[cap](loadKubeconfig(t, v))
	}

	require.Empty(t, missing, "the following required environment variables are not set: %v", missing)

[ntnn]

You could also use a combination of assert and a boolean:

	if slices.Contains(caps, KCPFrontProxyKubeconfig) {
		cfg.FrontProxyKubeconfig = os.Getenv(string(KCPFrontProxyKubeconfig))
		assert.NotEmpty(t, cfg.FrontProxyKubeconfig, "env var must be set", KCPFrontProxyKubeconfig))
		anyMissing = true
	}
	// ...
	require.False(t, anyMissing, "missing required components and/or settings")

Keeps it streamlined while staying flexible to check if e.g. additional tooling is present if the configuration requires it.

[SimonTheLeg]

sorry might be Monday brain: But this would not work for the happy path or?
If I have the env set correctly then assert is fine, but we still would set anyMissing to true, even though it is not missing?

[ntnn]

Ah true :D I wasn't thinking correctly - but you can check on testing.T if it is already failed:
https://pkg.go.dev/testing#T.Failed

	if slices.Contains(caps, KCPFrontProxyKubeconfig) {
		cfg.FrontProxyKubeconfig = os.Getenv(string(KCPFrontProxyKubeconfig))
		assert.NotEmpty(t, cfg.FrontProxyKubeconfig, "env var must be set", KCPFrontProxyKubeconfig))
	}
	// ...
	require.False(t, t.Failed(), "missing required components and/or settings")

[kcp-ci-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from ntnn. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[SimonTheLeg]

/hold there is currently an issue where the unit-tests are passing, but skipping the unit-tests of the framework. I'll look into it, but for now hold the PR so we don't accidentally merge/forget about it.

[SimonTheLeg]

/unhold
I think the E2E are currently borked by our recent rebase k8s 1.35 update, but previously skipped unit tests for the load testing framework are now correctly executed :)
So ready for review

[SimonTheLeg]

/retest

[SimonTheLeg]

/restest