thread pacing needed

[bengland2]

for large tests we need threads to start slow so that all threads can get into the mix and 1 thread doesn't pull way ahead of others. This is particularly important for tests with large numbers of threads (i.e. lots of client hosts). fio does this with "ramp_time" parameter and "startdelay" parameter. Would like to have something more automatic for smallfile, but we could start out this way.

[bengland2]

I thought of doing this algorithm instead:

let x[t] be the response time of the file t op done by this thread.

_x_ = weighted average of last K samples 
       = (sum (x[t-k] * 2^k) + 2^-K) / K

To see this, imagine that x[t-k] = x[t-(k-1)] = x is a constant. Then this sum reduces to

(K*x * (sum(2^k) + 2^-K)) / K

suppose K = 3 for example. Then:

(K*x*(sum(2^k) + 2^-K)) / K =
 (3 * x * ((2^-3 + 2^-2 + 2^-1) + 2^-3))) / 3 =
 (3 * x * (1) / 3 = 
 x

So it's just a weighted average giving more weight to recent samples.

Then we compute the think time between files using formula

think_time = (sqrt(T) - 1) * x

For large thread counts, this slows down threads that start early so that they don't get done before other threads get most of their work done. For a single thread test it has no effect at all, a nice feature.

So the other question is whether this is a stable system, I'm still chewing on that. I believe it is stable because of the weighted averaging, but I'm not sure. We don't want think time to oscillate.

[bengland2]

As Mark Nelson pointed out, if some thread has abnormally high response time, we don't want it throttling itself still more. But how do we get threads to cooperatively do this pacing without imposing a lot of inter-thread communication and synchronization overhead?

[bengland2]

This commit helps threads to start at closer to the same time, but doesn't control what happens once the threads are started.

[bengland2]

PR 27 hopefully will address this problem.

[bengland2]

I intended that the "autopause" feature would address this issue, but autopause significantly reduces throughput at this time, so it distorts the results significantly. So --autopause Y is not the default at this time, but it is there to experiment with.