perf: eagerly flush shuffle partitions once batch_size rows accumulated [experimental]

perf: eagerly flush shuffle partitions once batch_size rows accumulated [experimental] #6669