Data Aggregation is currently a single-threaded process and while smaller accounts finish within minutes. Basic testing on larger organizations with a large number of workspaces/runs, results in an extremely slow process.
Ideally, we should be able to parallelize this to some degree with an appropriate rate limiter and channels to reduce the output.
Data Aggregation is currently a single-threaded process and while smaller accounts finish within minutes. Basic testing on larger organizations with a large number of workspaces/runs, results in an extremely slow process.
Ideally, we should be able to parallelize this to some degree with an appropriate rate limiter and channels to reduce the output.