Extend memory tracking resources

[huuanhhuyn]

Extend memory tracking resources with another queue-based approach (as opposed to the current sampling-rate notification approach).

Usage:
If the sampling interval is given, the notification approach is selected:
raft::memory_tracking_resources tracked(res, oss, 1ms);
If it is NOT given, the queue-based approach is selected:
raft::memory_tracking_resources tracked(res, oss);

Compare two approaches:

• queue-based recording approach preserves all (de)allocation events in the queue and associate the full nvtx range to each event. Additionally, it labels a deallocation to the nvtx range where the corresponding allocation occurs. This should be used when labels are essential and certain overhead for debugging is accepted.
• notification approach is less invasive with low overhead. This is used when label accuracy is not important and several dropped events are acceptable.

Unit test benchmark (H100, 64 threads, each thread 200x allocations, each allocation 256KiB):

• recording approach: 34ms, all 25600 events recorded
• sampling approach: 7ms, around 100 events recorded

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[achirkin]

From the description it follows that the new queue approach attaches NVTX annotations to the allocation events one-to-one. If so, it seems logical that NVTX annotations should be pulled from the allocating thread during the allocation rather than from the main thread. Then you'd also not need mutex locking of nvtx records since they are not accessed across threads.

Please refactor this as a separate resource type rather than changing the behavior of the existing resource, because the difference is significant.