[dotnet-trace][collect-linux] Unable to specifically trace a cross-container process

[mdh1418]

Tracing a single process in a separate container with -p|--process-id fails with

[ERROR] Microsoft.Diagnostics.NETCore.Client.ServerNotAvailableException: Unable to connect to Process 652542. Please verify that /tmp/ is writable by the current user. If the target process has environment variable TMPDIR set, please set TMPDIR to the same directory. Please also ensure that the target process has {TMPDIR}/dotnet-diagnostic-{pid}-{disambiguation_key}-socket shorter than 108 characters. Please see https://aka.ms/dotnet-diagnostics-port for more information
   at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.GetDefaultAddress(Int32 pid)
   at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.GetDefaultAddress()
   at Microsoft.Diagnostics.NETCore.Client.PidIpcEndpoint.Connect(TimeSpan timeout)
   at Microsoft.Diagnostics.NETCore.Client.IpcClient.SendMessageGetContinuation(IpcEndpoint endpoint, IpcMessage message)
   at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.TryGetProcessInfo3()
   at Microsoft.Diagnostics.NETCore.Client.DiagnosticsClient.GetProcessInfo()
   at Microsoft.Diagnostics.Tools.Trace.CollectLinuxCommandHandler.ProcessSupportsUserEventsIpcCommand(Int32 pid, String processName, Int32& resolvedPid, String& resolvedName, String& detectedRuntimeVersion)
   at Microsoft.Diagnostics.Tools.Trace.CollectLinuxCommandHandler.CollectLinux(CollectLinuxArgs args)

When dotnet-trace collect-linux is tracing a single process, it attempts to connect to the process to probe for userevents support by sending a GetProcessInfo command. In this scenario, where the target process lives in a separate container, I believe the DiagnosticClient is unable to discover the cross-container .NET process from the PID alone, because the diagnostic-port of the Process in the separate container may live elsewhere.

We should be able to trace the cross-container process as record-trace seems to still be able to trace based on the PID because it currently enumerates PerfMaps and matches the corresponding PID from there. The limitation in this case is CollectLinuxCommand's approach to determining whether a specific process can support userevents or not. Cross-container single process tracing works in dotnet-trace collect using the diagnostic-port.

So maybe we should remove the tool's self-imposed limitation https://github.com/dotnet/diagnostics/blob/3272544fa65aa8c6af5285cecceb01ccb4acb029/src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs#L95C20-L99C22.
If we want collect-linux --probe to still have utility, as it currently does not require sudo to run (which would use the global namespace for ProcessIDs), so users need to be careful to use the same terminal/namespace to acquire the cross-container pid and pass it into dotnet-trace collect-linux --probe.
collect-linux requires the user to have r/w access to tracefs, which can be done through modifying user group permissions rather than running as sudo, requiring the same caution with pid acquisition/usage with respect to terminal/namespace.

Moreover, the tooling should be resilient to DiagnosticClient APIs throwing exceptions.

[noahfalk]

We should be able to trace the cross-container process as record-trace seems to still be able to trace based on the PID because it currently enumerates PerfMaps and matches the corresponding PID from there

I'm skeptical record-trace is going to do this regardless of what dotnet-trace does. For example imagine you have container A running Foo.exe, PID 1, and peer container B running Bar.exe, also PID 1 in B's namespace. If we run record-trace in container A and want to capture Bar.exe I doubt we can. Even if we use IPC tricks to exfiltrate Bar's PID within its own container, record-trace probably can't make use of it. I expect record-trace always resolves PIDs within the namespace where it is running which means it can't target processes in peer containers.

[zachcmadsen]

I can confirm that record-trace works for cross-container traces, but you need a specific setup. We run from a privileged daemonset that's configured to share the host namespace. Since we deal with PIDs in the host namespace, instead of the process's namespace, record-trace can resolve them.