stacks API results in permission denied, despite following instructions for proper setup

[orange-puff]

I followed the instructions here https://github.com/dotnet/dotnet-monitor/blob/main/documentation/api/stacks.md and followed relevant links, but I cannot seem to use the stacks API. The trace and dump API work without any issue.

Setup:

1. running on Kubernetes, dotnet monitor is in a separate container from main application container, both in same pod
2. dotnet-monitor image version:
   dotnet/monitor:9
3. dotnet-monitor environment variables:

env:
- name: DOTNETMONITOR_DiagnosticPort__ConnectionMode
  value: Listen
- name: DOTNETMONITOR_DiagnosticPort__EndpointName
  value: /diag/dotnet-monitor.sock
- name: DOTNETMONITOR_Storage__DefaultSharedPath
  value: /diag
- name: DOTNETMONITOR_Urls
  value: http://localhost:52323
- name: DotnetMonitor_Egress__FileSystem__files__DirectoryPath
  value: /diag
- name: DotnetMonitor_InProcessFeatures__CallStacks__Enabled
  value: "true"
- name: DotnetMonitor_InProcessFeatures__Enabled
  value: "true"

4. application environment variables:

- name: DOTNET_DiagnosticPorts
  value: /diag/dotnet-monitor.sock,nosuspend

Behavior:
On the application container:
curl http://localhost:52323/stacks?egressProvider=files -v
returns a 202 response, but when I check on the operation, I see

{
    "resourceLocation": null,
    "error": {
        "code": "400",
        "message": "Permission denied"
    },
    "operationId": "ae184ff3-2805-47f1-a49d-7e628dccc283",
    "createdDateTime": "2025-12-04T22:42:24.6929287Z",
    "status": "Failed",
    "process": {
        "pid": 1,
        "uid": "305640f3-2e37-416f-b50b-fb429845f377",
        "name": "My.Application"
    },
    "egressProviderName": "files",
    "isStoppable": false,
    "tags": []
}

curl http://localhost:52323/trace?egressProvider=files -v works without issue.

curl http://localhost:52323/stacks -v
results in a 500 response and the following log from the monitor container

{
    "Timestamp": "2025-12-04T22:36:31.9286750Z",
    "EventId": 13,
    "LogLevel": "Error",
    "Category": "Microsoft.AspNetCore.Server.Kestrel",
    "Message": "Connection id \u00220HNHJG7N29LRL\u0022, Request id \u00220HNHJG7N29LRL:00000001\u0022: An unhandled exception was thrown by the application.",
    "Exception": "System.Net.Sockets.SocketException (13): Permission denied\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)\n   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket, Boolean saeaCancelable)\n   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)\n   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP)\n   at Microsoft.Diagnostics.Monitoring.WebApi.ProfilerChannel.SendMessage(IEndpointInfo endpointInfo, IProfilerMessage message, CancellationToken token)\n   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine\u0026 stateMachine)\n   at Microsoft.Diagnostics.Monitoring.WebApi.ProfilerChannel.SendMessage(IEndpointInfo endpointInfo, IProfilerMessage message, CancellationToken token)\n   at Microsoft.Diagnostics.Tools.Monitor.Stacks.StacksOperation.StacksOperationPipeline.StartAsync(CancellationToken token)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecutionContextCallback(Object s)\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext(Thread threadPoolThread)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext()\n   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)\n   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.SetResult(TResult result)\n   at Microsoft.Diagnostics.Monitoring.EventPipe.EventSourcePipeline\u00601.StartAsync(CancellationToken token)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecutionContextCallback(Object s)\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext(Thread threadPoolThread)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext()\n   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)\n   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)\n   at System.Threading.Tasks.Task\u00601.TrySetResult(TResult result)\n   at System.Threading.Tasks.UnwrapPromise\u00601.TrySetFromTask(Task task, Boolean lookForOce)\n   at System.Threading.Tasks.UnwrapPromise\u00601.ProcessInnerTask(Task task)\n   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)\n   at System.Threading.Tasks.Task\u00601.TrySetResult(TResult result)\n   at System.Threading.Tasks.Task.TwoTaskWhenAnyPromise\u00601.Invoke(Task completingTask)\n   at System.Threading.ThreadPoolWorkQueue.Dispatch()\n   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()\n--- End of stack trace from previous location ---\n   at Microsoft.Diagnostics.Monitoring.WebApi.ProfilerChannel.SendMessage(IEndpointInfo endpointInfo, IProfilerMessage message, CancellationToken token)\n   at Microsoft.Diagnostics.Tools.Monitor.Stacks.StacksOperation.StacksOperationPipeline.StartAsync(CancellationToken token)\n   at Microsoft.Diagnostics.Tools.Monitor.PipelineArtifactOperation\u00601.ExecuteAsync(Stream outputStream, CancellationToken token)\n   at Microsoft.Diagnostics.Tools.Monitor.PipelineArtifactOperation\u00601.ExecuteAsync(Stream outputStream, CancellationToken token)\n   at Microsoft.Diagnostics.Monitoring.WebApi.OutputStreamResult.\u003C\u003Ec__DisplayClass6_0.\u003C\u003CExecuteResultAsync\u003Eb__0\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n   at Microsoft.Diagnostics.Monitoring.WebApi.ActionContextExtensions.\u003C\u003Ec__DisplayClass1_0.\u003C\u003CInvokeAsync\u003Eb__0\u003Ed.MoveNext()\n--- End of stack trace from previous location ---\n   at Microsoft.Diagnostics.Monitoring.WebApi.EgressOperationService.ExecuteEgressOperationAsync(EgressRequest egressRequest, CancellationToken stoppingToken)\n   at Microsoft.Diagnostics.Monitoring.WebApi.ExecutionHelper.InvokeAsync[T](Func\u00602 action, ILogger logger, CancellationToken token)\n   at Microsoft.Diagnostics.Monitoring.WebApi.ActionContextExtensions.InvokeAsync(ActionContext context, Func\u00602 action, ILogger logger)\n   at Microsoft.Diagnostics.Monitoring.WebApi.OutputStreamResult.ExecuteResultAsync(ActionContext context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextResultFilterAsync\u003Eg__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeResultFilters\u003Eg__Awaited|28_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeFilterPipelineAsync\u003Eg__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)\n   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)",
    "State": {
        "Message": "Connection id \u00220HNHJG7N29LRL\u0022, Request id \u00220HNHJG7N29LRL:00000001\u0022: An unhandled exception was thrown by the application.",
        "ConnectionId": "0HNHJG7N29LRL",
        "TraceIdentifier": "0HNHJG7N29LRL:00000001",
        "{OriginalFormat}": "Connection id \u0022{ConnectionId}\u0022, Request id \u0022{TraceIdentifier}\u0022: An unhandled exception was thrown by the application."
    },
    "Scopes": [
        {
            "Message": "SpanId:b3368dfad94f8433, TraceId:c151d4510851ae7b1365e54f0df5f698, ParentId:0000000000000000",
            "SpanId": "b3368dfad94f8433",
            "TraceId": "c151d4510851ae7b1365e54f0df5f698",
            "ParentId": "0000000000000000"
        },
        {
            "Message": "ConnectionId:0HNHJG7N29LRL",
            "ConnectionId": "0HNHJG7N29LRL"
        },
        {
            "Message": "RequestPath:/stacks RequestId:0HNHJG7N29LRL:00000001",
            "RequestId": "0HNHJG7N29LRL:00000001",
            "RequestPath": "/stacks"
        }
    ]
}

I am not sure why this would be, the trace and dump API work perfectly well

[orange-puff]

This comment helped me
#7016 (comment)
I added

securityContext:
   runAsUser: 0
   runAsGroup: 0

to my dotnet-monitor container spec and can now use the stacks API. Perhaps the documentation can be improved to specify this? Might save the next person some time.

[wiktork]

We document some of this at https://github.com/dotnet/dotnet-monitor/blob/a2bc2c118f282b4cab0161ecf53dfba82fb20e3f/documentation/compatibility/8.0/README.md#docker-non-root-by-default but I agree it needs to be more visible.

[orange-puff]

Thank you for the quick reply. I was wondering something else

https://github.com/dotnet/dotnet-monitor/blob/main/documentation/configuration/in-process-features-configuration.md
Can I learn more about the in process feature and what will be loaded into the target application? I am confused why some additional library is needed if I can run dotnet-stack against my target application right now.