Unhandled exception. Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request.

[mechnixlab]

it is working fine without vnet integration and private endpoints for storage and fhir server. for azure function app FhirtoDatalake

Getting following error when putting vnet integration on:

023-09-19T14:55:58Z [Error] Unhandled exception. Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
2023-09-19T14:55:58Z [Error] ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
2023-09-19T14:55:58Z [Error] at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
2023-09-19T14:55:58Z [Information] at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
2023-09-19T14:55:58Z [Information] at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
2023-09-19T14:55:58Z [Information] at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
2023-09-19T14:55:58Z [Information] at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2023-09-19T14:55:58Z [Information] --- End of inner exception stack trace ---
2023-09-19T14:55:58Z [Information] at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2023-09-19T14:55:58Z [Information] at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2023-09-19T14:55:58Z [Information] at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2023-09-19T14:55:58Z [Information] at Grpc.Net.Client.Internal.GrpcCall2.RunCall(HttpRequestMessage request, Nullable1 timeout)")
2023-09-19T14:55:58Z [Information] at Grpc.Net.Client.Internal.HttpContentClientStreamWriter2.WriteAsyncCore(TRequest message) 2023-09-19T14:55:58Z [Information] at Microsoft.Azure.Functions.Worker.GrpcWorker.SendStartStreamMessageAsync(IClientStreamWriter1 requestStream) in D:\a\1\s\src\DotNetWorker.Grpc\GrpcWorker.cs:line 86
2023-09-19T14:55:58Z [Information] at Microsoft.Azure.Functions.Worker.WorkerHostedService.StartAsync(CancellationToken cancellationToken) in D:\a\1\s\src\DotNetWorker.Core\WorkerHostedService.cs:line 25
2023-09-19T14:55:58Z [Information] at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
2023-09-19T14:55:58Z [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2023-09-19T14:55:58Z [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
2023-09-19T14:55:58Z [Information] at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
2023-09-19T14:55:58Z [Information] at Microsoft.Health.AnalyticsConnector.FunctionApp.Program.Main() in /home/vsts/work/1/s/FhirToDataLake/src/Microsoft.Health.AnalyticsConnector.FunctionApp/Program.cs:line 26
2023-09-19T14:55:59Z [Verbose] Handling WorkerErrorEvent for runtime:dotnet-isolated, workerId:dotnet-isolated. Failed with: Microsoft.Azure.WebJobs.Script.Workers.WorkerProcessExitException: dotnet exited with code 134 (0x86)
---> System.Exception: Unhandled exception. Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request., ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1)., at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode)
--- End of inner exception stack trace ---
2023-09-19T14:55:59Z [Verbose] Attempting to dispose webhost or jobhost channel for workerId: '01f90328-9cf1-4124-a9d6-0612b755d12a', runtime: 'dotnet-isolated'
2023-09-19T14:55:59Z [Verbose] Disposing language worker channel with id:01f90328-9cf1-4124-a9d6-0612b755d12a
2023-09-19T14:55:59Z [Verbose] Disposed language worker channel with id:01f90328-9cf1-4124-a9d6-0612b755d12a
2023-09-19T14:55:59Z [Verbose] No initialized worker channels for runtime 'dotnet-isolated'. Delaying future invocations
2023-09-19T14:55:59Z [Verbose] Restarting worker channel for runtime: 'dotnet-isolated'
2023-09-19T14:55:59Z [Error] Exceeded language worker restart retry count for runtime:dotnet-isolated. Shutting down and proactively recycling the Functions Host to recover
2023-09-19T14:55:59Z [Verbose] Hosting stopping
2023-09-19T14:55:59Z [Verbose] Stopping file watchers.
2023-09-19T14:55:59Z [Verbose] Waiting for RpcFunctionInvocationDispatcher to shutdown
2023-09-19T14:55:59Z [Verbose] Draining invocations from language worker channel completed. Shutting down 'RpcFunctionInvocationDispatcher'
2023-09-19T14:55:59Z [Information] Stopping JobHost
2023-09-19T14:55:59Z [Verbose] Stopping ScriptHost instance '345f9d0c-c78e-4b97-8e2c-9a9e395c0366'.
2023-09-19T14:55:59Z [Information] Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener' for function 'JobManagerFunction'

we have convert the JSON template to bicep using https://bicep.kwitantie.app/

Can anyone suggest what will be workaround.
azure-function-fhirtodatalake.txt

[ardoric]

Just to save anyone looking into this some investigation:

This error seems to be due to the old version of some Function App libraries that are being used in the bundle that used to have this issue when connecting to a VNET: Azure/azure-functions-dotnet-worker#524

I have reproduced the issue by deploying the current release bundle to a linux function app using the provided template and then adding a vnet integration.

I was also able to remove this error by upgrading the following packages:

Microsoft.Azure.Functions.Worker.Extensions.Timer 4.0.1 -> 4.2.0
Microsoft.Azure.Functions.Worker.Sdk 1.0.3 -> 1.14.1
Microsoft.Azure.Functions.Worker 1.1.0 -> 1.19.0

We are currently working through other bundle issues since my build was missing the linux runtime libraries.