-
Notifications
You must be signed in to change notification settings - Fork 109
Description
Expected Behavior
Functions in Python 3.13 decorated with @app.retry(...) run and upon exceptions re-run correctly.
Actual Behavior
Functions in Python 3.13 decorated with @app.retry(...) cannot start.
Steps to Reproduce
- Create a new Function Project in Python 3.13 using VS Code in Win 11, the v2 programming model and choose to use a "EventHub trigger" template
- In tasks.json add "--verbose" to the task's command like this "command": "host start --verbose", so you can see additional logs
- Configure the @app.event_hub_message_trigger() properly, in my case I was using an Microsoft Entra identity
- run the function to see that it is working
- add @app.retry() decorator to the function and configure at least the strategy and max_retry_count args
- run the function again to see that it is not working anymore
- now add "PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION": "python" to local.setting.json
- run again function app and you'll see a proper traceback pointing to a problem with build_variable_interval_retry() or build_fixed_delay_retry() in loader.py from the azure functions runtime
- optionally you can try to run the Function App in a Debian devcontainer, there I also got the same problem
Relevant code being tried
import azure.functions as func
import logging
import os
EVENTHUB__fullyQualifiedNamespace = os.environ["EVENTHUB__fullyQualifiedNamespace"]
REQUEST_EVENT_HUB_NAME = os.environ["REQUEST_EVENT_HUB_NAME"]
REQUEST_CONSUMER_GROUP_NAME = os.environ[
"REQUEST_CONSUMER_GROUP_NAME"
]
app = func.FunctionApp()
@app.retry(
strategy="exponential_backoff",
max_retry_count="3",
# manually setting the retry intervals will also not work
# minimum_interval="00:00:02",
# maximum_interval="00:01:00",
)
@app.event_hub_message_trigger(
arg_name="azeventhub",
event_hub_name=REQUEST_EVENT_HUB_NAME,
connection="EVENTHUB",
consumer_group=REQUEST_CONSUMER_GROUP_NAME,
)
def eventhub_trigger(azeventhub: func.EventHubEvent):
body = azeventhub.get_body().decode("utf-8")
logging.info(f'Python EventHub trigger processed an event: {body}')Relevant log output
# The log output can be different. I'm posting here only the (imho) important lines of the logs from the different configurations I've tried.
## 1. Without setting the "PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION": "python" in local.settings.json it is this:
Found Python version 3.13.11 (py).
Selected out-of-process host.
Core Tools Version: 4.6.0+ab90faafcab539d63cd3d0ce5faf1bca4395fccc (64-bit)
Function Runtime Version: 4.1045.200.25556
Skipping 'AzureWebJobsStorage' from local settings as it's already defined in current environment variables.
[2026-02-10T13:13:13.910Z] Applying platform release channel configuration LATEST. Bundle version 4.30.0 will be used
[2026-02-10T13:13:13.911Z] Found a matching extension bundle at C:\Users\DEVALMIH\.azure-functions-core-tools\Functions\ExtensionBundles\Microsoft.Azure.Functions.ExtensionBundle\4.30.0
[2026-02-10T13:13:13.939Z] Loading functions metadata
[2026-02-10T13:13:13.942Z] Worker indexing is enabled
[2026-02-10T13:13:13.944Z] Fetching metadata for workerRuntime: python
[2026-02-10T13:13:13.945Z] Reading functions metadata (Worker)
[2026-02-10T13:13:14.886Z] INFO: Starting proxy worker. Worker ID: e7fbc598-8e33-407c-8a55-8a65201468ff, Request ID: bd4fcd34-a0a5-46a0-84f1-b20d4362998b, Host Address: 127.0.0.1:55356
[2026-02-10T13:13:15.064Z] INFO: Successfully opened gRPC channel to 127.0.0.1:55356
[2026-02-10T13:13:15.066Z] INFO: Detaching console logging.
[2026-02-10T13:13:17.122Z] Switched to gRPC logging.
[2026-02-10T13:13:17.136Z] Received WorkerInitRequest, python version 3.13.11 (tags/v3.13.11:6278944, Dec 5 2025, 16:26:58) [MSC v.1944 64 bit (AMD64)], worker version 4.40.1, request ID bd4fcd34-a0a5-46a0-84f1-b20d4362998b.
[2026-02-10T13:13:17.145Z] Finished prioritize_customer_dependencies: worker_dependencies_path: C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64, customer_dependencies_path: C:\Users\***\PycharmProjects\delete_me2\.venv\Lib\site-packages, working_directory: C:\Users\***\PycharmProjects\delete_me2, Placeholder: False, sys.path: ['C:\\Users\\***\\PycharmProjects\\delete_me2\\.venv\\Lib\\site-packages', 'C:\\Program Files\\Microsoft\\Azure Functions Core Tools\\workers\\python\\3.13/WINDOWS/X64', 'C:\\Users\\***\\AppData\\Local\\Programs\\Python\\Python313\\python313.zip', 'C:\\Users\\***\\AppData\\Local\\Programs\\Python\\Python313\\DLLs', 'C:\\Users\\***\\AppData\\Local\\Programs\\Python\\Python313\\Lib', 'C:\\Users\\***\\AppData\\Local\\Programs\\Python\\Python313', 'C:\\Users\\***\\PycharmProjects\\delete_me2\\.venv', 'C:\\Program Files\\Microsoft\\Azure Functions Core Tools\\workers\\python\\3.13\\WINDOWS\\X64', 'C:\\Users\\***\\PycharmProjects\\delete_me2']
[2026-02-10T13:13:17.182Z] Using library: <module 'azure_functions_runtime' from 'C:\\Program Files\\Microsoft\\Azure Functions Core Tools\\workers\\python\\3.13/WINDOWS/X64\\azure_functions_runtime\\__init__.py'>, library version: 1.1.0b1
[2026-02-10T13:13:17.829Z] Language Worker Process exited. Pid=13012.
[2026-02-10T13:13:17.830Z] py exited with code -1073741819 (0xC0000005). .
## 2. If I set "PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION": "python" the log output is the same until the last two lines. After that I get:
[2026-02-10T13:16:18.084Z] Initializing worker process failed
[2026-02-10T13:16:18.085Z] Result: Failure
Type:
Exception: TypeError: strptime() argument 0 must be str, not <class 'NoneType'>
Stack: File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 99, in worker_init_request
load_function_metadata(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 410, in load_function_metadata
_metadata_result = (index_functions(function_path, function_app_directory)) \
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 420, in index_functions
process_indexed_function(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 140, in process_indexed_function
retry_protos = build_retry_protos(protos, indexed_function)
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 61, in build_retry_protos
return build_variable_interval_retry(protos, retry, max_retry_count,
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 104, in build_variable_interval_retry
seconds=convert_to_seconds(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 32, in convert_to_seconds
x = time.strptime(timestr, '%H:%M:%S')
File "C:\Users\***\AppData\Local\Programs\Python\Python313\Lib\_strptime.py", line 783, in _strptime_time
tt = _strptime(data_string, format)[0]
File "C:\Users\***\AppData\Local\Programs\Python\Python313\Lib\_strptime.py", line 524, in _strptime
raise TypeError(msg.format(index, type(arg)))
.
[2026-02-10T13:16:18.089Z] Failed to start language worker process for runtime: python. workerId:f7d730d4-880b-4c07-ac48-6780737dadd9
[2026-02-10T13:16:18.090Z] Removing errored webhost language worker channel for runtime: python workerId:f7d730d4-880b-4c07-ac48-6780737dadd9
[2026-02-10T13:16:18.091Z] System.Private.CoreLib: Result: Failure
Type:
Exception: TypeError: strptime() argument 0 must be str, not <class 'NoneType'>
Stack: File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 99, in worker_init_request
load_function_metadata(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 410, in load_function_metadata
_metadata_result = (index_functions(function_path, function_app_directory)) \
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 420, in index_functions
process_indexed_function(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 140, in process_indexed_function
retry_protos = build_retry_protos(protos, indexed_function)
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 61, in build_retry_protos
return build_variable_interval_retry(protos, retry, max_retry_count,
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 104, in build_variable_interval_retry
seconds=convert_to_seconds(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 32, in convert_to_seconds
x = time.strptime(timestr, '%H:%M:%S')
File "C:\Users\***\AppData\Local\Programs\Python\Python313\Lib\_strptime.py", line 783, in _strptime_time
tt = _strptime(data_string, format)[0]
File "C:\Users\***\AppData\Local\Programs\Python\Python313\Lib\_strptime.py", line 524, in _strptime
raise TypeError(msg.format(index, type(arg)))
.
[2026-02-10T13:16:18.100Z] 0 functions loaded
# [here there were a lot of lines showing different configurations]
[2026-02-10T13:16:18.501Z] Starting JobHost
[2026-02-10T13:16:18.504Z] Starting Host (HostId=delt01353-157237864, InstanceId=2304af69-73f2-4daa-b481-8b1f3d1948d6, Version=4.1045.200.25556, ProcessId=47672, AppDomainId=1, InDebugMode=False, InDiagnosticMode=False, FunctionsExtensionVersion=(null))
[2026-02-10T13:16:18.508Z] Loading functions metadata
[2026-02-10T13:16:18.509Z] Worker indexing is enabled
[2026-02-10T13:16:18.510Z] Fetching metadata for workerRuntime: python
[2026-02-10T13:16:18.511Z] Reading functions metadata (Worker)
[2026-02-10T13:16:19.315Z] Traceback (most recent call last):
[2026-02-10T13:16:19.317Z] File "<frozen runpy>", line 198, in _run_module_as_main
[2026-02-10T13:16:19.318Z] File "<frozen runpy>", line 88, in _run_code
[2026-02-10T13:16:19.319Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy\__main__.py", line 71, in <module>
[2026-02-10T13:16:19.319Z] cli.main()
[2026-02-10T13:16:19.320Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\cli.py", line 508, in main
[2026-02-10T13:16:19.321Z] run()
[2026-02-10T13:16:19.322Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\cli.py", line 342, in run_file
[2026-02-10T13:16:19.324Z] start_debugging(target)
[2026-02-10T13:16:19.325Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\cli.py", line 328, in start_debugging
[2026-02-10T13:16:19.325Z] debugpy.listen(options.address)
[2026-02-10T13:16:19.327Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy/..\debugpy\public_api.py", line 47, in wrapper
[2026-02-10T13:16:19.328Z] return wrapped(*args, **kwargs)
[2026-02-10T13:16:19.329Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\api.py", line 133, in debug
[2026-02-10T13:16:19.331Z] log.reraise_exception("{0}() failed:", func.__name__, level="info")
[2026-02-10T13:16:19.332Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\api.py", line 131, in debug
[2026-02-10T13:16:19.333Z] return func(address, settrace_kwargs, **kwargs)
[2026-02-10T13:16:19.334Z] File "c:\Users\***\.vscode\extensions\ms-python.debugpy-2025.18.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\api.py", line 260, in listen
[2026-02-10T13:16:19.336Z] raise RuntimeError(str(endpoints["error"]))
[2026-02-10T13:16:19.337Z] RuntimeError: Can't listen for client connections: [WinError 10048]
[2026-02-10T13:16:19.359Z] Language Worker Process exited. Pid=52912.
[2026-02-10T13:16:19.360Z] py exited with code 1 (0x1). log.reraise_exception("{0}() failed:", func.__name__, level="info"), raise RuntimeError(str(endpoints["error"])),RuntimeError: Can't listen for client connections: [WinError 10048]
[2026-02-10T13:16:19.369Z] Exceeded language worker restart retry count for runtime:python. Shutting down and proactively recycling the Functions Host to recover
[2026-02-10T13:16:23.490Z] Host lock lease acquired by instance ID '0000000000000000000000009AE3DE10'.
[2026-02-10T13:17:18.521Z] Starting worker process failed
[2026-02-10T13:17:18.523Z] Microsoft.Azure.WebJobs.Script.Grpc: The operation has timed out.
[2026-02-10T13:17:18.525Z] Failed to start language worker process for runtime: python. workerId:5c73827d-09ea-4e86-b595-c15261bbb78a
[2026-02-10T13:17:18.531Z] Reading functions metadata (Custom)
[2026-02-10T13:17:18.548Z] 0 functions found (Custom)
## 3. With minimum_interval and maximum_interval set on the @app.retry() the stack trace is a bit different:
[2026-02-10T14:54:51.715Z] System.Private.CoreLib: Result: Failure
Type:
Exception: AttributeError: Fail to convert to Duration. Expected a timedelta like object got Duration: 'Duration' object has no attribute 'days'
Stack: File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 99, in worker_init_request
load_function_metadata(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 410, in load_function_metadata
_metadata_result = (index_functions(function_path, function_app_directory)) \
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\handle_event.py", line 420, in index_functions
process_indexed_function(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 140, in process_indexed_function
retry_protos = build_retry_protos(protos, indexed_function)
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 61, in build_retry_protos
return build_variable_interval_retry(protos, retry, max_retry_count,
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\azure_functions_runtime\loader.py", line 111, in build_variable_interval_retry
return protos.RpcRetryOptions(
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\google\protobuf\internal\python_message.py", line 560, in init
field_copy._internal_assign(field_value)
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\google\protobuf\internal\well_known_types.py", line 449, in _internal_assign
self.FromTimedelta(td)
File "C:\Program Files\Microsoft\Azure Functions Core Tools\workers\python\3.13/WINDOWS/X64\google\protobuf\internal\well_known_types.py", line 443, in FromTimedelta
raise AttributeError(
.requirements.txt file
azure-functionsWhere are you facing this problem?
Local - Core Tools in Windows 11 and in a Debian devcontainer
Function app name
No response
Additional Information
I've searched but didn't found any information if the retry policies are depricated in Python 3.13. So the aren't right? Because there is an issue that it states they were removed temporary from azure-functions 1.17.0.
I've also tried to use Copilot to trobleshoot my problem and I was told that the problem is incompatibility with the protobuf package. Namely, only in a devcontainer I get the following additional log:
Attempt to remove module cache for google._upb but failed with 'google'. Using the original module cache.
Because of this error I was suggested to try to use the Python native implementation of protobuf package by setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION to python.
I also ran the app with azure_functions_runtime downgraded to 1.0.0 (last stable version) by adding it to the requirements.txt, but the problem was still there.