Airflow Runtime error : Unique Constraint Violations observed

[pruyai]

Apache Airflow version

Other Airflow 3 version (please specify below)

If "Other Airflow 3 version" selected, which one?

3.1.2

What happened?

Hello people , I am facing an issue in my airflow code. I am working with celery executer and airflow version 3.1.2. It seems like when multiple tasks are rendering fields in parallel, the insert conflicts and raises the error below. I have once observed this happening when single task tries to execute as well but usually seen with parallel executions.

AirflowRuntimeError: API_SERVER_ERROR: {'status_code': 409, 'message': 'Server returned error', 'detail': {'detail': {'reason': 'Unique constraint violation', 'statement': 'INSERT INTO rendered_task_instance_fields (dag_id, task_id, run_id, map_index, rendered_fields, k8s_pod_yaml) VALUES (%(dag_id)s, %(task_id)s, %(run_id)s, %(map_index)s, %(rendered_fields)s, %(k8s_pod_yaml)s)', 'orig_error': 'duplicate key value violates unique constraint "rendered_task_instance_fields_pkey"

This issue seems to be intermittent and happens with multiple tasks, sometimes even 3 attempts for task resolution results into the same error causing entire dag failure.

Is this a known issue and would be resolved sooner in next couple of releases? Could you please guide if there is any way to fix this up. Any suggestions would be highly appreciated.

What you think should happen instead?

No response

How to reproduce

While multiple parallel insertions working together in airflow, It fails with the unique constraint violation error. Airflow is unable to assign unique ids to every task at runtime

Operating System

It is deployed to cloud and runs as per schedule

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

Do we have any information of getting this airflow bug resolved

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[Rishav23av]

Hi @pruyai, I'm planning to work on this and submit a PR let me know if you'd like to collaborate or if I should hold off.

[abhijeets25012-tech]

i want to contribute

[eladkal]

Can you share reproduce example?

[manipatnam]

@kaxil @eladkal
I can reproduce the issue by introducing a delay in the execution api call
The issue is due to execution_api_retries by the worker to the execution api when there is a delay in response from the API server.
The fix suggested in #63581 is applying a band-aid
Instead, we should add insert and update on conflict logic to rendererdtifileds.py by removing the session.merge()

Can I go ahead and raise a separate PR?