feat: Adding DAG parsing profiler tool #13645
Conversation
product-auto-label
bot
added
samples
tiranux
force-pushed
the
add-dag-parsing-profiler
branch
from
a74ffb7 to
8552eb2
Compare
|
Here is the summary of changes. You are about to add 2 region tags.
This comment is generated by snippet-bot.
|
tiranux
force-pushed
the
add-dag-parsing-profiler
branch
from
8552eb2 to
507bff9
Compare
| When triggered, it offloads the resource-intensive DAG parsing process to a temporary, isolated Kubernetes Pod. Its primary goal is to detect **parsing latency issues** and identify heavy top-level code execution without impacting your environment's workload resources. As a byproduct of this analysis, it also validates DAG integrity and catches syntax errors. | ||
|
|
||
| ## 🌟 Key Features | ||
| * **Isolated Execution:** Offloads parsing logic to a separate Pod, protecting the Scheduler from resource contention and crashes. |
There was a problem hiding this comment.
note, in Composer 3 dag processor is run separately from Scheduler, in a dedicated workload - let's add info about Composer 2/Composer 3 and when this tool is applicable
There was a problem hiding this comment.
Done, ommited the specific mention of the scheduler as refer to environment resources as it moving to the Kubernetes pod, isolates the workload of this analysis wether it is a Composer 2 or Composer 3.
There was a problem hiding this comment.
Added further clarification regarding the version for image setup (when no automatic detection).
2 participants
Description
Adding a new DAG tool to troubleshoot and profile DAGs parsing.
Note: Before submitting a pull request, please open an issue for discussion if you are not associated with Google.
Checklist
nox -s py-3.9(see Test Environment Setup)nox -s lint(see Test Environment Setup)