perf: filter source networks using overrides in network mapping#40
perf: filter source networks using overrides in network mapping#40albertoflorez wants to merge 2 commits into
Conversation
|
Great optimization — the etcd 1MB limit issue in large vSphere environments is a real problem and this is the right approach to fix it. One issue I spotted: exclude-only overrides will break with this change. If a user provides overrides with only The fix is to key the loop condition on whether include IDs were actually extracted, not on whether overrides exist: loop: >-
{{
(mtv_networks | selectattr('id', 'in', mtv_allowed_ids) | list)
if (mtv_allowed_ids | default([]) | length > 0)
else mtv_networks
}}This way the filter only kicks in when there are actual include IDs, and exclude-only overrides still iterate the full list (letting the existing downstream task handle the exclusion as it does today). Minor nit: might also be worth initializing |
Description
This PR optimizes the network map creation process by filtering the source networks (vSphere/Ovirt) based on the provided overrides before entering the task loop.
Motivation
Beyond performance, this change addresses a critical scalability issue. In large-scale vSphere environments (e.g., thousands of PortGroups), attempting to map every available network results in an oversized NetworkMap Custom Resource.
This often leads to:
API Server Rejection: The generated resource exceeds the standard 1MB limit for objects in etcd, causing the k8s module to fail with a "Request entity too large" error.
Controller Timeout: The MTV controller struggles to reconcile objects with massive spec.map arrays, leading to timeouts and instability.
By filtering the networks at the source, we ensure that the resulting NetworkMap object only contains relevant entries, keeping the resource footprint small and within Kubernetes API limits.
Changes
mtv_allowed_idsfact to capture required network IDs from overrides.Process VMware NetworksandProcess Ovirt Networksto only iterate over the filtered list when overrides are defined.Impact
Significant reduction in playbook execution time for large-scale environments.