[SAP] modify select_datastore_by_name to use a cache#323
Open
hemna wants to merge 1 commit intostable/2025.1-m3from
Open
[SAP] modify select_datastore_by_name to use a cache#323hemna wants to merge 1 commit intostable/2025.1-m3from
hemna wants to merge 1 commit intostable/2025.1-m3from
Conversation
Optimize the select_datastore_by_name method to significantly reduce vCenter API load during high-volume operations like boot-from-volume. Problem: The original implementation called _get_datastores() which fetches ALL datastores with their 'host' and 'summary' properties. With ~40 datastores and ~50 hosts each, this transferred ~500KB-2MB of data per call. Under high load (e.g., 96 volume creates in 5 minutes), this caused vCenter connection pool exhaustion and cascading timeouts averaging 530 seconds per _select_ds_for_volume call. Solution: 1. Add _get_datastore_by_name() method that fetches properties for only the specific datastore needed, reducing data transfer to ~10-20KB. 2. Add a 5-minute TTL cache for datastore name -> moref mappings in get_ds_ref_by_name(). Since volume creates are bursty and typically target the same datastores, this eliminates repeated vCenter queries for the lightweight name lookup on cache hits. 3. The host availability check still queries vCenter on every call to ensure we always have fresh data about which hosts are connected and not in maintenance mode. Performance impact: - First call: 1 lightweight query (names only) + 1 targeted query - Subsequent calls (within 5 min): 1 targeted query only (cache hit) - Original: 1 heavy query fetching all datastores with all host mounts Change-Id: If9d46fe833418b67393535384af075a95f2ca4cb
grandchild
reviewed
Apr 1, 2026
| self._ds_regex = ds_regex | ||
| self._profile_id_cache = {} | ||
| self._ds_name_cache = {} # {name: (moref, timestamp)} | ||
| self._ds_cache_ttl = 300 # 5 minutes |
There was a problem hiding this comment.
i get that this is a hotfix, but this should be a configurable. at least a #TODO(): make configurable.
| for obj_content in objects: | ||
| props = self._get_object_properties(obj_content) | ||
| if props['name'] == name: | ||
| # Cache the result |
There was a problem hiding this comment.
comment not needed, it's right there in the _ds_name_cache attr name.
There was a problem hiding this comment.
same as the other comments, "check cache", "delete from cache".
Scsabiii
approved these changes
Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Optimize the select_datastore_by_name method to significantly reduce
vCenter API load during high-volume operations like boot-from-volume.
Problem:
The original implementation called _get_datastores() which fetches ALL
datastores with their 'host' and 'summary' properties. With ~40 datastores
and ~50 hosts each, this transferred ~500KB-2MB of data per call. Under
high load (e.g., 96 volume creates in 5 minutes), this caused vCenter
connection pool exhaustion and cascading timeouts averaging 530 seconds
per _select_ds_for_volume call.
Solution:
Add _get_datastore_by_name() method that fetches properties for only
the specific datastore needed, reducing data transfer to ~10-20KB.
Add a 5-minute TTL cache for datastore name -> moref mappings in
get_ds_ref_by_name(). Since volume creates are bursty and typically
target the same datastores, this eliminates repeated vCenter queries
for the lightweight name lookup on cache hits.
The host availability check still queries vCenter on every call to
ensure we always have fresh data about which hosts are connected
and not in maintenance mode.
Performance impact:
Backport of #322 from stable/2023.1-m3.
Change-Id: If9d46fe833418b67393535384af075a95f2ca4cb