You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sync'ing large number of objects takes a significant portion of time. Currently the package processes synchronously one at time and any failure stops the entire process. So if you have 5k devices that could be 500k interfaces and the design does not scale. By implementing asynchronous and batch processing it would greatly reduce the time and allow for more data and integrations. Another benefit would be that you can add timestamps during processing.
User Story Discovery Tool (IP Fabric, Slurp'it, etc.): Users want to add a timestamp for when the data was last refreshed or when a device was discovered. If a user decides to do this then every sync would have to update every object in Infrahub. Without batch processing this could take hours every day.
Design:
Add a flag to the Schema Mapping config (perhaps after testing a global flag could be introduced).
Requires #118.
Summary:
Sync'ing large number of objects takes a significant portion of time. Currently the package processes synchronously one at time and any failure stops the entire process. So if you have 5k devices that could be 500k interfaces and the design does not scale. By implementing asynchronous and batch processing it would greatly reduce the time and allow for more data and integrations. Another benefit would be that you can add timestamps during processing.
User Story Discovery Tool (IP Fabric, Slurp'it, etc.): Users want to add a timestamp for when the data was last refreshed or when a device was discovered. If a user decides to do this then every sync would have to update every object in Infrahub. Without batch processing this could take hours every day.
Design: