Skip to content

feature: Add Batch Processing #121

@jjeff07

Description

@jjeff07

Requires #118.

Summary:

Sync'ing large number of objects takes a significant portion of time. Currently the package processes synchronously one at time and any failure stops the entire process. So if you have 5k devices that could be 500k interfaces and the design does not scale. By implementing asynchronous and batch processing it would greatly reduce the time and allow for more data and integrations. Another benefit would be that you can add timestamps during processing.

User Story Discovery Tool (IP Fabric, Slurp'it, etc.): Users want to add a timestamp for when the data was last refreshed or when a device was discovered. If a user decides to do this then every sync would have to update every object in Infrahub. Without batch processing this could take hours every day.

Design:

  • Add a flag to the Schema Mapping config (perhaps after testing a global flag could be introduced).
  • To keep the design simple users must enable continue on failure flag from feature: Continue on Failure for Sync #118
  • If continue on failure flag is not set then the config file check should fail prior to running any other code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions