Skip to content

fix: Keep sync_container_lifecycles() bgtask alive in a loop.#2178

Merged
fregataa merged 3 commits intomainfrom
topic/05-23-fix_enhanced_kernel_termination_handling
Jul 5, 2024
Merged

fix: Keep sync_container_lifecycles() bgtask alive in a loop.#2178
fregataa merged 3 commits intomainfrom
topic/05-23-fix_enhanced_kernel_termination_handling

Conversation

@fregataa
Copy link
Copy Markdown
Member

@fregataa fregataa commented May 26, 2024

Why resource sync API depends on this PR?

TL; DR

sync_containers_lifecycle() should work

Detail

We have 3 different sources of Kernel / Container data

  1. Manager side DB
  2. Agent side kernel_registry
  3. Agent side actual docker containers

I tried to synchronize ALL Agent's data to Manager's DB but synchronizing actual containers to kernel_registry or manager's data needs to process container destruction tasks which take a long time.
Since sync_containers_lifecycle() task synchronizes actual containers to kernel_registry periodically in background, resource-sync API only needs to sync kernel_registry to Manager side DB if it is ensured that the background task stays alive and works as we expect.

Changes

Fix

  • Keep sync_container_lifecycles() bgtask alive in a loop by handling any exceptions raised in the task.
  • In the try-except context of container creation, do not use unbound local variable

Enhance

  • Fetch all containers eagerly in sync_container_lifecycles()

Checklist: (if applicable)

  • Milestone metadata specifying the target backport version

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:agent Related to Agent component comp:common Related to Common component size:L 100~500 LoC urgency:4 As soon as feasible, implementation is essential.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant