fix: sync agent's kernel-registry to actual container periodically

[fregataa]

Intro

The status information of the container is divided into three types in BackendAI system: DB on the manager side, agent's kernel registry, and actual container. This PR is about agent's kernel registry and actual container.

Problem

In the current implementation, kernel data is inserted and removed from the agent's kernel registry in the task of creating and destroying containers. In the case of a container creation, when any unhandled error occurs, the kernel data inserted into the kernel registry is removed. Such removing is not reliable and any other unpredictable errors can cause mismatch between kernel registry and actual container state.
So, let's sync kernel registry to the actual container state in a periodic loop.

Checklist: (if applicable)

• Milestone metadata specifying the target backport version

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “flow:merge-queue” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “flow:hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[fregataa]

• fix: sync agent's kernel-registry to actual container periodically #2179 👈
• fix: Keep sync_container_lifecycles() bgtask alive in a loop. #2178 : 3 other dependent PRs (#2262 , #2317 , #2338 )
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @fregataa and the rest of your teammates on Graphite

[fregataa]

Close this PR since the idea of it is not confirmed