Something is not cleaning up smf instances when regions are deleted.
After running a test that creates then deletes a disk in a loop, I can see both the
memory usage going up for agent, and the log file listing more and more messages each time apply_smf() runs:
Message like these:
Nov 29 06:27:36.485 ERRO remove downstairs instance: downstairs-66dab510-7749-4191-adb3-330aec2124d7, component: worker
Nov 29 06:27:36.485 INFO instance states: (Some(Disabled), None), component: worker
Nov 29 06:27:36.485 ERRO remove downstairs instance: downstairs-98be2db0-165b-4adb-9c3d-52f2afab8fed, component: worker
Nov 29 06:27:36.486 INFO instance states: (Some(Disabled), None), component: worker
Nov 29 06:27:36.486 ERRO remove downstairs instance: downstairs-0c941299-6abe-427e-9d39-c9ff67f2aa6b, component: worker
Nov 29 06:27:36.487 INFO instance states: (Some(Disabled), None), component: worker
Nov 29 06:27:36.487 ERRO remove downstairs instance: downstairs-de23a0eb-1c79-4f28-8eb5-df689ad5dbd7, component: worker
Nov 29 06:27:36.487 INFO instance states: (Some(Disabled), None), component: worker
Each loop of the create/delete disk script results in another pair of messages printed.
Something yet TBD is not completely removing items from the svc.instances() list.