[Feature]: Delete nodes from the UI and reshard?

[Jonathan-Eid]

Is your feature request related to a problem?

I setup a 4 node shard, the scheduler is in a remote k8s cluster. Worker nodes are in a different remote k8s cluster.
One worker pod was coming up, but got rescheduled, leaving a ghost node inside the parallax join process, it came back as a duplicate node and now the parallax cluster is waiting for the ghost node to come back up.

So the parallax cluster now detects 4 out of 6 nodes to be online, but is still waiting on 1 ghost node and 1 duplicate node before the chat can start to work

Describe the Solution you'd like

Remove unresponsive nodes from parallax cluster UI and rebalance the weights across the working/online worker nodes.

Also I see your discord invite is down. I'm an employee at a top cryptocurrency company building out AI infrastructure, would love to incorporate your stuff into our products, would be nice to chat with the devs to gain a better understanding of Parallax. Please let me know if I'm able to reach out. Thanks Gradient for your cool project!

Alternatives Considered (Optional)

No response

Additional Context (Optional)

No response

[Jonathan-Eid]

I've been digging into the code a bit more, seems like you have some rebalancing logic in there. Still the frontend doesn't really display everything going on I assume and doesn't expose controls to override what's going on in the backend. I'm wondering why the heartbeat logic never kicked in in my case tho