What is the current behavior?
When running Helm charts on GKE, the engine.modelManager.models.add and engine.modelManager.models.remove fields do not add or remove models to the connected volume.
The required workaround is to:
- create an entirely new volume,
- add all your models to the new volume, then
- upgrade the Helm chart with the new Engine deployment pointing to the new volume.
- Once all the new Engines are up and there are none left pointing to the old volume, you can recycle the old volume.
Please tell us about your environment
Google Cloud Platform / GCP / Google Kubernetes Engine / GKE
Historical Context
At the time of writing the Helm chart, GKE had restrictions around GPU usability that, surprisingly, limited the available storage options. The only other alternative was to choose a high-cost storage option, which was also undesirable.
As a result, the Helm chart does not allow models to be added or removed to an existing volume when managed by GKE.
Google / GKE may have fixed this limitation recently, in which case the Helm charts could be updated to support the model add/remove behavior for GKE.
What is the current behavior?
When running Helm charts on GKE, the
engine.modelManager.models.addandengine.modelManager.models.removefields do not add or remove models to the connected volume.The required workaround is to:
Please tell us about your environment
Google Cloud Platform / GCP / Google Kubernetes Engine / GKE
Historical Context
At the time of writing the Helm chart, GKE had restrictions around GPU usability that, surprisingly, limited the available storage options. The only other alternative was to choose a high-cost storage option, which was also undesirable.
As a result, the Helm chart does not allow models to be added or removed to an existing volume when managed by GKE.
Google / GKE may have fixed this limitation recently, in which case the Helm charts could be updated to support the model add/remove behavior for GKE.