Outdated Text Embeddings Inference containers

[pocman]

This is a copy of this issue on Google Issuetracker

What you would like to accomplish:

Enable the deployment of Text Embeddings Inference (TEI) containers with versions huggingface-text-embeddings-inference-cu122.1-6-1.ubuntu2204 and huggingface-text-embeddings-inference-cu122.1-7.ubuntu2204 on Vertex AI endpoints. This would allow customers to utilize embedding models that require these specific TEI versions, such as snowflake-arctic-embed-m-v2.0.

How this might work:

The Vertex AI Product Team would build and make available the requested TEI container versions in the Vertex AI Model Registry.

If applicable, reasons why alternative solutions are not sufficient:

Current TEI versions are insufficient: The currently available TEI container versions on Vertex AI endpoints (up to 1.6.0) do not support the requirements of newer embedding models like snowflake-arctic-embed-m-v2.0.

Other information (workarounds you have tried, documentation consulted, etc):

The customer attempted to deploy snowflake-arctic-embed-m-v2.0 using the existing TEI containers on Vertex AI endpoints but encountered compatibility issues due to the older TEI version.

[alex35121]

Hey folks, we have been using the DLC image on GCP and it works great. TEI 1.8 added support for a lot of newer models, so we made some changes and built the image with TEI 1.8, it is now deployed on our Vertex AI endpoint with Qwen3 and it works amazing. I can open a PR to port our changes if it makes sense!

[rcleveng]

I'm hitting the same issue as in huggingface/text-embeddings-inference#672 where it seems since Google's repo has TEI 1.6 it's not really that usable (I'm trying to use the "Snowflake/snowflake-arctic-embed-l-v2.0" model)

The docs state to use the pre-built images, but they aren't getting prebuilt. What's the best option here? Where's the Dockerfile for the pre-built image and I'll build my own.

Reading up at the first comment, it's not clear who owns the overall issue of getting new builds made.

For anyone needing this right away, setup an artifact registry remote with the ghcr.io as it's upstream. Then just pull from there and use the images that are not google specific. I'm not sure what's added into the google images but it seems to work for the most part.

[alvarobartt]

Hey @pocman, @alex35121 and @rcleveng thanks for the comments and apologies for the wait, we were cooking something, see https://huggingface.co/blog/google-cloud.

As per #151 you can expect Text Embeddings Inference (TEI) v1.8.3 being released soon on Google Cloud as a DLC since the last release of the same DLC (it was v1.6!).

This being said, I'd suggest you to check actively the existing Hugging Face DLCs on Google Cloud with the following command:

gcloud container images list --repository="us-docker.pkg.dev/deeplearning-platform-release/gcr.io" | grep "huggingface-text-embeddings"

Additionally, I'll make sure to ping you guys here when those are released and happy to work with you moving forward to make sure that the aforementioned issues are resolved 🤗

[LiamTharp]

Hello! I just wanted to quickly echo this, I saw there was a release for 1.9.2 today, and I'd be very interested in using it for the qwen3 embedding models-

I'm attempting to follow the guide here and having troubles with the 1.6 version (outdated/mismatched documentation between the guide + TEI)

[alvarobartt]

Hey @LiamTharp thanks for flagging, I'm currently writing a dedicated example for Vertex AI (let me know if that'd be helpful!), but other than that you should be able to run Qwen3-Embedding in a similar fashion as https://huggingface.co/docs/google-cloud/examples/cloud-run-finewiki-embeddings-with-embedding-gemma but using the Google Cloud Deep Learning Container (DLC) instead.

Let me know if there's anything in particular that would be helpful other than adding a dedicated example for it, I'd love to know about use cases or potential applications to ship examples accordingly 🤗

[LiamTharp]

@alvarobartt That guide is exactly what I needed, I got a version of it working, but appreciate the direction. Thank you!

[alvarobartt]

Hey @pocman, @alex35121, @rcleveng, and @LiamTharp, sorry I didn't ping you guys earlier, but the latest Text Embeddings Inference (TEI) DLC is already released, but under a new distribution channel being "us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden", so the new images for both CPU and GPU live at:

• us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/hf-tei-official-cpu.1-9.debian12
• us-docker.pkg.dev/deeplearning-platform-release/vertex-model-garden/hf-tei-official-gpu.cu129.1-9.ubuntu2404

Feel free to use those moving forward, and closely monitor that channel for upcoming releases on our Deep Learning Containers (DLCs)! Closing this as resolved, but please let me know if there's anything I can do to help 🤗