cld2labs/llama-3.1-8b-instruct#98
Open
arpannookala-12 wants to merge 8 commits into
Open
Conversation
…ell EI Signed-off-by: arpannookala-12 <ganesh.arpan.nookala@cloud2labs.com>
Collaborator
|
Model deployment works. Testing inference is showing a Gateway Timeout error. vLLM pod is fine, but ingress-nginx-controller is giving an upstream timeout: |
alexsin368
reviewed
May 22, 2026
| @@ -0,0 +1,73 @@ | |||
|
|
|||
| # Deployed with EI Version-1.3.1 | |||
Collaborator
There was a problem hiding this comment.
we can delete this file since the instructions will all be in deployment.md
Collaborator
|
inference is functional after increasing ingress and APISIX timeout to 300s |
alexsin368
reviewed
May 26, 2026
| kubectl get ingress -A | grep <model-name> | ||
| ``` | ||
|
|
||
| Then annotate each ingress: |
Collaborator
There was a problem hiding this comment.
Suggested change
| Then annotate each ingress: | |
| Then annotate **EACH** ingress: |
Let's emphasize EACH since 2 is created.
alexsin368
reviewed
May 26, 2026
|
|
||
| **Cause:** | ||
|
|
||
| CPU-based model inference (`vllm-cpu`) generates tokens at ~0.3-0.4 tokens/s. Responses requiring more than ~24 tokens exceed the default 60s upstream timeout enforced by ingress-nginx and APISIX. |
Collaborator
There was a problem hiding this comment.
The performance is different for every Xeon SKU and will change over time. Let's just keep the note generic by only mentioning the root cause is the upstream timeout exceeds 60 seconds.
alexsin368
reviewed
May 26, 2026
| **Notes:** | ||
|
|
||
| - The nginx ingress annotation takes effect immediately; no pod restart required. | ||
| - For GPU-based deployments this timeout is rarely needed as throughput is significantly higher (30-50 tokens/s vs 0.3-0.4 tokens/s on CPU). |
Collaborator
There was a problem hiding this comment.
Same here, let's remove mentions of performance numbers as it will vary from SKU, config, and over time
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
third_party/Dell/model-deployment/llama-3.1-8b-instruct/