Queue size is always high and the grid is drastically underutilized #9601
Description
🐛 Bug Report
This is the average utilization and queue size at all times on the grid:
Sessions wait about 20 seconds in the queue, seemingly for no reason.
This problem is not present with grid version: 4.0.0-beta-4-prerelease-20210527, but present in the latest release 4.0.0-rc-1-prerelease-20210618.
To Reproduce
Detailed steps to reproduce the behavior:
Deploy the grid with the following configuration:
---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: selenium-grid-advanced
namespace: production
spec:
releaseName: selenium-grid-advanced
chart:
# https://github.com/zzvara/docker-selenium/tree/selenium-grid-helm-chart-0.2.0
repository: redacted
name: selenium-grid
version: 0.2.1
values:
global:
seleniumGrid:
imageTag: "4.0.0-rc-1-prerelease-20210618"
nodesImageTag: "4.0.0-rc-1-prerelease-20210618"
isolateComponents: false
chromeNode:
replicas: 60
resources:
limits:
cpu: '1'
memory: 2000Mi
requests:
cpu: '1'
memory: 2000Mi
firefoxNode:
replicas: 0
operaNode:
replicas: 0
hub:
imageTag: "4.0.0-rc-1-prerelease-20210618"
resources:
requests:
memory: "32Gi"
limits:
memory: "32Gi"
extraEnvironmentVariables:
- name: JAVA_OPTS
# @note "POOL_MAX" not confirmed that it is working.
value: "-Xmx20g -Xms2g -XX:+UseG1GC -DPOOL_MAX=512"
- name: SE_OPTS
value: "--max-threads 100 --allow-cors true"
- name: SE_SESSION_REQUEST_TIMEOUT
value: "30"
Expected behavior
A clear and concise description of what you expected to happen.
The queue should be close to 0 and concurrency should be high. Previous beta versions did not have this issue.
Test script or set of commands reproducing this issue
Unfortunately, scripts can not be disclosed, however, we reproduced the same problem with multiple tests/applications.
Environment
OS: Linux sigma02 5.10.43-flatcar #1 SMP Tue Jun 15 19:46:57 -00 2021 x86_64 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz GenuineIntel GNU/Linux
Grid version:
Browser: Chrome
Browser version: v.91.0
Selenium Grid version (if applicable): Grid version: 4.0.0-rc-1 (revision 23ece4f)