Documentation on the services created by the VMCluster object: Which ones to expose via ingress? How to find out the proper URLs?

[kastl-ars]

Dear maintainers,

we started looking into VictoriaMetrics (finally! :-)) and are very impressed by the smoothness of the deployment and the documentation.

One thing which confuses me and where I think the documentation could benefit from more substance and examples is regarding Kubernetes services and ingress(es) and URLs.

Using VMSingle

When using the VMsingle storage, I found a URL somewhere that I can expose via the VMAuth ingress (or rather disable authentication while exposing it).

spec:
  selectAllByDefault: true
  unauthorizedUserAccessSpec:
    url_prefix:
      - http://<vmsingle-victoriametrics-single>:8428
  ingress:
    class_name: openshift-default
    host: victoriametrics.example.org

(I also figured out which remoteWrite URL I need to set in the VMAgent...)

This works, I can query prometheus metrics and can use this as a datasource in Grafana.
(The targets are not shown, as reported in #1697 but that is just a minor thing and not my topic in this issue)

Using VMCluster

When using VMCluster instead, I had to guess a lot until I figured out that there are two things that can be exposed and do similar things.

• Using http://<vmagent-victoriametrics-agent>:8429 as `url_prefix I can expose the vmagent's GUI.
• Using http://<vmselect-victoriametrics-cluster>:8481/select/0/vmui/ I can expose the vmui

But I have not found documentation on how to expose two services? Do I need two VMAuth objects with different hostnames and url_prefixes (in case I want to disable authentication)?

And the vmui one throws an error when clicking the Explore Prometheus metrics button:

SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 5 of the JSON data

This seems to come from an error in the vmselect pod:

cannot fetch metric names from vmstorage nodes: cannot search metric names on vmstorage vmstorage-victoriametrics-cluster-<redacted>:8401: cannot execute funcName="searchMetricNames_v3" on vmstorage "100.125.138.99:8401": error when searching for metricIDs: the number of matching timeseries exceeds 30000; either narrow down the search or increase -search.max* command-line flag values (the most likely limit is -search.maxUniqueTimeseries)

This is a little surprising, as there is next to no workload being scraped. And this worked out of the box without issues using the VMSingle storage (in the same cluster). Did I pick the right URLs (or did I put the wrong URL somewhere?

Also, there are some kubernetes services called ...-internal which I did not find documentation about.

It could really be that I just misread or did not find the proper parts of the documentation, so I would gladly accept a link to the proper docs...

Thanks in advance!

Kind Regards,
Johannes