Empty resultset on instant query regex

[niksajakovljevic]

Describe the bug
Promscale returns empty results although data is present in the DB.

To Reproduce

Send query:
/api/v1/query?query=count by (job, instance, version) (prometheus_build_info{job=~".+", instance=~".+"})

Response:
{"status":"success","data":{"resultType":"vector","result":[]}}

Version

• Distribution/OS: K8s
• Promscale: 0.16
• TimescaleDB: 2.8.1

[JamesGuthrie]

I've had difficulty reproducing this issue (on Promscale master).

> curl -v 'http://localhost:9201/api/v1/query?query=count%20by%20(job%2C%20instance%2C%20version)%20(prometheus_build_info%7Bjob%3D~%22.%2B%22%2C%20instance%3D~%22.%2B%22%7D)&time=1669380801.153&_=1669380787421'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"cbpt:9090","job":"prometheus","version":"2.31.2+ds1"},"value":[1669380801.153,"1"]}]}}

When removing the time parameter, I was able to reproduce this once:

> curl -v 'http://localhost:9201/api/v1/query?query=count%20by%20(job%2C%20instance%2C%20version)%20(prometheus_build_info%7Bjob%3D~%22.%2B%22%2C%20instance%3D~%22.%2B%22%7D)'
{"status":"success","data":{"resultType":"vector","result":[]}}

But subsequent requests all returned the result:

>curl -v 'http://localhost:9201/api/v1/query?query=count%20by%20(job%2C%20instance%2C%20version)%20(prometheus_build_info%7Bjob%3D~%22.%2B%22%2C%20instance%3D~%22.%2B%22%7D)'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"cbpt:9090","job":"prometheus","version":"2.31.2+ds1"},"value":[1669383051.781,"1"]}]}}

> curl -v 'http://localhost:9201/api/v1/query?query=count%20by%20(job%2C%20instance%2C%20version)%20(prometheus_build_info%7Bjob%3D~%22.%2B%22%2C%20instance%3D~%22.%2B%22%7D)'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"cbpt:9090","job":"prometheus","version":"2.31.2+ds1"},"value":[1669383087.091,"1"]}]}}

> curl -v 'http://localhost:9201/api/v1/query?query=count%20by%20(job%2C%20instance%2C%20version)%20(prometheus_build_info%7Bjob%3D~%22.%2B%22%2C%20instance%3D~%22.%2B%22%7D)'
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"instance":"cbpt:9090","job":"prometheus","version":"2.31.2+ds1"},"value":[1669383122.1,"1"]}]}}

At the very least, it doesn't look like the issue is because regexes are broken in principle.

[niksajakovljevic]

I didn't get deep into this but if it's only reproducible when time is not present I have one idea. Since timestamp is in that case generated on Promscale (time.now()) and sent to query engine. So possibly we might not have a record in DB for given timestamp? I'll look more into this.

[niksajakovljevic]

Hmm in my new environment I am somehow unable to reproduce this problem.

~ > seq 1 10 | xargs -n1 -P 10 curl -v --data-urlencode 'query=/api/v1/query?query=count by (job, instance, version) (prometheus_build_info{job=~".+", instance=~".+"})' http://localhost:9201/api/v1/query