Is your feature request related to a problem? Please describe
The query_stats pipe is underused because most users either don't know about it or don't understand VictoriaLogs internals:
> * | query_stats;
BlocksProcessed 389
BytesProcessedUncompressedValues 3471122
BytesReadBlockHeaders 22168
BytesReadBloomFilters 0
BytesReadColumnsHeaderIndexes 43169
BytesReadColumnsHeaders 303231
BytesReadTimestamps 83977
BytesReadTotal 979963
BytesReadValues 527418
QueryDurationNsecs 9928948
RowsFound 13588
RowsProcessed 22400
TimestampsRead 22400
With very basic internal knowledge, they can understand why this query takes a lot of time and what can be improved. Besides giving this explanation, we can ask the user for disk I/O stats (e.g., IOPS) to infer some other metrics.
Describe the solution you'd like
It’s better to propose a blog post for internal understanding, but I’d like to propose a UI with visualizations that explain the query in plain language (and may include a link to the internal blog post). For example, above results could be written as:
These are the values returned by the | query_stats pipe:
QueryDurationNsecs = 9,928,948 (~9.93ms): total wall time spent executing the query.BlocksProcessed = 389: the query had to touch 389 data blocks. The fastest way to reduce this is using a narrower _time filter and a stream filter ({...} / _stream_id:...).RowsProcessed = 22,400: total log rows scanned inside these blocks.RowsFound = 13,588: rows that matched the query filters. In your case the selectivity is 22400 ~ 60.7% (so the filter isn't very selective).TimestampsRead = 22,400: _time values read for the processed rows (usually equals RowsProcessed for many queries).
Disk read breakdown (these counters are "bytes read by VictoriaLogs" during query execution):
BytesReadTotal = 979,963 (~0.98MB): total bytes read for executing the query.BytesReadValues = 527,418 (~0.53MB): bytes read for field values.BytesReadTimestamps = 83,977 (~0.08MB): bytes read for _time.BytesReadColumnsHeaders = 303,231 and BytesReadColumnsHeaderIndexes = 43,169: overhead for figuring out where needed columns are stored.BytesReadBlockHeaders = 22,168: block metadata overhead.BytesReadBloomFilters = 0: bloom filters weren't read for this query (either because the query didn't need bloom-filter-based checks, or because it used a path which doesn't read bloom filters).
Work done on uncompressed data:
BytesProcessedUncompressedValues = 3,471,122 (~3.47MB): this is the amount of uncompressed values bytes that were actually processed after decompression. It is usually bigger than BytesReadValues, because stored values are compressed on disk.
Quick rates for intuition (not a benchmark, just arithmetic for this query):
- Uncompressed processing rate: 3.47MB / 9.93ms ~ 350MB/s
- Read rate: 0.98MB / 9.93ms ~ 99MB/s
Is your feature request related to a problem? Please describe
The
query_statspipe is underused because most users either don't know about it or don't understand VictoriaLogs internals:With very basic internal knowledge, they can understand why this query takes a lot of time and what can be improved. Besides giving this explanation, we can ask the user for disk I/O stats (e.g., IOPS) to infer some other metrics.
Describe the solution you'd like
It’s better to propose a blog post for internal understanding, but I’d like to propose a UI with visualizations that explain the query in plain language (and may include a link to the internal blog post). For example, above results could be written as:
These are the values returned by the
| query_statspipe:QueryDurationNsecs= 9,928,948 (~9.93ms): total wall time spent executing the query.BlocksProcessed= 389: the query had to touch 389 data blocks. The fastest way to reduce this is using a narrower_timefilter and a stream filter ({...}/_stream_id:...).RowsProcessed= 22,400: total log rows scanned inside these blocks.RowsFound= 13,588: rows that matched the query filters. In your case the selectivity is 22400 ~ 60.7% (so the filter isn't very selective).TimestampsRead= 22,400: _time values read for the processed rows (usually equalsRowsProcessedfor many queries).Disk read breakdown (these counters are "bytes read by VictoriaLogs" during query execution):
BytesReadTotal= 979,963 (~0.98MB): total bytes read for executing the query.BytesReadValues= 527,418 (~0.53MB): bytes read for field values.BytesReadTimestamps= 83,977 (~0.08MB): bytes read for_time.BytesReadColumnsHeaders= 303,231 andBytesReadColumnsHeaderIndexes= 43,169: overhead for figuring out where needed columns are stored.BytesReadBlockHeaders= 22,168: block metadata overhead.BytesReadBloomFilters= 0: bloom filters weren't read for this query (either because the query didn't need bloom-filter-based checks, or because it used a path which doesn't read bloom filters).Work done on uncompressed data:
BytesProcessedUncompressedValues= 3,471,122 (~3.47MB): this is the amount of uncompressed values bytes that were actually processed after decompression. It is usually bigger thanBytesReadValues, because stored values are compressed on disk.Quick rates for intuition (not a benchmark, just arithmetic for this query):