perf(cdr): optimize queries for large CDR installations

[edospadoni]

Problem

The tasks component queries take 15-20 minutes on installations with ~2M CDR records.
Main bottlenecks:

• date_format() on indexable columns preventing index usage
• No indexes on generated cdr_YYYY and cdr_YYYY-MM tables
• Correlated subqueries for trunk type detection executed per-row
• Correlated subqueries for geo-lookup (region/province) executed millions of times
• No indexes on the source cdr table
• REGEXP usage where LIKE suffices
• Sequential execution of SQL views

Optimizations

1. Indexes on source cdr table

• Added idx_cdr_calldate (calldate) for range scans
• Added idx_cdr_linkedid (linkedid) for GROUP BY
• Created both in schema.sql.tmpl and idempotently in cdr.go (ensureCDRSourceIndexes)

2. Fix date_format() anti-pattern with range scans

• cdr_year.sql: date_format(calldate, "%Y") = "YYYY" replaced with calldate >= 'YYYY-01-01' AND calldate < ... + INTERVAL 1 YEAR
• cdr_month.sql: date_format(calldate, "%Y-%m") replaced with range scan using INTERVAL 1 MONTH
• Daily INSERT IGNORE: uses DATE(NOW() - INTERVAL 1 DAY) / DATE(NOW())

3. Pre-compute trunk type with LEFT JOIN

• Replaced correlated subquery get_trunk_name(channel) IN (SELECT channelid FROM asterisk.trunks) with a temp table _trunk_list + LEFT JOIN
• From N subqueries per row to a single JOIN

4. Indexes on generated cdr_YYYY and cdr_YYYY-MM tables

• Yearly tables: idx_type_calldate, idx_type, idx_cnum, idx_dst, idx_channel, idx_dstchannel, idx_type_cnum_calldate
• Monthly tables: idx_type_calldate, idx_type, idx_cnum, idx_dst, idx_calldate
• Created in SQL templates (ADD INDEX IF NOT EXISTS) and idempotently in Go (ensureTableIndexes)

5. Pre-computed geographic columns

• Added 4 columns to cdr_YYYY: src_region, src_province, dst_region, dst_province
• Populated via lookup table on unique phone numbers (order of magnitude fewer than total records)
• Incremental updates: WHERE src_region IS NULL (only new records)
• Dashboard views (6, 7, 17, 18) rewritten to read columns directly instead of correlated subqueries

6. REGEXP replaced with LIKE for dispositions

• Replaced REGEXP 'ANSWERED' with LIKE '%ANSWERED%' across 12 SQL files
• Updated Go function ExtractDispositions in utils.go
• REGEXP 'FOO$' becomes LIKE '%FOO' (semantically equivalent, much faster)

7. Parallel view execution

• views.go now executes SQL views in parallel with 4 worker goroutines
• Views are independent, no conflicts

Benchmark (installation with ~665K records/year, ~8M total)

Typical dashboard query (SELECT type, COUNT(*) ... WHERE type='IN' AND calldate >= ... GROUP BY type):

	Time	Rows scanned
Before (full scan)	1.25s	665,282
After (index scan)	0.092s	332,641
Speedup	13.6x	Using index (index-only)

Daily INSERT on source cdr table:

	Scan type	Rows
Before	ALL	7,756,643
After	range	59,302
Speedup	130x fewer rows

Migration

No manual operations required. Migration is fully automatic:

1. SQL templates (cdr_year.sql, cdr_month.sql, dashboard views): re-executed nightly by cron (tasks cdr + tasks views), so they update automatically on the first run after the update.

2. Indexes on existing tables: ensureCDRSourceIndexes() and ensureTableIndexes() in cdr.go check for index existence via information_schema and add them if missing. After the first run they become a fast no-op (metadata SELECT only).

3. Geographic columns: ADD COLUMN IF NOT EXISTS + WHERE src_region IS NULL ensures:

   • First run after update: all rows are updated (new columns = NULL)
   • Subsequent runs: only yesterday's new records are updated (incremental)

4. Schema template (schema.sql.tmpl): indexes on cdr use information_schema logic for compatibility with the @database_name placeholder.

All operations are idempotent and safe to re-execute.

Test plan

• Verify indexes: SHOW INDEX FROM cdr_YYYY WHERE Key_name LIKE 'idx_%'
• Verify geo columns: SELECT src_region, COUNT(*) FROM cdr_YYYY WHERE type='IN' GROUP BY src_region
• Verify EXPLAIN uses indexes: EXPLAIN SELECT COUNT(*) FROM cdr_YYYY WHERE type='IN' AND calldate >= ...
• Run tasks cdr with DEBUG=1 and verify completion
• Run tasks views and verify completion
• Compare before/after times on installation with ~2M records