Commit 1293ac1
committed
feat(store): replace PostgreSQL FTS with BM25 in HybridStore
Replace ts_rank_cd (PostgreSQL Full-Text Search) with BM25 algorithm
for better keyword search ranking in hybrid search.
Changes:
- Add bm25Language parameter (configurable via YAML)
- Replace FTS CTEs with bm25topk() function calls
- Add DISTINCT ON fixes to prevent duplicate results
- Add fuzzy matching with word_similarity (pg_trgm)
- Add score normalization (0-100 range)
- Add searchable attributes with field-specific boosting
- Bundle configuration in options.php and AiBundle.php
Tests:
- Update 6 existing tests for BM25 compatibility
- Add 3 new tests for fuzzy matching and searchable attributes
- All 19 tests passing (132 assertions)
Breaking changes:
- Requires plpgsql_bm25 extension instead of native FTS
- BM25 uses short language codes ('en', 'fr') vs FTS full names1 parent 2c7b49a commit 1293ac1
File tree
4 files changed
+564
-82
lines changed- src
- ai-bundle
- config
- src
- store
- src/Bridge/Postgres
- tests/Bridge/Postgres
4 files changed
+564
-82
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
765 | 765 | | |
766 | 766 | | |
767 | 767 | | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
768 | 772 | | |
769 | 773 | | |
770 | 774 | | |
| |||
774 | 778 | | |
775 | 779 | | |
776 | 780 | | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
| 791 | + | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
777 | 831 | | |
778 | 832 | | |
779 | 833 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1432 | 1432 | | |
1433 | 1433 | | |
1434 | 1434 | | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
| 1438 | + | |
| 1439 | + | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
| 1444 | + | |
| 1445 | + | |
| 1446 | + | |
| 1447 | + | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
| 1451 | + | |
| 1452 | + | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
1435 | 1467 | | |
1436 | 1468 | | |
1437 | 1469 | | |
| |||
0 commit comments