semcode: Guard against compaction hang with many fragments

LanceDB compaction encounters a pathological case when a table
accumulates thousands of small fragments. The compact operation
enters a CPU loop where the main thread spins at 100% CPU
utilization while worker threads remain idle. Any table
subjected to many small appends without intervening compaction
can reach this state.

A check now examines fragment count before compaction proceeds.
When fragment count exceeds 500, compaction is skipped and a
warning directs the user to rebuild the database with --clear.
This threshold prevents the hang condition while allowing normal
compaction for tables with moderate fragmentation. Prune, index,
and checkout operations remain unaffected; only the compact step
is gated by this fragment limit.

Fixes: 4a16e15 ("semcode-index: optimize database periodically during long-running indexing")
Signed-off-by: Chuck Lever <cel@kernel.org>

Author: chucklever

src/database/schema.rs

@@ -1016,15 +1016,49 @@ impl SchemaManager {
         let mut success = true;
 
         // 1. Compact files
-        if let Err(e) = table
-            .optimize(OptimizeAction::Compact {
-                options: Default::default(),
-                remap_options: None,
-            })
-            .await
-        {
-            tracing::warn!("Failed to compact table {}: {}", table_name, e);
-            success = false;
+        const MAX_COMPACT_FRAGMENTS: usize = 500;
+
+        let should_compact = match table.stats().await {
+            Ok(stats)
+                if stats.fragment_stats.num_fragments
+                    > MAX_COMPACT_FRAGMENTS =>
+            {
+                tracing::warn!(
+                    "Skipping compaction for table {} \
+                     ({} fragments exceeds {} limit -- \
+                     rebuild with --clear to resolve)",
+                    table_name,
+                    stats.fragment_stats.num_fragments,
+                    MAX_COMPACT_FRAGMENTS
+                );
+                false
+            }
+            Ok(_) => true,
+            Err(e) => {
+                tracing::warn!(
+                    "Failed to read stats for table {}: {}",
+                    table_name,
+                    e
+                );
+                true
+            }
+        };
+
+        if should_compact {
+            if let Err(e) = table
+                .optimize(OptimizeAction::Compact {
+                    options: Default::default(),
+                    remap_options: None,
+                })
+                .await
+            {
+                tracing::warn!(
+                    "Failed to compact table {}: {}",
+                    table_name,
+                    e
+                );
+                success = false;
+            }
         }
 
         // 2. Prune ALL old versions