perf(dir-catalog): avoid unchanged manifest reloads

[jackye1995]

Summary:

• add dataset stale/successor helpers and commit-handler version probes
• use the successor probe to skip unchanged directory manifest reloads
• cover V1/V2 successor checks, external-manifest fallback, and cross-namespace manifest refresh

[codecov]

Codecov Report

❌ Patch coverage is 89.53488% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/dir/manifest.rs	58.33%	2 Missing and 3 partials ⚠️
rust/lance/src/dataset.rs	80.95%	2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[jackye1995]

Benchmark comparison against #7176

Compared this PR's unchanged-read benchmark with the prior copy-on-write __manifest benchmark from #7176: #7176 (comment)

Important caveat: #7176 measured write-create-namespace commit throughput on EC2 c7i.48xlarge; this run measured warm-read-describe-table on EC2 c7i.12xlarge. So this is not an apples-to-apples replacement benchmark. It is useful as a reference because #7176 established the CoW write-side cost, while this PR targets unchanged read-side reload cost.

This PR: unchanged warm reads

Setup: S3 jack-devland-build, us-east-1, inline optimization enabled, 80 measured ops, 10 warmup ops.

__manifest entries	upstream main ops/s	this PR ops/s	throughput delta	upstream avg ms	this PR avg ms	avg latency delta	upstream p50 / p99 ms	this PR p50 / p99 ms
1,000	6.342	21.959	3.46x	125.83	38.80	-69.2%	87.59 / 339.00	37.00 / 84.40
10,000	14.690	18.634	1.27x	57.87	44.65	-22.8%	49.29 / 158.08	40.90 / 92.98
100,000	9.908	13.900	1.40x	87.72	63.65	-27.4%	75.90 / 215.51	46.96 / 123.09

All runs had 0 errors.

Reference from #7176: CoW write throughput

For the same row counts, #7176 reported continuous single-process write-create-namespace numbers:

__manifest entries	#7176 inline write ops/s	#7176 inline p50 / p99 ms	#7176 no-index write ops/s	#7176 no-index p50 / p99 ms	this PR warm-read ops/s	this PR warm-read p50 / p99 ms
1,000	2.04	442 / 990	3.55	271 / 394	21.959	37.00 / 84.40
10,000	1.50	639 / 1034	2.43	408 / 520	18.634	40.90 / 92.98
100,000	1.13	811 / 1722	2.06	478 / 637	13.900	46.96 / 123.09

Takeaway: #7176 shows CoW commits remain single-writer and roughly O(rows). This PR does not change write-side commit behavior. It improves the unchanged read path by avoiding a full latest-manifest reload when current_version + 1 does not exist. In this S3 run, that gives 1.27x-3.46x higher unchanged-read throughput versus rebased upstream main, and keeps read p99 substantially below the CoW write baseline for the overlapping sizes.