fix: Native_datafusion reports correct files and bytes scanned

[0lai0]

Which issue does this PR close?

Closes #3791

Rationale for this change

In CometScanExec, calling getFilePartitions() unconditionally executes sendDriverMetrics(). Because getFilePartitions() can be evaluated multiple times during planning (e.g., converting to CometNativeScanExec) and execution (e.g., fetching partitions), the SQLMetric accumulators like numFiles and filesSize were being duplicated. This led to incorrect double-counted values rendering in the Spark UI.

What changes are included in this PR?

• Replaced metrics(...).add() with metrics(...).set() in CometScanExec to ensure idempotency when reporting metrics.
• Wrapped the driver metric updates and Spark listener event dispatching inside a lazy val. This prevents both double-counting during Catalyst transformations (makeCopy) and sending redundant UI events.

How are these changes tested?

• Added a dedicated end-to-end unit test in CometExecSuite.
• The test writes a dummy Parquet dataset, sequentially triggers multiple UI actions (count and collect) to force severe plan evaluations, and strictly asserts that numFiles is exactly 2 without any duplication.

[andygrove]

LGTM. Thanks @0lai0.

[mbutrovich]

So basically it's an artifact of wrapping CometNativeScan in CometScan, which we hopefully won't do in the future anyway.

Thanks for the fix in the meantime, @0lai0!

[comphead]

Thanks @0lai0 I'll quickly check it out today

[comphead]

Somehow on UI I can now see 0

number of files read: 0
size of files read: 0.0 B

[comphead]

Please include --conf spark.comet.scan.impl=native_datafusion

[0lai0]

Thank you all for the feedback. I’ll investigate this matter and fix it.