[Improvement]: Collect table_summary metrics independently of self-optimizing

[j1wonpark]

Search before asking

• I have searched in the issues and found no similar issues.

What would you like to be improved?

Currently, table_summary metrics (total files, file sizes, health score, etc.) are only collected when self-optimizing.enabled=true. This is because the metric update path (setTableSummary()) is gated behind the optimizingConfig.isEnabled() check in TableRuntimeRefreshExecutor.tryEvaluatingPendingInput().

As a result, tables with self-optimizing.enabled=false always show 0 or N/A for key monitoring metrics (Health Score, Total Files, Total Files Size) in Grafana dashboards, even though the table has actual data files.

Metric collection and self-optimizing execution are conceptually independent concerns. Users should be able to monitor table health without enabling the optimizing process.

How should we improve?

Introduce a new table property self-optimizing.table-summary.enabled that allows metric collection to be enabled independently of self-optimizing.

Behavior matrix:

self-optimizing.enabled	table-summary.enabled	Optimizing runs	Metrics collected
true	(any)	Yes	Yes
false	true	No	Yes
false	false (default)	No	No

Implementation approach:

1. Add SELF_OPTIMIZING_TABLE_SUMMARY_ENABLED constant in TableProperties
2. Add tableSummaryEnabled field to OptimizingConfig
3. Parse the new property in TableConfigurations.parseOptimizingConfig()
4. Add an else if branch in TableRuntimeRefreshExecutor.tryEvaluatingPendingInput() to collect summary metrics when optimizing is disabled but table-summary is enabled

The default value of false ensures no behavioral change for existing tables.

Usage example:

ALTER TABLE db.my_table SET TBLPROPERTIES (
  'self-optimizing.enabled'               = 'false',
  'self-optimizing.table-summary.enabled' = 'true'
);

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Subtasks

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[xxubai]

Collecting table summaries for all tables is quite expensive.
I support making the table summary collection configurable independently from self-optimizing, but I wouldn’t suggest enabling it for all tables by default, as reading metadata files for large tables can incur heavy I/O.

Perhaps we can provide an interface to trigger summary collection on demand and then compute the health score. What do you think?

[j1wonpark]

Thanks for the feedback, @xxubai!

Regarding the I/O concern — today, self-optimizing.enabled defaults to true, which means metadata scanning is already happening for all Iceberg tables. This proposal only targets the case where users explicitly disable self-optimizing (e.g. at the catalog level). Even then, table-summary.enabled defaults to false, so no additional scanning occurs unless a user opts in per table. There should be no extra overhead for existing users.

An on-demand API trigger is an interesting idea. However, considering Amoro's current UX — where table behavior is driven by table properties and the system takes care of the rest — I think automatic periodic collection via a property feels more natural and consistent. Users wouldn't need to integrate a separate API call into their workflow; they just set a property and monitoring works.

That said, an on-demand endpoint could complement this as a future enhancement for ad-hoc use cases. Happy to discuss further.