Fix: Disable WAP support by default for Spark and Iceberg#5415
Fix: Disable WAP support by default for Spark and Iceberg#5415izeigerman merged 1 commit intomainfrom
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR disables Write-Audit-Publish (WAP) support by default for Spark and Iceberg combinations due to deteriorated functionality after removing Airflow integration tests. The change introduces a configurable wap_enabled flag that defaults to false, providing an option to re-enable WAP in the future once proper testing is restored.
- Add
wap_enabledconfiguration flag with default value ofFalseto disable WAP by default - Update conditional logic to check WAP enablement before applying WAP-related operations
- Modify tests to parameterize WAP behavior and ensure proper coverage for both enabled/disabled states
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| sqlmesh/core/config/connection.py | Add wap_enabled configuration field to SparkConnectionConfig |
| sqlmesh/core/engine_adapter/base.py | Add wap_enabled property to base engine adapter |
| sqlmesh/core/engine_adapter/spark.py | Update create table logic to conditionally apply WAP based on enablement |
| sqlmesh/core/snapshot/evaluator.py | Add WAP enablement check to evaluation logic |
| tests/core/engine_adapter/test_spark.py | Parameterize tests for WAP enabled/disabled scenarios |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| do_dummy_insert = False | ||
| if self.wap_enabled: |
There was a problem hiding this comment.
The variable do_dummy_insert is declared but never used in this diff. This appears to be dead code that should be removed or the logic that uses this variable is missing from the implementation.
After deprecate Airflow integration and removing the corresponding test suite we lost our only avenue for testing Spark + Iceberg combination. As a result, the Iceberg-specifc WAP support fell behind and deteriorated as the recent wave of Spark + Iceberg issues suggests.
This update disables WAP by default with the option to enable it in future. This should buy us time to introduce back continuous testing for the Spark + Iceberg combo and address all issues.