Allow users with delete permissions to drop replica tables

[Will-Lo]

Summary

Issue] Briefly discuss the summary of the changes made in this
pull request in 2-3 lines.

Replica tables have limited permissions compared to source tables. This prevents users from updating their replica tables manually as it would lead to table corruption.

However, being unable to drop replica tables easily causes a lot of operational toil. This is because users who want to perform backwards incompatible schema changes or want to drop their table for another reason must reach out to system administrators for them to delete.

Additionally, since most replica tables are copied on a schedule, they generally can be backfilled if deletions were accidentally done.

This PR relaxes the drop restriction on replica tables to anyone with the drop capability for the db (generally just table owners).

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[maluchari]

Summary

Issue] Briefly discuss the summary of the changes made in this pull request in 2-3 lines.

Replica tables have limited permissions compared to source tables. This prevents users from updating their replica tables manually as it would lead to table corruption.

However, being unable to drop replica tables easily causes a lot of operational toil. This is because users who want to perform backwards incompatible schema changes or want to drop their table for another reason must reach out to system administrators for them to delete.

Additionally, since most replica tables are copied on a schedule, they generally can be backfilled if deletions were accidentally done.

This PR relaxes the drop restriction on replica tables to anyone with the drop capability for the db (generally just table owners).

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

Replica tables are more for DR. Dropping them should not be easy for users. Can we consider having them soft deleted instead ? This would unblock users and also ensure DR is available if required.

[Will-Lo]

Soft deletion will not work because soft deletion keeps them on the same file path without the catalog reference, we don't move files to a .trash path. We could consider that as part of the soft delete implementation but it'll add to storage overhead.

I think we need to decide on a stance with replica tables. Do we want to make them self serve or fully managed, because if we want to make them fully managed we need to invest in more automation around the drop patterns. I'm more in favor of self serve as I've yet to see a use case where users need to backfill from destination cluster.

[cbb330]

This PR significantly reduces operational burden and addresses what is likely a top-5 production issue. I believe it’s important to merge.

The main concern is the safety implication. Like many changes, this is a tradeoff between safety and operability. Here’s why I still think merging is the right call, along with some mitigations and context:

1. DROP TABLE is widely understood to be a high-risk operation, and users generally treat it as such.
2. Users can already drop their primary table, and they’ve been able to do so for years without a built-in recovery mechanism.
3. Replication is not always used as a disaster-recovery mechanism by customers; many use it for other workflows.
4. A total-loss scenario requires a fairly unlikely sequence, e.g., a user issuing DROP TABLE on both the source and destination clusters with no alternative backups.
5. This moves us toward more automation, and we can continue improving safeguards over time (e.g., a stronger soft purge / better guardrails).
6. Recovery is still possible in some cases, even if it’s not as robust as a true soft delete. Implementation detail: when soft delete is disabled, HdfsStorage calls deallocateTableStorage() to physically move/delete data.

For reference, LinkedIn’s implementation does the following, which is typically sufficient in practice:

• Cleans up internal folders (e.g., .trash, .temp subfolders within the table)
• Attempts Trash.moveToAppropriateTrash() using Hadoop’s native trash mechanism
• Moves data to the user’s .trash directory with a 6-hour retention (fs.trash.interval = 360)
• Falls back to permanent delete if the trash move fails (e.g., quota exceeded, cross-cluster issues)

[teamurko]

1. how many times users get blocked on this + oh on call load vs 2) drop by mistake + recovery overhead, we kind of have a sense of 1 being a problem, but no data on 2. i'd suggest trying 2 and if it causes more problems than it solves, can we revert

[maluchari]

Soft deletion will not work because soft deletion keeps them on the same file path without the catalog reference, we don't move files to a .trash path. We could consider that as part of the soft delete implementation but it'll add to storage overhead.

I think we need to decide on a stance with replica tables. Do we want to make them self serve or fully managed, because if we want to make them fully managed we need to invest in more automation around the drop patterns. I'm more in favor of self serve as I've yet to see a use case where users need to backfill from destination cluster.

Most of the requests are because primary got recreated. So having soft delete + 1 day retention for these should work. If same uuid table needs to be recreated, that indicates an underlying issues that needs to be fixed, like multi schema updates.

[maluchari]

1. how many times users get blocked on this + oh on call load vs 2) drop by mistake + recovery overhead, we kind of have a sense of 1 being a problem, but no data on 2. i'd suggest trying 2 and if it causes more problems than it solves, can we revert

2 could be a problem as well and that is the motivation for soft delete. If an incorrect flow is deployed on multiple instances that leads to DR table also getting deleted, then enabling drop defeats the purpose of having DR in place. soft delete should also help bring down oh oncall load and we can have it just for a day

[dushyantk1509]

Merging this PR based on internal alignment as this provides a self-serve way for users to fix broken replication. This will be revisited once we've source controlled replication.