Skip to content

Remove unused utils, improve Iceberg and DeltaLake loaders #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
fordN merged 4 commits into from
Oct 23, 2025
Merged

Remove unused utils, improve Iceberg and DeltaLake loaders #3

fordN merged 4 commits into from
Oct 23, 2025

Conversation

@fordN
Copy link
Contributor

@fordN fordN commented Oct 1, 2025 •
edited
Loading

Changes

  • Remove loaders/utils.py (unused)
  • delta_lake_loader:
    • Simplify batch statistics, no compaction stats
  • iceberg_loader:
    • Add and use _table_cache to avoid catalog interactions for every batch
    • Simplify metadata, simply enumerating the namespace instead of noting the partition_columns

@fordN fordN force-pushed the ford/improvements branch 2 times, most recently from 4904525 to cd68def Compare October 2, 2025 16:32
@fordN fordN requested a review from incrypto32 October 2, 2025 16:45
@fordN fordN self-assigned this Oct 2, 2025
@fordN fordN force-pushed the ford/improvements branch from cd68def to 5179918 Compare October 23, 2025 23:20
@fordN fordN merged commit 61d1537 into main Oct 23, 2025
7 checks passed
@fordN fordN deleted the ford/improvements branch October 23, 2025 23:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@incrypto32 incrypto32 incrypto32 approved these changes

Assignees

@fordN fordN

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fordN @incrypto32