Skip to content

Switch LDLite to using the new server side transform#66

Merged
brycekbargar merged 12 commits intolibrary-data-platform:release-v4.0.0from
Five-Colleges-Incorporated:parallel-transformation
Mar 4, 2026
Merged

Switch LDLite to using the new server side transform#66
brycekbargar merged 12 commits intolibrary-data-platform:release-v4.0.0from
Five-Colleges-Incorporated:parallel-transformation

Conversation

@brycekbargar
Copy link
Collaborator

@brycekbargar brycekbargar commented Mar 4, 2026

This switches LDLite to using the new transform. The legacy transform is still available by using a parameter passed to the query method. After making the switch and testing with both the unit tests and against Five Colleges production instance a number of regressions were found and fixed. While testing against the production instance the load_history table proved to be a thorn in my side and the v1 was added in lieu of actually using a migration framework for it.

  • The catalog table was never implemented
  • The array index column was misnamed
  • Booleans weren't created as boolean columns
  • The column named "value" used during expansion conflicted with underlying column names
  • The columns in the output tables were ordered in a random way
  • Fields that started with caps would get an underscore at the front of their column name

Indexing for the new transformation will be implemented in the next PR. In testing, this implementation runs out of memory for large tables with large objects. I've experimented with fixes and there will be a follow up performance PR to address this issue.

@brycekbargar brycekbargar merged commit 493b5fc into library-data-platform:release-v4.0.0 Mar 4, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant