Feature/S3 to Postgres optional column order detection #67
Open
sleblanc23 wants to merge 3 commits intomainfrom
Open
Feature/S3 to Postgres optional column order detection #67sleblanc23 wants to merge 3 commits intomainfrom
sleblanc23 wants to merge 3 commits intomainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description & motivation:
The
s3_to_postgresands3_dir_to_postgrescallables currently rely on the S3 file(s) and the Postgres table to have matching column orders unless the order is specified in thecolumn_customizationconfig. This PR adds the option to set the column order by reading the header of the file in S3. It will be helpful for the upgrade of theload_heimdall_dag, currently the only user of this callable, to avoid errors due to column order.I'm not convinced that I decided on the ideal implementation of this idea, with the user specifying the column delimiter to use the feature. I considered making the argument a boolean and parsing the delimiter from the
optionsargument. That would be a simpler argument but would require some ugly regex, so I decided against it.PR Merge Priority:
Tests and QC done: