Skip to content

Feature/inLineSource: adding another source type#131

Draft
gnguyen87 wants to merge 15 commits intomainfrom
feature/inLineSource
Draft

Feature/inLineSource: adding another source type#131
gnguyen87 wants to merge 15 commits intomainfrom
feature/inLineSource

Conversation

@gnguyen87
Copy link

@gnguyen87 gnguyen87 commented Oct 9, 2024

Description & motivation

From this Monday item:

Consider adding another source type, possibly called inline, where the data is specified in the earthmover YAML:

sources:
  inlineSource:
    orientation: rows # or columns
    data:
      - key1: value1a
        key2: value2a
        key3: value3a
      - key1: value1b
        key2: value2b
        key3: value3b
      - key1: value1c
        key2: value2c
        key3: value3c
    # or, if orientation=columns
    data:
      key1: [ "value1a", "value1b", "value1c" ]
      key2: [ "value2a", "value2b", "value2c" ]
      key3: [ "value3a", "value3b", "value3c" ]

This PR supports this feature by adding a new inLineSource class with minor refactoring to other helper classes.

PR Merge Priority:

  • Low
  • Medium
  • High

Changes to existing files:

  • source.py : Add an inLineSource class
  • runs_file.py: Import inLineSource class to take care of file-hashing
  • column.py: Cast dict vals to str type in a lambdato appropriately join row values together
  • earthmover/tests: add an inLineSource to the test suite

Tests and QC done:

  • Tested locally via the test suite as well as earthmover_edfi_bundles

)
raise

class inLineSource(Source):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider InlineSource to fit convention of other nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants