Skip to content

fix(low-code): handle all ScannerError exceptions in ConfigComponentsResolver #854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
devin-ai-integration wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(low-code): handle all ScannerError exceptions in ConfigComponentsResolver #854

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d99a964
fix(low-code): handle all ScannerError exceptions in ConfigComponents…
devin-ai-integration[bot] Nov 26, 2025
22e9d85
style: apply ruff formatting to test file
devin-ai-integration[bot] Nov 26, 2025
6c4ca2c
fix(low-code): handle tab character ScannerError specifically
devin-ai-integration[bot] Nov 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion airbyte_cdk/sources/declarative/resolvers/config_components_resolver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,8 +204,10 @@ def _parse_yaml_if_possible(value: Any) -> Any:
return yaml.safe_load(value)
except ParserError: # "{{ record[0] in ['cohortActiveUsers'] }}" # not valid YAML
return value
except ScannerError as e: # "%Y-%m-%d' # not valid yaml
except ScannerError as e: # "%Y-%m-%d" or strings with tabs - not valid YAML
if "expected alphabetic or numeric character, but found '%'" in str(e):
return value
if "found character '\\t' that cannot start any token" in str(e):
return value
raise e
Comment on lines +207 to 212
Copy link

Copilot AI Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pattern of checking specific error message strings and returning value for each case suggests a fragile approach. According to the PR description's root cause analysis, the intent is to catch ALL ScannerError exceptions since they indicate values that cannot be parsed as YAML. The current implementation still only catches two specific cases. Consider simplifying to except ScannerError: return value which would be more maintainable and align with the PR's stated goal of 'catch all ScannerError exceptions'.

Suggested change
except ScannerError as e: # "%Y-%m-%d" or strings with tabs - not valid YAML
if "expected alphabetic or numeric character, but found '%'" in str(e):
return value
if "found character '\\t' that cannot start any token" in str(e):
return value
raise e
except ScannerError: # any ScannerError means not valid YAML
return value

Copilot uses AI. Check for mistakes.
return value
23 changes: 22 additions & 1 deletion unit_tests/sources/declarative/resolvers/test_config_components_resolver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,20 @@ def to_configured_catalog(
}
)

# Manifest with component definition with value containing tab characters
# which would cause YAML ScannerError (tabs cannot start tokens in YAML)
_MANIFEST_WITH_TAB_SCANNER_ERROR = deepcopy(_MANIFEST)
_MANIFEST_WITH_TAB_SCANNER_ERROR["dynamic_streams"][0]["components_resolver"][
"components_mapping"
].append(
{
"type": "ComponentMappingDefinition",
"create_or_update": True,
"field_path": ["retriever", "requester", "$parameters", "custom_query"],
"value": "SELECT\n\tcampaign.name,\n\tcampaign.id\nFROM campaign", # Contains tab characters
}
)


@pytest.mark.parametrize(
"manifest, config, expected_exception, expected_stream_names",
Expand All @@ -173,8 +187,15 @@ def to_configured_catalog(
),
(_MANIFEST_WITH_STREAM_CONFIGS_LIST, _CONFIG, None, ["item_1", "item_2", "default_item"]),
(_MANIFEST_WITH_SCANNER_ERROR, _CONFIG, None, ["item_1", "item_2", "default_item"]),
(_MANIFEST_WITH_TAB_SCANNER_ERROR, _CONFIG, None, ["item_1", "item_2", "default_item"]),
],
ids=[
"no_duplicates",
"duplicates",
"stream_configs_list",
"scanner_error",
"tab_scanner_error",
],
ids=["no_duplicates", "duplicates", "stream_configs_list", "scanner_error"],
)
def test_dynamic_streams_read_with_config_components_resolver(
manifest, config, expected_exception, expected_stream_names
Expand Down
Loading