Refactor: Improve error handling, testing, and documentation.#1
Open
Refactor: Improve error handling, testing, and documentation.#1
Conversation
This commit introduces several improvements to the codebase:
- **Bug Fix:**
- `defs/etl_fetch.py`: Corrected the `fetch_resources` function to use the `source` parameter instead of a hardcoded URL.
- **Testing:**
- Established a unit testing framework using `unittest`.
- Added tests for `defs/etl_fetch.py` (mocking `AsyncWebCrawler`).
- Added tests for `defs/etl_convert.py` (including HTML and PDF conversion, with programmatic PDF fixture creation using `reportlab`).
- `bamlTest.py` identified as a utility script; its purpose is now documented.
- **Error Handling & Logging:**
- Python scripts in `defs/` (etl_convert, etl_fetch, etl_query) now use Python's `logging` module for errors, warnings, and info messages, replacing `print()`.
- `defs/etl_query.py`: Enhanced CSV parsing to specifically catch `polars.exceptions.ShapeError` when `truncate_ragged_lines=False`, logging an informative error and re-raising.
- Shell scripts (`scripts/loadDirToTriplestore.sh`, `scripts/loadSitemapToTriplestore.sh`):
- Added `set -e` for safer execution.
- Added checks for `jsonld` and `curl` command existence.
- Implemented robust error checking for `jsonld` and `curl` command failures during processing, with errors reported to stderr.
- `loadSitemapToTriplestore.sh` now includes a summary of processed URLs and failures.
- **Documentation (README.md):**
- Added a new "Running Tests" section with instructions.
- Documented the dependency on the `jsonld` command-line tool, including installation instructions (e.g., via `npm`).
- Clarified the purpose of `bamlTest.py`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit introduces several improvements to the codebase:
Bug Fix:
defs/etl_fetch.py: Corrected thefetch_resourcesfunction to use thesourceparameter instead of a hardcoded URL.Testing:
unittest.defs/etl_fetch.py(mockingAsyncWebCrawler).defs/etl_convert.py(including HTML and PDF conversion, with programmatic PDF fixture creation usingreportlab).bamlTest.pyidentified as a utility script; its purpose is now documented.Error Handling & Logging:
defs/(etl_convert, etl_fetch, etl_query) now use Python'sloggingmodule for errors, warnings, and info messages, replacingprint().defs/etl_query.py: Enhanced CSV parsing to specifically catchpolars.exceptions.ShapeErrorwhentruncate_ragged_lines=False, logging an informative error and re-raising.scripts/loadDirToTriplestore.sh,scripts/loadSitemapToTriplestore.sh):set -efor safer execution.jsonldandcurlcommand existence. - Implemented robust error checking forjsonldandcurlcommand failures during processing, with errors reported to stderr. -loadSitemapToTriplestore.shnow includes a summary of processed URLs and failures.Documentation (README.md):
jsonldcommand-line tool, including installation instructions (e.g., vianpm).bamlTest.py.