Skip to content

Miscellaneous improvements for 2026-W03 #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
khaeru wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Miscellaneous improvements for 2026-W03 #58

khaeru wants to merge 11 commits into from

Conversation

@khaeru
Copy link
Contributor

@khaeru khaeru commented Jan 15, 2026 •
edited
Loading

  • Add new CLI command tdc check-record.
    This command retrieves the metadata for a given record from the TDC CKAN API and checks/prints some properties of the record. For example:

    $ tdc check-record registered-vehicles-in-tanzania
- https://portal.transport-data.org/@ministry of works and transport of tanzania/registered-vehicles-in-tanzania
- Title: 'Registered Vehicles in Tanzania'
- Category: tdc_formatted

- Number of files by extension: 1 .csv, 2 .pdf, 1 .xlsx
- Number of data files: 2
- Number of possible SDMX-CSV files: 1

Criteria for a TDC Formatted record:
- At least one file in CSV format: True
- Correct category assigned: True
- CSV file(s) are in SDMX-CSV format (not implemented yet): True
- Overall: YES

Criteria for a TDC Harmonized record—all of the above, plus:
- Correct category assigned: False
- Overall: NO
    • Retrieve and cache the data file(s).
    • Check the contents of data file(s).

  • Filter/ignore the warning due to Replace pkg_resources with importlib.metadata ckan/ckanapi#218. The upstream PR was merged, but the package has not been released since 2024, so the warning still appears.

How to review

  • Try running tdc check-record for some known record IDs.
    • Report whether the outputs appear correct.

PR checklist

  • Checks all ✅
  • Update documentation
  • Update doc/whatsnew.rst

@khaeru khaeru self-assigned this Jan 15, 2026
@khaeru khaeru added the enh New feature or request label Jan 15, 2026
khaeru added 10 commits January 26, 2026 15:07
@khaeru
Work around ckan/ckanapi#218
f8652b6
@khaeru
Improve .util.ckan.Package
8992ebf 
- Convert "resources" collection to instances of Resource.
- Add portal_url() method.
- Add type hints for known members/attributes.
@khaeru
Improve .util.ckan.Resource
bfb56aa 
- Add .fetch() method.
- Add type hints for known attributes.
@khaeru
Add tdc check-record CLI command
d47773b
@khaeru
Add cli_modules() plugin hook
9b3be95 
- Use in existing modules.
@khaeru
Add transport_data.hook as a top-level item
da94051
@khaeru
Add cli_modules() hooks to existing modules
d8d0f3b
@khaeru
Add CLI commands from cli_modules() hooks
1ad31a2 
- Reduce MODULES_WITH_CLI to internal/non-provider modules.
@khaeru
Separate check- CLI commands into distinct modules
700d779
@khaeru
Add .util.sdmx.structure_from_csv()
6909bf3
@khaeru
Adjust oica.update_registry() per CI failures
a20fcb6 
- Use sub-paths from registry in .is_available() call.
- Handle ConnectionError/HTTPSConnectionPool max retries exceeded.
  This may be caused by repeated queries to incorrect URLs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@khaeru khaeru

Labels

enh New feature or request

Projects

CCG Agile 2025–2026
Status: No status

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@khaeru