Skip to content

HDXDSYS-2287 Support dev to create plan for detection of identical versions of uploaded resources #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mcarans merged 16 commits into from
Aug 26, 2025
Merged

HDXDSYS-2287 Support dev to create plan for detection of identical versions of uploaded resources #91

mcarans merged 16 commits into from
Aug 26, 2025

Conversation

@mcarans
Copy link
Contributor

@mcarans mcarans commented Aug 18, 2025

No description provided.

@github-actions
Copy link

github-actions bot commented Aug 18, 2025 •
edited
Loading

Test Results

149 tests  +4   149 ✅ +4   1m 30s ⏱️ -1s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit d9ce86f. ± Comparison against base commit c05787a.

This pull request removes 1 and adds 5 tests. Note that renamed tests count towards both. 
tests.hdx.data.test_resource.TestResource ‑ test_check_url_filetoupload

tests.hdx.api.utilities.test_size_hash.TestSizeHash ‑ test_get_size_and_hash
tests.hdx.data.test_resource.TestResource ‑ test_check_types
tests.hdx.data.test_resource.TestResource ‑ test_correct_format
tests.hdx.data.test_resource.TestResource ‑ test_url_filetoupload
tests.hdx.data.test_update_dataset_resources.TestUpdateDatasetResourcesLogic ‑ test_dataset_update_resources_position

♻️ This comment has been updated with latest results.

@coveralls
Copy link

coveralls commented Aug 18, 2025 •
edited
Loading

Coverage Status

coverage: 97.046% (+0.05%) from 96.992%
when pulling d9ce86f on hashing
into c05787a on main.

mcarans added 9 commits August 19, 2025 14:10
@mcarans
Ensure filestore resources are checked when creating datasets
0ad9170 
Increase coverage
@mcarans
Split format correction into separate method with tests
01e9611 
Only set url_type to api when there's a url if it is not already set to upload (alternatively could check if url looks like a filestore one but I think there are some datasets set up linking to other datasets' resources)
@mcarans
Fix resource_view and showcase to check required fields on create pat…
16e3baa 
…h only in create_in_hdx
@mcarans
Fix resource_view and showcase to check required fields on create pat…
5928b58 
…h only in create_in_hdx
@mcarans
Add slogging for when we don't update filestore when hash and size same
2427ba4
@mcarans
Add warning for when we don't update filestore when hash and size same
ad0e559
@mcarans
Add warning for when we don't update filestore when hash and size same
c99ba7d
@mcarans
Updating resource view - CKAN returns None so must check fields befor…
93ec4f7 
…e update not after
@mcarans
Check that new or updated resources do not have both a url and a file…
ae336a7 
… to upload
baripembo
baripembo approved these changes Aug 22, 2025
View reviewed changes
Copy link
Member

@baripembo baripembo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this against two pipelines and they ran without issue

b-j-mills
b-j-mills approved these changes Aug 26, 2025
View reviewed changes
Copy link
Contributor

@b-j-mills b-j-mills left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's working well with two of my pipelines and correctly detecting that resources have not changed!

@mcarans mcarans merged commit b2e4310 into main Aug 26, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@baripembo baripembo baripembo approved these changes

@b-j-mills b-j-mills b-j-mills approved these changes

@maxmalynowsky maxmalynowsky Awaiting requested review from maxmalynowsky

@isatotun isatotun Awaiting requested review from isatotun

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mcarans @coveralls @baripembo @b-j-mills