Lutaml integration#175
Draft
andrew2net wants to merge 52 commits into
Draft
Conversation
![hound[bot]](https://avatars.githubusercontent.com/in/3598?s=60&v=4){kind=small}
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
There was a problem hiding this comment.
Some files could not be reviewed due to errors:
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unre...
Error: unrecognized cop Performance/CaseWhenSplat found in .rubocop.yml, unrecognized cop Performance/Count found in .rubocop.yml, unrecognized cop Performance/Detect found in .rubocop.yml, unrecognized cop Performance/FlatMap found in .rubocop.yml, unrecognized cop Performance/ReverseEach found in .rubocop.yml, unrecognized cop Performance/Size found in .rubocop.yml, unrecognized cop Performance/StringReplacement found in .rubocop.yml
The openssl 3.3.0 causes OpenSSL::SSL::SSLError error Some small bugfixes have also been done
…; update scraper to use except method for params; modify bibliography_spec to enable tests and adjust expectations; fix scraper_spec to enable isoref test; update VCRs
…iso dependency version to 1.15.8
…djust VCR re_record_interval to 7 days
…pubid Add handling for Pubid::Core::Identifier objects in HitCollection#find and create_pubid methods. This allows the code to work with Pubid::Core::Identifier instances directly and use Relaton::Index binary search for better performance when possible. Using ID keys for conrol of index ID scructure. Cache parsed index data in tests to avoid re-parsing the zip/YAML every test, improving test performance.
remove dates outside the cut-off dates and add corresponding tests
…e gem dependency for relaton-core
…preservation issue #178
* update Gemfile to use GH version of relaton-bib & lutaml-model 0.8.0 * fix: update document identifier to use pubid for consistency in file writing * feat: add model ItemData to Bibdata and Bibitem classes; update schema version mapping in Ext class * Update VCRs * refactor: update ItemBase class to inherit from Lutaml::Model::Serializable and remove unnecessary attributes * chore: remove stale commented-out attribute mutation in Contributor Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add key_value mapping for attributes in Ext class * chore: remove unused create_relation method from ItemData class * fix Gemfile --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…for 2.1.0 release
Replace the ICS-page scraper-driven DataFetcher (and its persistent queue/threaded model) with a streaming consumer of `iso_deliverables_metadata.jsonl` and `iso_technical_committees.jsonl`, plus a new DataParser that converts one record into an Iso::ItemData. Adds `iso-open-data` (incremental, gated on upstream Last-Modified) and `iso-open-data-all` (full refresh) source modes; Scraper is retained only as a fallback for `Bibliography.get` lookups missing from the curated index. Refreshes fixtures, cassettes, and docs accordingly.
Build a publicationDate index alongside the reference index so DataParser can stamp each emitted relation's bibitem with a `published` date when the related document is itself present in the Open Data feed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Align spec fixtures and assertions with the Open Data ingest output: relation types use obsoletedBy/updatedBy, abstracts are flattened, place uses <city>, and the corrected-date case is skipped since Open Data does not expose it.
Open Data emits stub records with a `Withdrawn` reference prefix for abandoned projects (publicationDate: null, stage *.98). The previous "Withdrawn" → "ISO" rewrite produced strings like "ISO 1701/Add 1" that pubid-iso can't parse, leaving String docids in the index and crashing `index.save` whenever a Pubid entry sorted ahead of them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picks up the `<date type="published">` elements that Open Data ingest now attaches to related bibitems, and re-records the HTTP cassettes that drifted in the meantime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ISO's Azure Blob endpoint intermittently returns HTTP 403 AuthorizationFailure, causing the scheduled crawler to fail. Retry up to 4 times (30/60/120/240s) before raising.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.