Releases: c-hydro/d3tools
Releases · c-hydro/d3tools
v2.4.0-alpha.2
Contributors
ltrotter
Assets 3
v2.4.0-alpha.1
✨ New Features
4766039- template-manager: add TemplateManager class for spatial template management (commit by @ltrotter)55a8145- dataset: integrate TemplateManager for template handling and add backward compatibility properties (commit by @ltrotter)1da0b1f- template-manager: enhance TemplateManager with validation, caching, and error handling features (commit by @ltrotter)cc3963f- dataset: optimize get_last_date with exponential backoff and add get_any_date method for faster data retrieval (commit by @ltrotter)c9d2d4e- data: Introduce DatasetLogManager and DatasetThumbnailManager (commit by @ltrotter)ce59860- config: refactor configuration parsing and integrate dataset managers (commit by @ltrotter)efca146- format: introduce format mixins for various data types (commit by @ltrotter)ccd2992- string_rendering: add string rendering utilities for parser placeholders (commit by @ltrotter)7ff56b6- config_resolution: add configuration resolution utilities with legacy compatibility (commit by @ltrotter)9051829- key_pattern: add key-pattern parsing and rendering utilities (commit by @ltrotter)3d4c8bb- parsing_pipeline: implement configuration parsing stages and update options parsing (commit by @ltrotter)6f12248- workflow: introduce workflow section handling with alias resolution and collection (commit by @ltrotter)e31367e- workflow: enhance workflow parsing with build and strict import flags (commit by @ltrotter)61b472d- errors: introduce WorkflowEngineImportError for handling workflow engine import failures; update workflow_section_from_config to raise this error (commit by @ltrotter)cc43081- modules: restructure imports and legacy module path handling in data and timestepping packages (commit by @ltrotter)9b567f2- workflow: implement run method for ordered execution of workflow sections (commit by @ltrotter)c87031b- config: enhance placeholder resolution in config_resolution and string_rendering modules; add comprehensive tests for set_env and set_dataset functions (commit by @ltrotter)e5adf55- datasets: add _get_recreate_kwargs method to Dataset subclasses for enhanced dataset recreation (commit by @ltrotter)f00635f- keyparser: add get_matching_keys method to filter keys by parser pattern (commit by @ltrotter)570fb60- datasets: implement _creation_kwargs for Dataset subclasses to store initialization parameters (commit by @ltrotter)0382f9a- parse: update parse utils make_hashable and transform back to preserve subclasses (commit by @ltrotter)bace296- logging: add workflow logging capabilities (commit by @ltrotter)0dd1211- config: allow to specify engine in WorkflowSection configuration (commit by @ltrotter)34f8533- parse: allow explicit definition of engine when creating WorkflowSections (commit by @ltrotter)
🐛 Bug Fixes
aebadd9- workflow: replace conditional checks on attributes with match-case on engine for process attribution (commit by @ltrotter)3fdbb5e- datasets: modify new method to accept positional arguments for key_pattern detection (commit by @ltrotter)97b79f0- raster: update validate_data method to handle np.ndarray input and return type (commit by @ltrotter)3a08712- Options: establish hyerarchy between Options and WorkflowDefinition (commit by @ltrotter)
♻️ Refactors
88d87a6- dataset: optimize template metadata extraction to avoid memory issues (commit by @ltrotter)ce4e53f- dataset: refactor Dataset initialization and add comprehensive tests for helper methods (commit by @ltrotter)bc29613- data: restructure dataset management by introducing RasterMixin (commit by @ltrotter)5eb6127- data: finalise format mixins for various data types (commit by @ltrotter)ce9fbb6- data: create DataCatalogue class and delegate all methods to index available data (commit by @ltrotter)4201ca2- data: complete DataCatalogue class (commit by @ltrotter)c499732- cases: rewrite withcases dectorator and move to cases module (commit by @ltrotter)0ba409b- data: reorder methods into Dataset (commit by @ltrotter)e0e70b5- data: migrate raster-related functions to io_utils_raster module (commit by @ltrotter)5098c92- change folder structure of data and timestepping for clarity (commit by @ltrotter)31a14bc- parse: move string substitution functions (commit by @ltrotter)8b3501a- parse: move structure utils functions (commit by @ltrotter)ad99cdf- parse: move special substitution functions (commit by @ltrotter)6f13794- time_utils: move time parsing functions and reorganise time_utils (commit by @ltrotter)dabbefe- parsing: restructure parsing utilities and remove legacy modules (commit by @ltrotter)b50562a- key_pattern: move extract_date_and_tags and set tests for KeyFactor (commit by @ltrotter)346c215- key_pattern: split up matching logic in _extract_date_and_tags into subfuctions, add caching (commit by @ltrotter)e6b7bb2- parsing: rename KeyPattern to KeyParser (commit by @ltrotter)929b1fa- data_catalogue: replace legacy key extraction logic with KeyParser for improved clarity and maintainability (commit by @ltrotter)fc24bd8- parsing: enhance KeyParser with time normalization and resolution methods; update Dataset to utilize KeyParser for key generation (commit by @ltrotter)- [
dfc447e](dfc447ebad99...
Contributors
ltrotter
Assets 3
v2.4.0-alpha
Contributors
ltrotter
Assets 3
D3TOOLS v2.3.0
✨ New Features
3bf06c3- data: add profile_name to S3 (commit by @michelisabellon)4f71cc8- data: add profile_name to S3 (commit by @michelisabellon)54700c2- data: add profile_name to S3 (commit by @michelisabellon)01669d7- io: add metadata to json ds (commit by @ltrotter)d145b26- parse: allow use of Julian days in dataset string fields (commit by @michelisabellon)
🐛 Bug Fixes
201599d- data: ensure today is accepted as last date from dataset (commit by @ltrotter)aed693f- data: ensure data is straight before setting it to template (commit by @ltrotter)936410b- data: ensure memory datasets don't use templates (commit by @ltrotter)bb17c6b- io: ensure time columns in gpd are converted to string before saving json (commit by @ltrotter)e9c5d9d- data: ensure tags are read correctly if time_signature is end+1 (commit by @ltrotter)311374f- parse: ensure string dates are not parsed in substitute_sting withoug format (commit by @ltrotter)adcfb58- io: ensure crs is maintained when reading json files (commit by @ltrotter)19d56e5- io: ensure we are not writing metadata to json dictionaries (commit by @ltrotter)dc2f3e9- data: improve day handling in datasets (commit by @michelisabellon)8bde062- dataset: use last_ts instead of first_date for template making (commit by @ltrotter)5d0d52a- timstepping: enure extend doens't try to return a timestep (commit by @ltrotter)
🔧 Chores
Contributors
ltrotter and michelisabellon
Assets 3
D3TOOLS v2.2.0
✨ New Features
4b70515- data: add support for parquet files (commit by @ltrotter)78fdc83- thumbs: allow making thumbnails also for shapefiles (commit by @ltrotter)ef9baed- options: enable using environment variables to populate options (commit by @ltrotter)06960e3- data: allow to save data as_is to avoid calling template (commit by @ltrotter)
🐛 Bug Fixes
843d74c- timestepping: ensore TimeRange.get_timesteps works with int freq (commit by @ltrotter)661dc03- thumbnails: fix colormaps, hopefully for real this time (commit by @ltrotter)2a42df2- data: make Datasets.qc_checks work with xr.Dataset input (commit by @ltrotter)72bdb19- timestep: make finding timesteps easier (commit by @ltrotter)5a673cc- data: ensure metadata is set correctly and name is string (commit by @ltrotter)a46f45d- data: use EPSG:4326 if you cannot find the crs for the template (commit by @ltrotter)457ae62- io: ensure nodata value is set when writing chunked data (commit by @ltrotter)
♻️ Refactors
6827ecf- data: ensure the logging string is created only if requested (commit by @ltrotter)95f8351- data: efficient saving of chunked data (commit by @ltrotter)
🔧 Chores
Contributors
ltrotter
Assets 3
D3TOOLS v2.1.0
✨ New Features
68afd27- cases: create CaseManager class and Case class (commit by @ltrotter)f33b819- cases: ensure options are saved when a layer is added (commit by @ltrotter)ea35786- case: save alltags together with tags for case (commit by @ltrotter)686810c- parse: allow # and - for substitutions in strings (commit by @ltrotter)5b1e2cb- parse: changes to allow # and - in parsers (commit by @ltrotter)6d21490- data: add arg agg to get_start to get start of aggregation (commit by @ltrotter)dd545d4- exit: add function to remove path at exit (commit by @ltrotter)b9396eb- timestepping: handle aggregation from hourly data (commit by @michelisabellon)33deb19- io_utils: handle no_data values -9999 for np.float (commit by @michelisabellon)418777c- data: add search limit to last_ts search (commit by @ltrotter)6bc6ec0- cases: allow merging options in cases, improve iterator (commit by @ltrotter)cbae227- case: allow duplicate options (commit by @ltrotter)d68d96e- cases: add tag property to case_manager (commit by @ltrotter)ee51234- thumbnail: add all labels and colors to legends (commit by @michelisabellon)3dee2e2- io_utils: remove row_names from csv output (commit by @michelisabellon)d3d1ede- data: add handling of S3 endpoint and region (commit by @michelisabellon)83d4bb9- data: add handling of S3 endpoint and region (commit by @michelisabellon)b5cf45f- options: allow to split options between multiple json files (commit by @ltrotter)ae98262- parser: allow '/' in file_version tag (commit by @ltrotter)10ad9f7- dataset: make checking for available data more efficient (commit by @ltrotter)
🐛 Bug Fixes
affcb7d- data: fixes to ensure no extra timesteps are returned with get_timesteps (commit by @ltrotter)8a38fbc- patches: minor patches to parser and remote dataset (commit by @ltrotter)da0d40a- data: handle timestep and time_agg together with setters (commit by @ltrotter)4eea216- cases: fix setting case_ids when n == 0 (commit by @ltrotter)010d48e- remote: fix cleanup function to new rm_at_exit (commit by @ltrotter)3a70aa0- exit: correct code on rm_at_exit (commit by @ltrotter)6ce3cac- parse: ensure points are alloewd in dataset parsing from options (commit by @ltrotter)a8ddc48- io_utils: separate set_nan (writing) from reset_nan (reading) (commit by @ltrotter)96190a7- io: remove = from value_checks to set type (commit by @ltrotter)32a122a- data: rm now from kwargs when calling get_last_date twice (commit by @ltrotter)5305fb4- case_manager: reset iterate_subtree as it was (commit by @ltrotter)e1a84be- case_manager: rename layer to depth in iterate_tree (commit by @ltrotter)d6222a2- data: ensure parents are maintained when updating a ds (commit by @ltrotter)e899031- data: enure parents are checked if availabel at check_data (commit by @ltrotter)cd072bc- cases: patch typos in case_manager (commit by @ltrotter)ad4834f- thumbnails: fix error in visualisation when thumbnail has no nan data (commit by @ltrotter)e909de0- data: ensure data type (and fillvalue) is set correctly if template and data diverge (commit by @ltrotter)6f4bfca- parse: correct error in extract_date_and_tags in case of multiple tags (commit by @ltrotter)0a4c058- data: do not write data in mem_dataset when creating from parents (commit by @ltrotter)ae22808- data: make template not write _FillValue in case different variables have different data types (commit by @ltrotter)481e47e- data: make template use the data fill value if available (commit by @ltrotter)efc7753- timestepping: fix aggregation from hourly data (commit by @michelisabellon)87db32d- parse: ensure parsing works as expected (file_version search) (commit by @ltrotter)02daf2d- data: changes to data to accomodate hcwi (commit by @ltrotter)59caa65- cases: ensore previous cases are modified when merging (commit by @ltrotter)e62bb3c- data: ensure check_data works with file_version (commit by @ltrotter)7e413b2- parse: ensure curly-braket tags are parsed correctly when left as are (commit by @ltrotter)a392c8c- io: ensure nan type is compatible with data type before saving (commit by @ltrotter)a8e7cf0- timeperiod: return 0 if length of time period is negative (commit by @ltrotter)420d554- timewindow: ensure timewindow of hours unit is applied correctly (commit by @ltrotter)bbce89e- data: ensure dataset is created in the same format as data is saved and loaded (commit by @ltrotter)9c3ca53- cases: hack to ensure we can add empty layers to case_tree (commit by @ltrotter)fa141a6- data: ensure template data is closed so that tmp dir can be deleted (commit by @ltrotter)890837b- data: fix GET method for OVHS3Dataset (commit by @michelisabellon)- [
85fb640](85fb...
Contributors
ltrotter and michelisabellon
Assets 3
d3tools v2.0.1
✨ New Features
3f26190- timestepping: add functionality to get unit of time from ts_per_year (commit by @ltrotter)f9bf561- timestepping: add fucntionality to check if units are multiples of each other (commit by @ltrotter)0aca813- timestepping: add functionality to create TimePeriod from Any (commit by @ltrotter)edef773- data: add method to get timesteps in TimeRange (commit by @ltrotter)2445d97- timestepping: add method to extend time_period (commit by @ltrotter)f00c503- time: add aggregation information to timesteps (commit by @ltrotter)c1bd7de- timestep: add factory to typesteo from unit (commit by @ltrotter)2c852a7- timestepping: make agg_window also a class property via the metaclass (commit by @ltrotter)fd67bac- data: edit dataset to explicitly set timestp and aggregation (commit by @ltrotter)e21e559- timestepping: add TimeWindow class (commit by @ltrotter)614fd54- aggwindow: add summation and subtration between windows (commit by @ltrotter)f942b06- timewindow: add method to determin if windows are multiple of each other (commit by @ltrotter)
🐛 Bug Fixes
973d807- data: check typo on get_time_signature with timestep (commit by @ltrotter)3b03df2- data: pass over timestep when updating dataset (commit by @ltrotter)97d323c- timeagg: reconfigure time steps with aggregation to make sure the classes are not modified (commit by @ltrotter)0378861- timestep: fix typos in typestep definition in data (commit by @ltrotter)24f9bb7- data: improve management of timeagg and timestep in dataset (commit by @ltrotter)
Contributors
ltrotter
Assets 3
v2.0.1-alpha
✨ New Features
f00c503- time: add aggregation information to timesteps (commit by @ltrotter)c1bd7de- timestep: add factory to typesteo from unit (commit by @ltrotter)2c852a7- timestepping: make agg_window also a class property via the metaclass (commit by @ltrotter)fd67bac- data: edit dataset to explicitly set timestp and aggregation (commit by @ltrotter)e21e559- timestepping: add TimeWindow class (commit by @ltrotter)614fd54- aggwindow: add summation and subtration between windows (commit by @ltrotter)f942b06- timewindow: add method to determin if windows are multiple of each other (commit by @ltrotter)
🐛 Bug Fixes
973d807- data: check typo on get_time_signature with timestep (commit by @ltrotter)3b03df2- data: pass over timestep when updating dataset (commit by @ltrotter)97d323c- timeagg: reconfigure time steps with aggregation to make sure the classes are not modified (commit by @ltrotter)0378861- timestep: fix typos in typestep definition in data (commit by @ltrotter)24f9bb7- data: improve management of timeagg and timestep in dataset (commit by @ltrotter)
Contributors
ltrotter
Assets 3
d3tools v2.0.0
✨ New Features
6711e61- errors: add clear error for gdal import failure (commit by @ltrotter)959161e- parse: add function to format dictionary (useful for error messages) (commit by @ltrotter)8fcd359- data: allow selection of nan value for dataset (commit by @ltrotter)
🐛 Bug Fixes
35c5ded- setup: fix names of extras in pyproject file (commit by @ltrotter)bdb9189- data: fix metadata when saving files, order of priority (commit by @ltrotter)1124dbb- io: patch opening of jsons containing lists (commit by @ltrotter)1b39d66- avoid circular imports on parse and options (commit by @ltrotter)0f002c1- imports: only import dependencies when needed for extras (commit by @ltrotter)e8c6446- io: restructure data type checks to allow flexible imports (commit by @ltrotter)4e674bd- data: ensure prefix returns path also when single file exists (commit by @ltrotter)9ea645d- data: set return of time_signature to 0 or 1 if field is not in key_pattern (commit by @ltrotter)ea0bf54- data: ensure find_times works with 'end+1' time_signature and daily ts (commit by @ltrotter)bd424f5- setup: redirect pyproj to correct folder and fix global init file (commit by @ltrotter)481f220- correct circular import in spatial (commit by @ltrotter)1b0fa70- io: ensure gdp type is checked before pd as it is a subclass (commit by @ltrotter)9d73d21- data: ensure available_keys are returned also without time (commit by @ltrotter)b5d35bf- io: patch opening of jsons containing lists (commit by @ltrotter)34ab054- errors: correct typo in init for import of GDAL_ImportError (commit by @ltrotter)c48f9f1- spatial: correct a few typos in spatial from carryover from door (commit by @ltrotter)cd2ea61- thumbnails: correct imports of functions from color (commit by @ltrotter)4ba4515- data: ensure resetting the nan value does not change the data type (for CDI) (commit by @ltrotter)ac29165- spatial: ensure the correct crs is set from rasterio (commit by @ltrotter)88fdd61- spatial: fix typo in crs getter (commit by @ltrotter)
♻️ Refactors
e4e7428- major changes to make d3tools installable (commit by @ltrotter)10e5daa- space: change spatial to make tools installable (commit by @ltrotter)582e57c- setup: move package folder into src for setuptools discovery (commit by @ltrotter)6f9798a- space: add import of space_utils directly from spatial (commit by @ltrotter)
Contributors
ltrotter
Assets 3
d3tools v1.3.10
Contributors
ltrotter