Release v3 proper#1298

Merged

kba merged 270 commits intomasterfrom

Jan 9, 2025

This pull request is big! We're only showing the most recent 250 commits

Commits on Jul 6, 2024

adapt to new Processor init (override metadata/version/executable name)
Robert Sachunsky
committed
tests: adapt to new Processor init (override metadata/version/executable name)
Robert Sachunsky
committed

Commits on Jul 29, 2024

generate_processor_help: include process_workspace docstring, too
Robert Sachunsky
committed

Commits on Aug 8, 2024

get_processor: also run setup if instance_caching
bertsky
authored

Commits on Aug 12, 2024

ocrd-tool CLI: pass class in context
Robert Sachunsky
committed
use more specific exception if parameters are invalid

bertsky
and
kba
authored
run_processor w/ mem_usage: pass as args tuple

bertsky
and
kba
authored
Processor.process_workspace: add fileGrp assertions
Robert Sachunsky
committed
process_page_pcgts: add (variadic) type checks

bertsky
and
kba
authored
run_processor: fix typo
Robert Sachunsky
committed
Processor init: deprecate passing workspace
Robert Sachunsky
committed
docs: fix relative VERSION path
Robert Sachunsky
committed
docs: do/not exclude tests/src
Robert Sachunsky
committed
docs: add ocrd_network module
Robert Sachunsky
committed
docs:regenerated rST
Robert Sachunsky
committed

Commits on Aug 13, 2024

test_mets_server: fix arg vs kwarg
Robert Sachunsky
committed
mets_server: ClientSideOcrdMets needs OcrdMets-like kwargs (without deprecation)
Robert Sachunsky
committed
Processor/CLI decorator: :fire: separate kwargs and constructor…
Robert Sachunsky
committed
Processor / ocrd-tool.json: :fire: fileGrp cardinality checks…
Robert Sachunsky
committed
test_processor: adapt to Processor init changes
Robert Sachunsky
committed
adapt to ocrd-tool.json cardinality changes
Robert Sachunsky
committed
use up-to-date kwargs (avoiding old deprecations)
Robert Sachunsky
committed
hide/test expected deprecation warnings
Robert Sachunsky
committed
improve output in case of assertion failures
Robert Sachunsky
committed

Commits on Aug 15, 2024

Commits on Aug 16, 2024

Update CHANGELOG.md

kba
and
bertsky
authored
ocrd: more convenience imports
Robert Sachunsky
committed
ocrd.cli: more fix module import order, export help cmd
Robert Sachunsky
committed
fix imports
Robert Sachunsky
committed
fix type assertion
Robert Sachunsky
committed
ocrd_utils: forgot to export scale_coordinates at toplvl
Robert Sachunsky
committed
fix 9cfd70cffcc
Robert Sachunsky
committed
fix 9cfd70cffcc (revert to wrong import order to avoid circle)
Robert Sachunsky
committed
s,PcGtsType,OcrdPage,
kba
committed

Commits on Aug 20, 2024

Processor.zip_input_files: warning instead of exception for missing input files
Robert Sachunsky
committed
Processor.zip_input_files: introduce NonUniqueInputFile exception
Robert Sachunsky
committed
Processor.process_workspace: zip_input_files w/o require_first
Robert Sachunsky
committed
Merge remote-tracking branch 'origin/download-files-config-var' into new-processor-api-input-file-errors
Robert Sachunsky
committed
Processor.zip_input_files: introduce MissingInputFile exception and config.OCRD_MISSING_INPUT
Robert Sachunsky
committed
OcrdPage: clearer docstring

kba
and
bertsky
authored
jsonschema: switch from draft6 to draft2019-09
kba
committed
require jsonschema>4 for draft 2019-09
kba
committed
OcrdToolValidator: set defaults, handle deprecated
kba
committed
processor.base: validate/setdefault ocrd-tool.json on first access
kba
committed
update spec and ocrd_tool.schema.yml
kba
committed
processor parameter decorator: no '{}' default (unnecessary)
Robert Sachunsky
committed
Processor: add error handling…
Robert Sachunsky
committed

Commits on Aug 21, 2024

ocrd_utils.config: add variables to module docstring
Robert Sachunsky
committed
improve docstrings, re-generate docs
Robert Sachunsky
committed
Processor.zip_input_files: more verbose log msg

bertsky
and
kba
authored
test_processor: test for specific exception

bertsky
and
kba
authored
test_processor: fix missing import
Robert Sachunsky
committed
Merge pull request #12 from bertsky/new-processor-api-input-file-errors
bertsky
authored
Merge branch 'new-processor-api' into processor-logger
bertsky
authored
Merge pull request #10 from OCR-D/processor-logger
bertsky
authored
Merge pull request #11 from OCR-D/ocrd-page-with-etree
bertsky
authored
Merge pull request #13 from OCR-D/validate-ocrd-tool-runtime
bertsky
authored
OcrdPage: fix typeing typo
Robert Sachunsky
committed
dummy_processor: fix typos from logging
Robert Sachunsky
committed
tests report.is_valid: improve output on failure
Robert Sachunsky
committed
JsonValidator: fix deprecation warning (by actually checking instance)
Robert Sachunsky
committed
predefine union types OcrdFileType and OcrdPageType
Robert Sachunsky
committed
processor CLI --debug: set all to ABORT (not just MISSING_OUTPUT)
Robert Sachunsky
committed
:memo: changelog
Robert Sachunsky
committed

Commits on Aug 22, 2024

:package: v3.0.0a2
kba
committed
remove make *-workaround, we will not do that for v3+
kba
committed
Processor.parameter: only validate when set…
Robert Sachunsky
committed
get_processor: ensure passing non-empty parameter, rely on `_setup` to call `setup`
Robert Sachunsky
committed
test_processor: adapt, check required parameters
Robert Sachunsky
committed
improve _setup docstring

bertsky
and
kba
authored
Processor._setup: raise with full ParameterValidator report

bertsky
and
kba
authored
get_processor: parameter only as kwarg

bertsky
and
kba
authored
tests: adapt for get_processor parameter only as kwarg
Robert Sachunsky
committed
Processor.parameter: make the bound dict read-only

bertsky
and
kba
authored
Processor.parameter: move ParameterValidator back to setter, convert to plain dict in getter for serialization etc
Robert Sachunsky
committed
Processor.parameter: frozendict instead of mappingproxy, add test
Robert Sachunsky
committed
introduce Processor.shutdown to be overridden (called at deinit or parameter re-assignment)
Robert Sachunsky
committed

Commits on Aug 23, 2024

Processor: introduce `max_instances` class attribute
Robert Sachunsky
committed
get_cached_processor: set lru_cache maxsize from min(cfg,class) at runtime
Robert Sachunsky
committed
test get_processor instance_caching w/ max_instances
Robert Sachunsky
committed
test get_processor instance_caching w/ clear_cache
Robert Sachunsky
committed

Commits on Aug 24, 2024

:package: v3.0.0a2

kba
authored and
Robert Sachunsky
committed
remove make *-workaround, we will not do that for v3+

kba
authored and
Robert Sachunsky
committed
Processor.metadata_location property to specify where in the package ocrd-tool.json is found

kba
authored and
Robert Sachunsky
committed
Processor.verify: always check cardinality (as we now have the defaults from ocrd-tool.json)
Robert Sachunsky
committed
fix --log-filename (6fc606027a): apply in ocrd_cli_wrap_processor
Robert Sachunsky
committed
fix exception
Robert Sachunsky
committed
adapt to PIL.Image moved constants
Robert Sachunsky
committed
ocrd_utils: add parse_json_file_with_comments
Robert Sachunsky
committed
cli.workspace: pass fileGrp as well, improve description
Robert Sachunsky
committed
OcrdMets.add_agent: does not have positional args
Robert Sachunsky
committed
remove misplaced kwargs from run_processor
Robert Sachunsky
committed
Processor.metadata: refactor…
Robert Sachunsky
committed
bashlib input-files: adapt, allow passing ocrd-tool.json path and executable name
Robert Sachunsky
committed
add to pylint karma
Robert Sachunsky
committed
update pylintrc
Robert Sachunsky
committed
processor.metadata_location: use self.__module__ not __package__
kba
committed
Merge pull request #17 from OCR-D/new-processor-api-parameter-setup
bertsky
authored

Commits on Aug 25, 2024

pylint: try ignoring generateds (again)
Robert Sachunsky
committed
Merge pull request #14 from bertsky/new-processor-api-parameter-setup
bertsky
authored
:memo: update changelog
Robert Sachunsky
committed
test_bashlib: allow testing prereleases successfully
Robert Sachunsky
committed
Processor.process_page_file / OcrdPageResultImage: allow PageType instead of AlternativeImageType
Robert Sachunsky
committed

Commits on Aug 26, 2024

Commits on Aug 27, 2024

generate_processor_help: avoid repeating docstrings from superclass
Robert Sachunsky
committed
Processor.process_workspace: abort anyway if too many failures (OCRD_MAX_MISSING_OUTPUTS)
Robert Sachunsky
committed
adapt tests for OCRD_MAX_MISSING_OUTPUTS
Robert Sachunsky
committed
Merge pull request #19 from OCR-D/new-processor-api-fix-editable
bertsky
authored
Processor: add per-page timeouts and parallelism…
Robert Sachunsky
committed
add tests for processor per-page timeout and parallelism
Robert Sachunsky
committed
:memo: update changelog
Robert Sachunsky
committed

Commits on Aug 28, 2024

ClientSideOcrdMets: use same logger name prefix as server
Robert Sachunsky
committed
Processor: fix ignore (negative/zero) cases for max_workers / max_page_seconds
Robert Sachunsky
committed
test_mets_server: use tmpdir to avoid side effects between suites
Robert Sachunsky
committed
test processor timeout/parallel: avoid side effects to dummy tool json
Robert Sachunsky
committed
tess: adapt to wording of exceptions
Robert Sachunsky
committed
ClientSideOcrdMets: partial revert of 9b80ae17ef
Robert Sachunsky
committed
disableLogging: re-instate root logger, to
Robert Sachunsky
committed
test-logging: also remove ocrd.log from tempdir
Robert Sachunsky
committed
Processor: fix 7966057f (deprecated passing of ocrd_tool or version via init)
Robert Sachunsky
committed
Processor.generate_processor_help: forgot to include --log-filename
Robert Sachunsky
committed
bashlib: re-add --log-filename, implement as stderr redirect
Robert Sachunsky
committed
test_processor: add legacy (v2-style) dummy case
Robert Sachunsky
committed
:memo: update changelog
Robert Sachunsky
committed
:memo: update readmes (esp. new config variables)
Robert Sachunsky
committed

Commits on Aug 30, 2024

:package: v3.0.0b2
kba
committed
ocrd_utils.config: add reset_defaults()
Robert Sachunsky
committed
add test for OcrdEnvConfig.reset_defaults()
Robert Sachunsky
committed
Processor: improve processing log messages
Robert Sachunsky
committed
ocrd.cli doc: don't rewrap description lists
Robert Sachunsky
committed
:package: v3.0.0b3
kba
committed
Processor.metadata_location: find location package prefix (necessary for namespace packages)
Robert Sachunsky
committed

Commits on Sep 1, 2024

Processor: log when max_workers / max_page_seconds are in effect
Robert Sachunsky
committed
Workspace.reload_mets: fix for METS server case
Robert Sachunsky
committed

Commits on Sep 2, 2024

:memo: changelog
kba
committed
:package: v3.0.0b4
kba
committed
OcrdMetsServer.add_file: pass on 'force' kwarg, too
Robert Sachunsky
committed
test_mets_server: add test for force (overwrite)
Robert Sachunsky
committed
test_processor: add test for force (overwrite) w/ METS Server
Robert Sachunsky
committed

Commits on Sep 3, 2024

add typing, extend docs

kba
authored and
Robert Sachunsky
committed

Commits on Sep 5, 2024

Processor.verify: revert 5819c816 (we still have no defaults in json loaded from v2)
Robert Sachunsky
committed
Processor.process_page_file / OcrdPageResultImage: allow None instead of AlternativeImageType
Robert Sachunsky
committed

Commits on Sep 15, 2024

PcGts.Page.id / make_xml_id: replace '/' with '_'
Robert Sachunsky
committed
ocrd.cli.ocrd-tool resolve-resource: fix (forgot to print result)
Robert Sachunsky
committed
processor CLI: delegate --resolve-resource, too
Robert Sachunsky
committed
METS Server: also export+delegate physical_pages
Robert Sachunsky
committed
ocrd.cli.workspace: consistently pass on --mets-server-url and --backup (also, simplify)
Robert Sachunsky
committed
ocrd.cli.workspace server: add 'reload' and 'save'
Robert Sachunsky
committed
ocrd.cli.bashlib input-files: pass on --mets-server-url, too
Robert Sachunsky
committed
ocrd.cli.validate tasks: pass on --mets-server-url, too
Robert Sachunsky
committed
Processor.process_workspace(): do not show NotImplementedError context if fallback process() raises anything itself
Robert Sachunsky
committed
Processor.verify: check output fileGrps as well (or OCRD_EXISTING_OUTPUT=OVERWRITE|SKIP or disjoint --page-id)
Robert Sachunsky
committed
run_processor: be robust if ocrd_tool is missing steps
Robert Sachunsky
committed
lib.bash: fix errexit
Robert Sachunsky
committed
lib.bash input-files: pass on --mets-server-url, --overwrite, and parameters (necessary for required params)
Robert Sachunsky
committed
add ocrd-filter processor
Robert Sachunsky
committed
ocrd-filter: also remove removed segments from ReadingOrder
Robert Sachunsky
committed
ocrd-filter: register XPath functions under PAGE prefix/NS, precompile, avoid buggy lxml global registration mechanism
Robert Sachunsky
committed
ocrd-filter: simplify parameters (just 'select' instead of 'type' and 'query'), use 'elementpath.XPathParser.external_function' with global registration instead of 'etree.FunctionNamespace' with lo…
Robert Sachunsky
committed
lib.bash input-files: do not try to validate tasks here (impossible to get right with required parameters, and now covered by wrapped Processor.verify() already)
Robert Sachunsky
committed
ocrd_models.OcrdPage: add XPath 2.0 parser and extended functions
Robert Sachunsky
committed
Processor / Workspace.add_file: always force if config.OCRD_EXISTING_OUTPUT==OVERWRITE
Robert Sachunsky
committed
ocrd-filter: adapt (just delegate to OcrdPage.xpath)
Robert Sachunsky
committed
test processors: no need for 'force' kwarg anymore
Robert Sachunsky
committed
OcrdPage: migrate to newest generateds, adapt user methods, re-generate
Robert Sachunsky
committed
tests: make sure ocrd_utils.config gets reset whenever changing it globally
Robert Sachunsky
committed
ocrd_modelfactory.page_from_file: set OcrdPage.revmap to actual reverse mapping
Robert Sachunsky
committed
OcrdPage: add PageType.get_ReadingOrderGroups()
Robert Sachunsky
committed
ocrd_page.to_xml: also allow non-root nodes
Robert Sachunsky
committed
update OcrdPage from generateds
Robert Sachunsky
committed
ocrd-filter: simplify further
Robert Sachunsky
committed

Commits on Sep 16, 2024

Commits on Sep 27, 2024

OcrdMets.get_physical_pages: cover return_divs w/o for_fileIds for_pageIds
Robert Sachunsky
committed
ocrd.cli.workspace: use physical_pages if possible, fix default output_field
Robert Sachunsky
committed
Processor.process_page_file: avoid process_page_pcgts() if OCRD_EXISTING_OUTPUT!=OVERWRITE
Robert Sachunsky
committed
OcrdMets.get_physical_pages: cover return_divs w/o for_fileIds for_pageIds
Robert Sachunsky
committed
ocrd.cli.workspace: use physical_pages if possible, fix default output_field
Robert Sachunsky
committed
Processor.process_page_file: avoid process_page_pcgts() if OCRD_EXISTING_OUTPUT!=OVERWRITE
Robert Sachunsky
committed

Commits on Oct 9, 2024

ocrd_utils.initLogging: also add handler to root logger (to be consistent with file config and prevent imported libraries from initing logging first), but disable propagation for ocrd loggers (to a…
Robert Sachunsky
committed

Commits on Oct 10, 2024

CLI decorator: only import ocrd_network when needed
Robert Sachunsky
committed
Processor w/ OCRD_MAX_PARALLEL_PAGES: ThreadPoolExecutor→ProcessPoolExecutor
Robert Sachunsky
committed

Commits on Oct 17, 2024

Processor.process_workspace: apply timeout on process_page_file worker itself (rather than future query)
Robert Sachunsky
committed
Processor w/ OCRD_MAX_PARALLEL_PAGES: concurrent.futures→loky
Robert Sachunsky
committed

Commits on Oct 19, 2024

Processor w/o OCRD_MAX_PARALLEL_PAGES: dummy instead of executor
Robert Sachunsky
committed
ocrd.process.profile logger: account for subprocess CPU time, too
Robert Sachunsky
committed

Commits on Oct 21, 2024

Processor.process_workspace: improve reporting, raise early if too many failures already (rate will be too low)
Robert Sachunsky
committed

Commits on Oct 23, 2024

Processor: refactor process_workspace into overridable subfuncs
Robert Sachunsky
committed

Commits on Oct 30, 2024

Processor.process_workspace_handle_page_task: do not handler sigint
Robert Sachunsky
committed
Processor.process_workspace_handle_tasks: log nr of ignored exceptions in the end
Robert Sachunsky
committed
Merge pull request #23 from bertsky/new-processor-api-process-worker
bertsky
authored
:package: v3.0.0b6
Robert Sachunsky
committed

Commits on Nov 7, 2024

tests: prevent side effects from ocrd_logging
Robert Sachunsky
committed
initLogging: do not remove any previous handlers/levels
Robert Sachunsky
committed
initLogging: only add root handler instead of multiple redundant handlers with propagate=false
Robert Sachunsky
committed
disableLogging: remove all handlers, reset all levels
Robert Sachunsky
committed
setOverrideLogLevel: override all currently active loggers' level
Robert Sachunsky
committed
logging: increase default root (not ocrd) level from INFO to WARNING
Robert Sachunsky
committed
Processor: update max_workers docstring
Robert Sachunsky
committed

Commits on Nov 11, 2024

initLogging: call disableLogging if already initialized and force_reinit
Robert Sachunsky
committed
Processor: replace weakref with __del__ to trigger shutdown
Robert Sachunsky
committed
Processor parallel pages: log via QueueHandler in subprocess, QueueListener in main
Robert Sachunsky
committed

Commits on Nov 12, 2024

:package: v3.0.0b7
Robert Sachunsky
committed

Commits on Jan 6, 2025

ocrd_cli_wrap_processor: always do initLogging

bertsky
authored and
kba
committed

Commits on Jan 8, 2025