Redesign config file

[marlanperumal]

Create a v2 psr config to keep backwards compatibility

Design Features of the new config format should include:

• Clearly delineate input format (describing how the pdf is structured) from processing (cleaning, transforming and other processing to a standard pandas dataframe format) and and output format (describing what the output should look like including file format, data type format, column selection and ordering)
• Config at the overall document/table level, which can be overridden at the page, column or row level
• This will likely involved some sort of nested duplication of config attributes
• Sensible defaults when not all config options are provided
• Extensibility as more features are added
• Support for reading in generic tables not just bank statements (though there should be further subtype options for subtypes like bank statements, time series data, matrix data etc.)
• Don't try to do too much. Functionality focus should be on extracting pdf data and outputting a valid, usable csv. Anything that can be easily dealt with the data in csv format can be left as a postprocessing task for the user in whatever toolchain they're most comfortable it
• Allow for config file format as JSON, YAML and perhaps other standards

Specific use case features should include:

• Current data cleaning options
  • Stripping of non-numeric characters
  • Thousand separator
  • Decimal character
  • Negative value indicator (e.g. -1000, 1000-, (1000), 1000 DR etc.)
• Selecting specific pages or page ranges to include or exclude
• Flagging of rows as invalid data
• Inclusion of invalid data rows as "extra" fields in the last preceding valid row
• Per column data format
• Programming language neutral date format specification
• Output column ordering

[marlanperumal]

Dumped the request in Cursor and this is the suggestion that came back from Claude. Not exactly what I'd like it to be but a good place to start. First impressions are that YAML is a nicer format for writing the config.

master...feature/75-redesign-config-file

[chandu]

Thanks will use this as baseline and we can refine it once I take a stab at it.
Are you comfortable with using typescript types for finalizing the required structure and then porting it to JSON schema or yaml (am more comfortable this way to express the contracts, but I can work with JSON schema or yaml if need be)

[marlanperumal]

Yeah that's fine if that's easier for you. For now it's more important to land on the right nested structure. We can figure out the specific types afterwards.

…

On Mon, 27 Jan 2025, 18:26 Chandu, ***@***.***> wrote: Thanks will use this as baseline and we can refine it once I take a stab at it. Are you comfortable with using typescript types for finalizing the required structure and then porting it to JSON schema or yaml (am more comfortable this way to express the contracts, but I can work with JSON schema or yaml if need be) — Reply to this email directly, view it on GitHub <#77 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPADLKBLIU5OZF2VNQRTQT2MZM4FAVCNFSM6AAAAABV6PJLHSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOJXGIYTSOI> . You are receiving this because you authored the thread.Message ID: <marlanperumal/pdf_statement_reader/repo-discussions/77/comments/11972199@ github.com>