Closed
Conversation
edescalona
force-pushed
the
17.0-imp-connector_importer
branch
from
8c1e08d to
e2b1aa8
Compare
[11.0] connector_importer more tests and cleanup fix tests make record handler and tracker real components connector_importer: add xmlid converter connector_importer: load mapper options on each chunk, log values debug c_importer: use serialized fields, add shared storage [FIX] invalidate_cache for set_shared (OCA#3) * [FIX] invalidate_cache for set_shared * fixup! [FIX] invalidate_cache for set_shared Correct method name in autogenerated cron Preserve original values on create/write Record handler: allow force of create_uid/write_uid Record handler: allow force of create_date [FIX] check None instead of falsy on reporter [FIX] base64 conversion [FIX] for csv writer as it expects a string in python3 Recordset importer: drop existing records before run Record handler better handling of create/write ctx The odoorecord handler now takes a custom context, if any, from the importer. In this way you can customize it via importer w/out having to define a custom record handler for each importer. All create/write actions now pass a new ctx key around: `connector_importer_session` Additionally, `tracking_disable` is ENABLED by default since it's very unlikely that you want to track all the changes when import from external sources. TODO: make tracking disabling customizeable via recordset conf. Record: store JSON data as attachment Add `is_last_importer` flag to mark last importer for a recorset Add FIXME reminder for job relations PEP8 [IMP] always create records, when unique key is not defined [FIX] csv file conversion [FIX] report generation [FIX] filesize computation update README connector_importer: disable some lint checks (OCA#9) [ADD] icon.png [MIG] connector_importer: Migration to 12.0 [UPD] README.rst connector_importer: Add field for encoding override connector_importer: Fix unicode error wen trying to sniff dialect connector_importer: allow override of record_handler and tracker conector_importer: allow log_* methods to be called without odoo_record [IMP] connector_importer: new components allowing to import standard Odoo CSV files - new record importer 'importer.record.csv.std' using the 'load()' method to import records - new mapper 'importer.base.mapper.csv.std' converting the data automatically without the internal keys (like '_line_nr') - new record handler 'importer.odoorecord.handler.csv.std' allowing to look for records based on the XML-ID provided by the CSV file - allow the source mixin to not chunk the extracted lines by setting chunk_size to 0 (before a ZeroDivisionError was raised) - the existing CSVReader (using csv.DictReader) has been kept to continue to read record as dictionaries and thus keeping the compatibility with all existing features (like tracker and reporter) - reporter has been fixed to do not crash when there are quotes in column names [IMP] connector_importer: black, isort [MIG] connector_importer: Migration to 13.0 [UPD] Update connector_importer.pot [FIX] connector_importer: fix computed fields connector_importer 13.0.1.0.1 [FIX] connector_importer: fix 'source_ref_id' field, don't store it to not crash Odoo [IMP] connector_importer: handle the unique key as an external/XML ID (OCA#56) * [IMP] connector_importer: handle the unique key as an external/XML ID 'importer.record' and 'importer.odoorecord.handler' components have been updated to handle the 'odoo_unique_key' attribute as an XML-ID if the new attribute 'odoo_unique_key_is_xmlid' is set to 'True'. This XML-ID will then be used as usual to find an existing record, and will be set on the created record if it doesn't already exist. connector_importer 13.0.1.1.1 connector_importer: ease override of mapper connector_importer 13.0.1.2.0 pre-commit update [IMP] connector_importer: new option 'use_job' on import type By default one job is spawned for each data model/importer used in a import type. It works well if the records to import have no relation between them, but if we choose to disable this behavior we are able to link records imported from one importer to another as we are in the same SQL transaction (e.g. an order and its lines). connector_importer 13.0.1.3.0 [FIX] connector_importer: guess the CSV encoding when generating the report [IMP] connector_importer: set a default_channel on jobs connector_importer 13.0.1.3.2 Fix odoo warning about oldname no longer supported In logs: Field import.recordset.report_data: parameter 'oldname' is not longer supported; use an upgrade script instead. Since 'oldname' was used in a previous odoo version, it can be removed. connector_importer 13.0.1.3.3 Make pre-commit happy connector_importer: trigger on_record_import_finished Trigger a specific event when the last importer has done its job. connector_importer: fix/imp/refactor tests * get rid of custom test model setup * sort out base classes * cleanup messy dependencies between test cases * fix not running source tests connector_importer: split source mixins to own file connector_importer: source summary tmpl support m2o connector_importer: remove useless caching connector_importer: fix permission for admin connector_importer: improve recordset view connector_importer: add me as maintainer connector_importer 13.0.1.4.0 connector_importer: fix import.backend ACL connector_importer 13.0.1.4.1 [IMP] : black, isort, prettier [14.0][MIG] connector_importer connector_importer: make job.related.mixin real model connector_importer: make 'check_delete' private connector_importer: fix warnings c_importer: recordset importer raise error w/ no source c_importer: refactor import type settings as YAML This change introduces way more flexibility so that we can pass advanced settings from import type down to all the components used in the import c_importer: purge not needed values on write * purge keys not matching model's schema * optionally purge not changed values via `options.record_handler.skip_fields_unchanged` This way you can avoid writing the same value again and again on a field that already has that value. Especially handy if you have event handlers that rely only on the field name to be there. connector_importer: update roadmap connector_importer: improve common tests utils connector_importer: test common improve component load c_importer: drop custom ctx key 'test_components_registry' Supported in core component now OCA/connector#387 c_importer: fix last_record_import_finished trigger w/ right colleciton c_importer: delegate full prep for import to recordset helper c_importer: importer allow write only c_importer: trigger import finished for specific model The importer now triggers a generic event for the whole recordset and a specific event for the current model. This way you can hook to model specific events. c_importer: importer return counters The string 'ok' was useless. Getting the counters right away helps testing and inspecting current result for the importer. c_importer: allow empty unique key This is very handy when you provide your own find domain. Just set 'unique_key = None'. c_importer: fix docstring c_importer: fix typo for onchange method name connector_importer: cleanup warnings Make pre-commit happy c_importer: finish migration to v14 [IMP] connector_importer: black, isort, prettier [MIG] connector_importer: Migration to 15.0 [UPD] Update connector_importer.pot [UPD] README.rst connector_importer: add bool converter connector_importer: manage break flag via options You can now configure an importer like this: options: importer: break_on_error: true to force it to break. connector_importer: transl key config via options You can now configure the lang key at import type level by settings these keys on the importer options: options: importer: translation_key_sep: _ translation_use_regional_lang: false connector_importer: fix import_record config from job When importer config is loaded from the DB via job arguments you don't have a DotDict anymore. It should be fixed by passing the importer_type_id IMO but for now is ok. connector_importer: fix rendering of docs Importers taken to generate docs must be initialized in the same way as whem they are used for importing records. This way 'options' and 'context' are propagated properly. c_importer: add base context for create/write c_importer: mapper.default_values add xmlid validation c_importer: improve datetime mapper func c_importer: propagate values to report item Allows for better / custom tracking info management. [UPD] Update connector_importer.pot [UPD] README.rst connector_importer 15.0.1.1.0 connector_importer: add dynamic mapper connector_importer: improve xid handling connector_importer: fix logger.warn deprecated connector_importer: mapper_utils store from key This way we can inspect the converter function to retrieve the original key. connector_importer: support multi valued xids c_importer: backend_to_rel support Odoo studio connector_importer: add import type view connector_importer: fix importer.record apply_on The importer for a record should be model agnostic otherwise we cannot use it for any other odoo model. This is very important to support custom mappers from settings w/out specific importers. connector_importer: allow custom mapper from conf You can now declare the mapper to be used via import.type options. connector_importer: improve dynamic mapper You can now pass via import type options: * source_key_prefix -> to filter all keys via specific prefix This is really handy when in your source (normally a CSV) you have a set of fields to be used for creating/updating other records * source_key_whitelist -> consider only these specific keys from the source The prefix will be cleaned up automatically and what remains after it must be a real Odoo field on the current model. connector_importer: support 'override_existing' from conf You can now pass override_existing flag in the importer options. connector_importer: allow mapper settings from conf You can now pass: * required_keys * translatable_keys * default_keys in the 'mapper' section of import type options. Eg: - model: foo importer: name: bla options: mapper: name: my.mapper required_keys: source_key: dest_key __source_key2: dest_key2 default_keys: field1: true connector_importer: unify importer lookup by type conf connector_importer: fix recordset name compute Addig the name of the backend is useless and can be broken till you have a backend. connector_importer: fix recordset docs compute connector_importer: add source lines slice option You can decide to work on a specific range of CSV lines. connector_importer: add source csv tree + menu connector_importer: add import type desc c_importer: dynamic mapper add source_key_empty_skip New config key to allow skipping fields when they are empty. Use cases: * field w/ unique constraint but not populated (eg: product barcode) * field not to override when empty [UPD] Update connector_importer.pot connector_importer: dynamicmapper tests + fixes connector_importer: validate translatable keys Translatable keys might refer non translatable fields or not existing fields: purge them if needed and log proper errors. c_importer: transl use regional lang False by default [UPD] Update connector_importer.pot connector_importer 15.0.1.2.0 c_importer: abandon need for unique_key_is_xmlid attribute c_importer: sanitize xids w/ xmlid_to_rel Make sure XIDs are always complete. Make module customizable anyway. c_importer: dynamic mapper handle selection field c_importer: dynamic mapper get required defaults Automatically load defaults for required keys that are missing. You can still bypass this behavior by using 'source_key_empty_skip' option. connector_importer: update ROADMAP connector_importer: assume 'id' is always an xmlid connector_importer: importer.record is the default one connector_importer: update ROADMAP connector_importer: break if unique_key_is_xmlid found In the v15 refactoring we eliminated the need for 'unique_key_is_xmlid' attribute. To ease the migration of existing modules, break immediately and let know people they have to fix it. [UPD] Update connector_importer.pot [UPD] README.rst connector_importer 15.0.1.3.0 connector_importer: fix dynamic mapper The inner func 'modifier' is returned, not the outer. Checking the '_from_key' attribute was useless as it was never found into 'direct' mappings. connector_importer: support unique key into orig values c_importer: dynamic mapper skip empty required keys Discard empty required keys a priori. If a destination key which is required is empty very likely is wrong (then you should fix your data) or is not meant to be overwritten to not wipe existing data. c_importer: dynamic mapper fix text convert c_importer: dynamic mapper log if no converter found [UPD] Update connector_importer.pot connector_importer 15.0.1.4.0 connector_importer: make not installable connector_importer: get rid of legacy settings connector_importer: use delayable recordset connector_importer: silent test logs connector_importer: improve backend view connector_importer: add test for source.config_summary connector_importer: test docs_html + report_html connector_importer: fix _compute_docs_html connector_importer: fix _compute_report_html connector_importer: add icon connector_importer: add TODO in ROADMAP connector_importer: fix source test connector_importer: skip import of not ready recordsets If the source is not ready yet, avoid running them. connector_importer: add recordset.last_run_on Ease understanding if/when a recordset has been ran already. connector_importer: source.csv handle no file set Fail gracefully if a source file is not set yet. connector_importer: add source shortcut on backend view connector_importer: fix source view connector_importer: update ROADMAP connector_importer: add safe conversion to b64 connector_importer: dynamic mapper blacklist connector_importer: allow custom find domain You can now configure a custom domain on record_handler options. Eg: options: record_handler: match_domain: "[('name', =, values['name'])]" When defined, it will take precedence over the unique key if any. connector_importer: fix odoo_find Before this change if you did not override 'odoo_find' itself your custom odoo_find_domain would be bypassed if you had an empty unique_key. This was a clear issue when you specified only 'options.record_handler.match_domain'. connector_importer: fix xmlid_to_rel w/ x2m Values where not properly converted for x2m fields. connector_importer: raise exc if no domain and no uniq key connector_importer: improve domain eval ctx You can now use ref and env to compute your domain. connector_importer: fix warning in tests connector_importer: fix handling of xid:: + prefix Keys having both prefix and a transformer like 'xid::' were discarded. A key like 'xid::foo.parent_id' would simply be ignored. connector_importer: allow pass options to converter You can now pass options for specific fields converter via conf. If for instance, the model you are importing has a relation `partner_id` you can state that missing records must be created automatically. Eg:: - model: product.product options: importer: odoo_unique_key: barcode mapper: name: product.product.mapper converter: categ_id: create_missing: True All the keys inside converted/field will be propagated to the `backend_to_rel` converter. c_importer: improve trigger events c_importer: add recordset.get_report_by_model To ease retrieval of import session reports. c_importer: allow exec server actions on recordset c_importer: allow ctx key via options You can now pass any context key to the importer (propagated to the record handler) by using 'ctx' key in the 'importer' options. connector_importer: fix import.type copy c_importer: fix recordset direct creation Was broken because available_importers requires an import type which is not available before editing. It was also impossible to set the backend directly. connector_importer: fix test_mapper obsolete code Make pre-commit happy connector_importer: migrate to v16 connector_importer: fix test_recordset.test_docs_html Without components registered this is broken however it is already tested in test_recordset_importer [UPD] Update connector_importer.pot [UPD] README.rst connector_importer: allow rename keys via conf You can now configure mapper options' key source_key_rename to specify a mapping of keys to be renamed to match odoo field names. connector_importer: improve job details [BOT] post-merge updates Translated using Weblate (Italian) Currently translated at 100.0% (154 of 154 strings) Translation: connector-interfaces-16.0/connector-interfaces-16.0-connector_importer Translate-URL: https://translation.odoo-community.org/projects/connector-interfaces-16-0/connector-interfaces-16-0-connector_importer/it/ connector_importer: define serialized fields as binary Replace the use of Serialized fields with Binary fields for report_data and shared_data in the ImportRecordset model. This change aims to improve performance and provide better storage management for large datasets. connector_importer: invalidate cache and flush pending computations [BOT] post-merge updates [IMP] connector_importer: fix modifier method is too complex [BOT] post-merge updates connector_importer: Fix _run_import method to respect debug mode and use_job parameter We want to handle debug mode and take into account the use_job parameter in order to spawn jobs only when requested Jobs are not spawn by default if not in debug mode [BOT] post-merge updates Added translation using Weblate (French) connector_importer: job_id index Add index on job_id in job.related.mixin to allow to delete a queue job without causing a full scan [BOT] post-merge updates Translated using Weblate (French) Currently translated at 38.3% (59 of 154 strings) Translation: connector-interfaces-16.0/connector-interfaces-16.0-connector_importer Translate-URL: https://translation.odoo-community.org/projects/connector-interfaces-16-0/connector-interfaces-16-0-connector_importer/fr/ [IMP] connector_importer: pre-commit execution [MIG] connector_importer: Migration to 17.0 [IMP] connector_importer: replace t-esc and t-raw with t-out for improved output handling in templates [17.0][IMP] connector_importer: Add new source for API [17.0][FIX] connector_importer: Pre-commit [17.0][IMP] connector_importer: Adding external dependency [17.0][IMP] connector_importer: Remove dependency [17.0][IMP] connector_importer: Dependencies
edescalona
force-pushed
the
17.0-imp-connector_importer
branch
from
72ac1ca to
25e0b90
Compare
rrebollo
reviewed
May 6, 2025
rrebollo
left a comment
rrebollo
left a comment
There was a problem hiding this comment.
Have you considered proposing this feature as a new add-on that extends connector_importer? This approach might offer some advantages. What are your thoughts on this direction?
Author
|
Hi @rrebollo , yes that could be a good option, thanks for your comments. |
simahawk
requested changes
May 7, 2025
Contributor
simahawk
left a comment
simahawk
left a comment
There was a problem hiding this comment.
Thanks for your contrib.
A bunch of things to be fixed:
- the module is not migrated yet to 17 -> we need a separate PR for this
- your commits are messed up, pls refer to the guidelines https://github.com/OCA/maintainer-tools/wiki/Migration-to-version-17.0
- a separated module should be added. You can see an example for an additional SFTP source in v14 😉
Author
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@BinhexTeam
A new source has been added for API use.
Main features: