Merge pull request #416 from digital-land/msj/amd-CA-dups

 De-duplication of CA data

Author: Swati-Dash

docs/data-operations-manual/Tutorials/Monitoring-Data-Quality.md

@@ -214,3 +214,42 @@ Sometimes it may take a long time for data to be transitioned to a newly created
 5. The entity-organisation range should be assigned to the new organisation for any of the entity numbers which are now being used for the new organisation's records.
 
 6. [Retire any endpoints](../../How-To-Guides/Retiring/Retire-endpoints.md) for the old organisation's provisions so they are no longer collected.
+
+
+## De-duplication of conservation-area data
+
+The purpose of this process is to ensure that duplicate data is not stored unnecessarily for the conservation-area dataset generated by an organisation which may have also been provided by Historic England(HE).
+
+The steps required for this process:-
+
+1. Run the add-data tasks for conservation-area dataset (making a note of how many entities were added in the lookup file).
+
+2. Raise the pull-request(PR) and ensure that it has been merged into the main branch so that the duplicate entities are picked up by the expectation report on the following day.
+
+3. `DO NOT` inform the organisation at this stage.
+
+4. On Power BI navigate to the "Digital Planning" workspace then to the "Planning Data Monitoring" report from where you select the "Duplicate Conservation Area" page.(Link_[0])
+
+5. Click on the reports TITLE in order for the options panel to appear to right hand side
+
+6. Click on the three dots for the more options dropdown menu, from which you select "Export data" to download the output.
+
+7. Open up the exported file to show the HE duplicate entites.
+
+8. Filter on the message column for "complete_match" criteria
+
+9. Filter on the entity_a_organisation.name column for the organisation Historic England and filter on the entity_b_organisation.name column for the organisation for which the data was added on the previous day (re:step 1)
+
+10. Copy the entities in columns entity_a and entity_b
+
+11. Prepare the data to be appended to the old-enity.csv located at Link_[1] in following format
+where entity_a=old-entity and entity_b=entity
+e.g. 44012512,301,44013703,,redirect Historic England duplicate to LPA entity,2025-08-28,
+
+12. `Also DO NOT forget` to update the entity-organisation file located at Link_[2]
+
+13. When this change is merged, check the PowerBI report to confirm the duplicate entities have been fixed.
+
+[0]: <https://app.powerbi.com/groups/80b5c556-2a94-402f-bd6a-225e9a9b6561/list?experience=power-bi>
+[1]: <config/pipeline/conservation-area/old-entity.csv at main · digital-land/config>
+[2]: <config/pipeline/conservation-area/entity-organisation.csv at main . digital-land/config>