GH-72, 77: Fix data inconsistencies and remove derived values across years for all datasets (#92)

* fix: standardize column names in various datasets

* fix typo

* fix: standardize purpose names across multiple datasets

* fix: remove 'Total' column from age group datasets for consistency

* fix: remove unnecessary columns from gender registration datasets for consistency

* fix: remove 'Percentage(%)' column from foreign employment registration datasets for consistency

* fix: remove 'Total' column from foreign employment registration dataset for consistency

* fix: standardize column name 'No. of Detainees removed' to 'Number of Deportations' across multiple datasets for consistency

* fix: remove 'Total' column from age group datasets for consistency

* Remove "Total" column, row and update district names for consistency in SLBFE registration data by district, manpower level, and gender for 2020, 2021, 2022, and 2023

* fix: standardize column names for across multiple datasets for slbfe registrations by district, manpower and gender for consistency

* fix inconsistencies columns in slbfe registration by manpower level

* update metadata column counts for slbfe registrations by age group

* update metadata column counts for slbfe registrations by gender

* update metadate column counts and remove total row from slbfe registration by country datasets

* updates metadata column counts for slbfe registrations by age and manpower

* update metadata row and column counts for slbfe registrations by district, manpower level and gender

* update 2024 slbfe manpower vs gender dataset column names to match the other years

* normalise coulmn names and remove total values in slbfe registration by country vs manpower level, update metadata files

* (Tourism) Normalize country names across arrivals by country datasets

* (Toursim) Normalise country names across arrivals by month vs country datasets

* update row counts in metadata for arrivals by month vs country

* remove 'others' from top 10 source markets

* (SLBFE) normalise country names across datasets for registration by country vs manpower level

* fix typos and normalise nationality names across asylum seekers datasets

* fix typos and normalise nationality names across deported foreign nationals datasets

* fix typos and normalise nationality names across refugees by nationality datasets

* fix typos and normalise nationality names across refused entry by nationality datasets

* fix typos in nationalities

* normalise place names for location vs revenue vs visitors count datasets

* remove percentage value from workers remittances across the years

* remove total value from arrivals by month

* remove derived values for slbfe registration by all sources vs country and private remittances

* normalise country names for slbfe registratiion by all sources vs country

* remove total values from workers' remittances by country dataset

* Update data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/slbfe_registration(AS_CATEGORY)/by_country(AS_CATEGORY)/SLBFE_registration_by_country/data.json

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update data/2020/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/State Minister of Foreign Employment Promotion and Market Diversification(minister)/Sri Lanka Foreign Employment Bureau(department)/foreign_employment(AS_CATEGORY)/slbfe_registration(AS_CATEGORY)/by_country(AS_CATEGORY)/SLBFE registration by country/data.json

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Author: sehansi-9

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/slbfe_registration(AS_CATEGORY)/by_country(AS_CATEGORY)/SLBFE_registration_by_country/data.json

@@ -1,62 +1,61 @@
 {
-  "columns": ["Country", "Number", "Percentage(%)"],
+  "columns": ["Country", "Number"],
   "rows": [
-    ["Saudi Arabia", 35415, 17.44],
-    ["Kuwait", 43073, 21.21],
-    ["U.A.E.", 328690, 16.18],
-    ["Qatar", 40783, 20.08],
-    ["Oman", 9016, 4.439],
-    ["South Korea", 6207, 3.056],
-    ["Maldives", 7767, 3.824],
-    ["Jordan", 4611, 2.27],
-    ["Bahrain", 3017, 1.486],
-    ["Lebanon", 1902, 0.937],
-    ["Malaysia", 3296, 1.623],
-    ["Israel", 1559, 0.768],
-    ["Cyprus", 2421, 1.192],
-    ["Singapore", 2124, 1.046],
-    ["Seychelles", 977, 0.481],
-    ["Bangladesh", 576, 0.284],
-    ["Hong Kong", 624, 0.307],
-    ["Mauritius", 91, 0.045],
-    ["Iraq", 183, 0.09],
-    ["Kurdistan", 156, 0.077],
-    ["India",244,0.12],
-    ["Afghanistan", 109, 0.054],
-    ["Japan", 1594, 0.785],
-    ["New Zealand", 460, 0.227],
-    ["Romania", 2312, 1.138],
-    ["Papua New Guinea", 69, 0.034],
-    ["Ethiopia", 296, 0.146],
-    ["Italy", 72, 0.035],
-    ["Kenya", 37, 0.018],
-    ["Uganda", 52, 0.026],
-    ["Pakistan", 38, 0.019],
-    ["Egypt", 68, 0.033],
-    ["Fiji", 30, 0.015],
-    ["Sudan", 17, 0.008],
-    ["Brunei", 17, 0.008],
-    ["Vietnam", 40, 0.02],
-    ["Australia", 40, 0.02],
-    ["Ireland",27,0.013],
-    ["Yemen", 2, 0.0003],
-    ["Libya", 0, 0.00],
-    ["United States", 15, 0.007],
-    ["China", 17, 0.008],
-    ["Greece", 28, 0.014],
-    ["Malta", 111, 0.055],
-    ["Myanmar",19,0.009],
-    ["Botswana", 19, 0.009],
-    ["Malawi",21,0.01],
-    ["Thailand", 31, 0.015],
-    ["South Africa", 8, 0.004],
-    ["United Kingdom", 25, 0.012],
-    ["Turkey", 68, 0.033],
-    ["Mozambique", 17, 0.008],
-    ["Lithuania", 73, 0.036],
-    ["Angola", 8, 0.004],
-    ["Djibouti", 10, 0.005],
-    ["Other", 435, 0.214],
-    ["Total", 203087, 100.00]
+    ["Saudi Arabia", 35415],
+    ["Kuwait", 43073],
+    ["U.A.E.", 328690],
+    ["Qatar", 40783],
+    ["Oman", 9016],
+    ["South Korea", 6207],
+    ["Maldives", 7767],
+    ["Jordan", 4611],
+    ["Bahrain", 3017],
+    ["Lebanon", 1902],
+    ["Malaysia", 3296],
+    ["Israel", 1559],
+    ["Cyprus", 2421],
+    ["Singapore", 2124],
+    ["Seychelles", 977],
+    ["Bangladesh", 576],
+    ["Hong Kong", 624],
+    ["Mauritius", 91],
+    ["Iraq", 183],
+    ["Kurdistan", 156],
+    ["India", 244],
+    ["Afghanistan", 109],
+    ["Japan", 1594],
+    ["New Zealand", 460],
+    ["Romania", 2312],
+    ["Papua New Guinea", 69],
+    ["Ethiopia", 296],
+    ["Italy", 72],
+    ["Kenya", 37],
+    ["Uganda", 52],
+    ["Pakistan", 38],
+    ["Egypt", 68],
+    ["Fiji", 30],
+    ["Sudan", 17],
+    ["Brunei", 17],
+    ["Vietnam", 40],
+    ["Australia", 40],
+    ["Ireland", 27],
+    ["Yemen", 2],
+    ["Libya", 0],
+    ["United States", 15],
+    ["China", 17],
+    ["Greece", 28],
+    ["Malta", 111],
+    ["Myanmar",19],
+    ["Botswana", 19],
+    ["Malawi",21],
+    ["Thailand", 31],
+    ["South Africa", 8],
+    ["United Kingdom", 25],
+    ["Turkey", 68],
+    ["Mozambique", 17],
+    ["Lithuania", 73],
+    ["Angola", 8],
+    ["Djibouti", 10],
+    ["Other", 435]
   ]
 }

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/slbfe_registration(AS_CATEGORY)/by_country(AS_CATEGORY)/SLBFE_registration_by_country/metadata.json

@@ -1,8 +1,8 @@
 {
   "storage_type": "tabular",
   "dataset_name": "SLBFE registration by country",
-  "column_count": 3,
-  "row_count": 57,
+  "column_count": 2,
+  "row_count": 56,
   "node_count": null,
   "edges_count": null,
   "resource": "https://www.slbfe.lk/wp-content/uploads/2023/09/Statistics-2020.pdf",

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/slbfe_registration(AS_CATEGORY)/by_gender(AS_CATEGORY)/SLBFE_registration_by_gender/data.json

@@ -1,6 +1,6 @@
 {
-"columns": ["Year","Male (Number)", "Male (%)", "Female (Number)", "Female (%)","Total"],
+"columns": ["Year","Male", "Female"],
 "rows": [
-  ["2019",122257,60.20,80830,39.80,203087]
+  ["2019",122257,80830]
 ]
 }

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/slbfe_registration(AS_CATEGORY)/by_gender(AS_CATEGORY)/SLBFE_registration_by_gender/metadata.json

@@ -1,7 +1,7 @@
 {
   "storage_type": "tabular",
   "dataset_name": "SLBFE registration by gender",
-  "column_count": 6,
+  "column_count": 3,
   "row_count": 1,
   "node_count": null,
   "edges_count": null,

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/slbfe_registration(AS_CATEGORY)/by_manpower_level(AS_CATEGORY)/SLBFE_registration_by_manpower_level/data.json

@@ -1,6 +1,24 @@
 {
-  "columns": ["Year", "Professional Level (Number)", "Professional Level (%)", "Middle Level (Number)", "Middle Level (%)", "Skilled (Number)", "Skilled (%)", "Clerical & Related (Number)", "Clerical & Related (%)", "Semi Skilled Domestic housekeeping assistants (Number)", "Semi-Skilled Others (Number)","Semi-Skilled Others (Total)", "Semi-Skilled Others (%)", "Low skilled (Number)", "Low skilled (%)", "Total"],
+  "columns": [
+    "Year",
+    "Professional Level",
+    "Middle Level",
+    "Skilled",
+    "Clerical & Related",
+    "Semi Skilled Domestic housekeeping assistants",
+    "Semi Skilled Others",
+    "Low Skilled"
+  ],
   "rows": [
-    ["2019", 9861, 4.85, 5725, 2.82, 62711, 30.88, 9163, 4.51, 61489, 2950, 64439, 31.73, 51188, 25.20, 203087]
+    [
+      "2019",
+      9861,
+      5725,
+      62711,
+      9163,
+      61489,
+      2950,
+      51188
+    ]
   ]
 }

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/slbfe_registration(AS_CATEGORY)/by_manpower_level(AS_CATEGORY)/SLBFE_registration_by_manpower_level/metadata.json

@@ -1,7 +1,7 @@
 {
   "storage_type": "tabular",
   "dataset_name": "SLBFE registration by manpower level",
-  "column_count": 16,
+  "column_count": 8,
   "row_count": 1,
   "node_count": null,
   "edges_count": null,

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/workers_remittances(AS_CATEGORY)/data.json

@@ -1,6 +1,6 @@
 {
-"columns": ["Year", "Remittances Middle East (Rs. Million)", "Remittances Middle East (US$ Million)",  "Remittances Total (Rs. Million)", "Remittances Total (US$ Million)", "Middle East as a % of total remittance"],
+"columns": ["Year", "Remittances Middle East (Rs. Million)", "Remittances Middle East (US$ Million)",  "Remittances Total (Rs. Million)", "Remittances Total (US$ Million)"],
 "rows": [
-  ["2019", 618394, 3459, 1200766, 6717, 51.5]
+  ["2019", 618394, 3459, 1200766, 6717]
 ]
 }

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Telecommunication, Foreign Employment and Sports(minister)/Sri Lanka Foreign Employment Bureau(department)/workers_remittances(AS_CATEGORY)/metadata.json

@@ -1,7 +1,7 @@
 {
   "storage_type": "tabular",
   "dataset_name": "Workers Remittances",
-  "column_count": 6,
+  "column_count": 5,
   "row_count": 1,
   "node_count": null,
   "edges_count": null,

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Tourism Development, Wildlife and Christian Religious Affairs(minister)/Sri Lanka Tourism Development Authority(department)/arrivals(AS_CATEGORY)/by_purpose(AS_CATEGORY)/tourist_arrivals_by_purpose_of_visit/data.json

@@ -5,27 +5,27 @@
   ],
   "rows": [
     [
-      "Pleasure / Vacation",
+      "Pleasure/Vacation",
       83.2
     ],
     [
-      "Visiting friends & relatives",
+      "Visiting Friends & Relatives",
       10.48
     ],
     [
       "Business",
       3.65
     ],
     [
-      "Convention & Meeting",
+      "Convention/Meeting",
       0.99
     ],
     [
       "Sports",
       0.72
     ],
     [
-      "Health / Ayurveda",
+      "Health/Ayurveda",
       0.59
     ],
     [
@@ -37,7 +37,7 @@
       0.3
     ],
     [
-      "Others",
+      "Other/Not Responded",
       0.1
     ],
     [

data/2019/Government of Sri Lanka(government)/Gotabaya Rajapaksa(citizen)/Minister of Tourism Development, Wildlife and Christian Religious Affairs(minister)/Sri Lanka Tourism Development Authority(department)/arrivals(AS_CATEGORY)/by_sex(AS_CATEGORY)/tourist_arrivals_by_gender/data.json

@@ -1,6 +1,6 @@
 {
   "columns": [
-    "Gender",
+    "Sex",
     "Percentage"
   ],
   "rows": [