From e5c2ef2d12fc5c841271329a968f078ab85ddfa4 Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:26:28 +0200 Subject: [PATCH 01/11] Update naming.rst correcting the naming convention description --- docs/naming.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/naming.rst b/docs/naming.rst index ba9bd14..49e61e3 100644 --- a/docs/naming.rst +++ b/docs/naming.rst @@ -7,13 +7,13 @@ Technology Technologies in OSeMBE are named according to the following structure: +---------+-----------+-------------------+--------------+-------+-------+-----+----------------+ -| **AA** | **BB** | **CC** | **DD** | **E** | **F** | | **AABBCCDDEF** | +| **AA** | **BB** | **CC** | **D** | **E** | **F** | | **AABBCCDEF** | +---------+-----------+-------------------+--------------+-------+-------+-----+----------------+ | Country | Commodity | Technology / | Energy level | Age | Size | --> | Technology | | | | connected Country | | | | | name | +---------+-----------+-------------------+--------------+-------+-------+-----+----------------+ -The different codes are available under :doc:`Countries <../countries>`, :doc:`Fuels/Commodities <../fuels>` and :doc:`Technologies <../technologies>`. The energy level is either P if a primary energy commodity is the output, or F if final electricity is the output, I indicates an import technology, and X an extraction or generation technology. +The different codes are available under :doc:`Countries <../countries>`, :doc:`Fuels/Commodities <../fuels>` and :doc:`Technologies <../technologies>`. The energy level is either P if a primary energy commodity is the output, or F if final electricity is the output, I indicates an import technology, and X an extraction or generation technology. The age can have the values H or N which stand for historic and new. The size can be 1 indicating small, 2 medium or 3 large technologies. Fuels ----- @@ -24,4 +24,4 @@ Commodities in OSeMBE are named according to the following structure: | **AA** | **BB** | | **AABB** | +---------+-----------+-----+----------------+ | Country | Commodity | --> | Commodity name | -+---------+-----------+-----+----------------+ \ No newline at end of file ++---------+-----------+-----+----------------+ From 77d7b55c7322f7a581fd08db319d6c6c294deb18 Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:28:17 +0200 Subject: [PATCH 02/11] Update naming.rst --- docs/naming.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/naming.rst b/docs/naming.rst index 49e61e3..87c46e2 100644 --- a/docs/naming.rst +++ b/docs/naming.rst @@ -7,7 +7,7 @@ Technology Technologies in OSeMBE are named according to the following structure: +---------+-----------+-------------------+--------------+-------+-------+-----+----------------+ -| **AA** | **BB** | **CC** | **D** | **E** | **F** | | **AABBCCDEF** | +| **AA** | **BB** | **CC** | **D** | **E** | **F** | | **AABBCCDEF** | +---------+-----------+-------------------+--------------+-------+-------+-----+----------------+ | Country | Commodity | Technology / | Energy level | Age | Size | --> | Technology | | | | connected Country | | | | | name | From 4db177682bc0f21d62fb1e15c3fffd1364b99fc5 Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:31:14 +0200 Subject: [PATCH 03/11] Update emissions.rst Included dummy emissions --- docs/emissions.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/emissions.rst b/docs/emissions.rst index 0d44441..db3a812 100644 --- a/docs/emissions.rst +++ b/docs/emissions.rst @@ -7,4 +7,11 @@ Emissions | CO2 | Carbon dioxide | +------+------------------------+ | PM25 | Particulate matter 2.5 | -+------+------------------------+ \ No newline at end of file ++------+------------------------+ + +Other values for emissions indicate "Dummy" emissions that are merely used to limit the activity of certain technologies. +Examples are: +HO: hydro power +SO: solar pv +WI: wind power +WO: In the model version in this repo it is not included, but in other versions it is the limit for offshore wind, if so WI covers only onshore wind. From 928908f1c2468e7b701d816cc9ec2e3d7a6de662 Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:36:52 +0200 Subject: [PATCH 04/11] Update technologies.rst --- docs/technologies.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/technologies.rst b/docs/technologies.rst index e8c6268..f05d0ee 100644 --- a/docs/technologies.rst +++ b/docs/technologies.rst @@ -35,4 +35,8 @@ Technologies | UT | Utility PV | +------+-----------------------------------------------+ | WV | Wave power | -+------+-----------------------------------------------+ \ No newline at end of file ++------+-----------------------------------------------+ +| RF | Refinery | ++------+-----------------------------------------------+ +| 00 | unspecified | ++------+-----------------------------------------------+ From ffe2beeda146b028c1442b2c25f9c0693242148b Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:44:22 +0200 Subject: [PATCH 05/11] Update emissions.rst including "Dummy"-emissions --- docs/emissions.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/emissions.rst b/docs/emissions.rst index db3a812..2114d70 100644 --- a/docs/emissions.rst +++ b/docs/emissions.rst @@ -9,9 +9,16 @@ Emissions | PM25 | Particulate matter 2.5 | +------+------------------------+ -Other values for emissions indicate "Dummy" emissions that are merely used to limit the activity of certain technologies. -Examples are: -HO: hydro power -SO: solar pv -WI: wind power -WO: In the model version in this repo it is not included, but in other versions it is the limit for offshore wind, if so WI covers only onshore wind. +Other values for emissions indicate "Dummy"-emissions that are merely used to limit the activity of certain technologies. + ++------+------------------------------------+ +| Code | "Dummy"-emission | ++======+====================================+ +| HO | Hydro power | ++------+------------------------------------+ +| SO | Solar pv | ++------+------------------------------------+ +| WI | Wind power (or only onshore wind) | ++------+------------------------------------+ +| WO | Wind offshore (not in this repo) | ++------+------------------------------------+ From 3bb8c9b5f650e335e0c68c8399021a31e67d980e Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 14:46:19 +0200 Subject: [PATCH 06/11] Update technologies.rst Including country codes in tech codes --- docs/technologies.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/technologies.rst b/docs/technologies.rst index f05d0ee..9a02f80 100644 --- a/docs/technologies.rst +++ b/docs/technologies.rst @@ -40,3 +40,6 @@ Technologies +------+-----------------------------------------------+ | 00 | unspecified | +------+-----------------------------------------------+ + +Technology codes can also contain the values of country codes, for example if they represent interconnecters. + From b53cb6907bcd9e1aff21d806a36351cb19217eb6 Mon Sep 17 00:00:00 2001 From: Timon Renzelmann Date: Thu, 12 Oct 2023 14:48:48 +0200 Subject: [PATCH 07/11] code decipherer --- code_decipherer/code_decipherer.py | 278 +++++++++++++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 code_decipherer/code_decipherer.py diff --git a/code_decipherer/code_decipherer.py b/code_decipherer/code_decipherer.py new file mode 100644 index 0000000..8a64210 --- /dev/null +++ b/code_decipherer/code_decipherer.py @@ -0,0 +1,278 @@ +""" +Script: Code Decoder and CSV Modifier + +Description: +This Python script decodes technology and fuel codes. +The main function creates a new or updates a CSV file with appended decoded descriptions in a new column. + +Functions: +1. decode_tech(tech_code): Decodes a single 9/-digit technology code. +2. decode_fuel(fuel_code): Decodes a single 4-digit fuel code. +3. add_code_descriptions_to_csv(input_csv_filename, output_csv_filename=None): creates a new csv file including a new column with code descriptions. + +Example Usage: +input_csv_filename = 'input_data\\WP1_NetZero\\data\\FUEL.csv' +output_csv_filename = 'output_codes_fuel.csv' +add_code_descriptions_to_csv(input_csv_filename, output_csv_filename) + +Author: Timon Renzelmann +""" + +import sys +import pandas as pd + +# Define code mappings +country_codes = { + 'AT': 'Austria', + 'BE': 'Belgium', + 'BG': 'Bulgaria', + 'CH': 'Switzerland', + 'CY': 'Cyprus', + 'CZ': 'Czech Republic', + 'DE': 'Germany', + 'DK': 'Denmark', + 'EE': 'Estonia', + 'ES': 'Spain', + 'FI': 'Finland', + 'FR': 'France', + 'GR': 'Greece', + 'HR': 'Croatia', + 'HU': 'Hungary', + 'IE': 'Ireland', + 'IT': 'Italy', + 'LT': 'Lithuania', + 'LU': 'Luxembourg', + 'LV': 'Latvia', + 'MT': 'Malta', + 'NL': 'the Netherlands', + 'NO': 'Norway', + 'PL': 'Poland', + 'PT': 'Portugal', + 'RO': 'Romania', + 'SE': 'Sweden', + 'SI': 'Slovenia', + 'SK': 'Slovakia', + 'UK': 'United Kingdom' +} + + +commodity_codes = { + 'BF': 'Biofuel', + 'BM': 'Biomass', + 'CO': 'Coal', + 'EL': 'Electricity', + 'E1': 'Electricity 1', + 'E2': 'Electricity 2', + 'GO': 'Geothermal', + 'HF': 'Heavy fuel oil', + 'HY': 'Hydro', + 'NG': 'Natural gas', + 'NU': 'Nuclear', + 'OC': 'Ocean', + 'OI': 'Oil', + 'OS': 'Oil Shale', + 'SO': 'Sun', + 'UR': 'Uranium', + 'WS': 'Waste', + 'WI': 'Wind', +} + +technology_codes = { + 'CC': 'Combined cycle', + 'CH': 'Combined heat and power', + 'CS': 'Carbon Capture and Storage', + 'CV': 'Conventional', + 'DI': 'Distributed PV', + 'DM': 'Dam', + 'DS': 'Pumped Storage', + 'FC': 'Fuel cell', + 'GC': 'Gas cycle', + 'G2': 'Generation 2', + 'G3': 'Generation 3', + 'HP': 'Internal combustion engine with heat recovery', + 'OF': 'Offshore', + 'ON': 'Onshore', + 'ST': 'Steam cycle', + 'UT': 'Utility PV', + 'WV': 'Wave power', + 'RF': 'Refinery', + '00': 'unspecified', +} + +energy_level_codes = { + 'P': 'Primary energy commodity', + 'F': 'Final electricity', + 'I': 'Import technology', + 'X': 'Extraction or generation technology', +} + +age_codes = { + 'H': 'Historic', + 'N': 'New', +} + +def decode_tech(tech_code): + """Decodes a single technology code of 9 digits. + + Args: + tech_code (str): The technology code to decode. + + Returns: + str: The decoded description of the technology code. + """ + if len(tech_code) != 9: + return "Error: Code length must be 9" + + country = tech_code[0:2] + commodity = tech_code[2:4] + technology = tech_code[4:6] + energy_level = tech_code[6] + age = tech_code[7] + size = tech_code[8] + + country_description = country_codes.get(country, f'Unknown country') + commodity_description = commodity_codes.get(commodity, f'Unknown commodity') + if technology in technology_codes: + technology_description = technology_codes[technology] + elif technology in country_codes: + technology_description = f'connected to {country_codes[technology]}' + else: + technology_description = 'Unknown technology' + energy_level_description = energy_level_codes.get(energy_level, f'Unknown energy level') + age_description = age_codes.get(age, f'Unknown age') + size_description = f'size {size}' + + full_description = ( + f"{country_description} ({country})| " + f"{commodity_description} ({commodity})| " + f"{technology_description} ({technology})| " + f"{energy_level_description} ({energy_level})| " + f"{age_description} ({age})| " + f"{size_description}" + ) + + return full_description + +def decode_fuel(fuel_code): + """Decodes a single fuel code of 4 digits. + + Args: + fuel_code (str): The fuel code to decode. + + Returns: + str: The decoded description of the fuel code. + """ + if len(fuel_code) != 4: + return "Error: Code length must be 4" + + country = fuel_code[0:2] + commodity = fuel_code[2:4] + + country_description = country_codes.get(country, f'Unknown country') + commodity_description = commodity_codes.get(commodity, f'Unknown commodity') + + full_description = ( + f"{country_description} ({country})| " + f"{commodity_description} ({commodity})" + ) + + return full_description + +def decode_code(code): + """Decodes a single code of 4 or 9 digits. + + Args: + code (str): The code to decode. + + Returns: + str: The decoded description of the code. + """ + if len(code) == 9: + return decode_tech(code) + elif len(code) == 4: + return decode_fuel(code) + else: + return "Error: Code length must be 4 or 9" + +def add_code_descriptions_to_csv(input_csv_filename, output_csv_filename=None): + """Modifies a CSV file by adding a ' #codedescription' column. + + This function reads a CSV file containing codes in one column, decodes each + code using the decode_tech function for 9 digit codes and decode_fuel for 4-digit codes, + and adds a new column to the CSV file with the decoded descriptions in the format ' #codedescription'. + It overrides the input CSV file with the modified data if the output file name is not provided + or if it's the same as the input file name. + + Args: + input_csv_filename (str): The name of the input CSV file. + output_csv_filename (str, optional): The name of the output CSV file. + + Returns: + None + """ + try: + # Read the input CSV file + df = pd.read_csv(input_csv_filename, header=None) + + # Store the existing header + header = df.iloc[0].tolist() + header.append("description") + df1 = pd.DataFrame([header]) + + # Remove the header from the DataFrame + df = df.iloc[1:] + + # Apply the decode_tech or decode_fuel function based on code length and add a new column + df[1] = df.apply(lambda row: f'#{decode_tech(row[0])}' if len(row[0]) == 9 and not pd.isna(row[0]) + else (f' #{decode_fuel(row[0])}' if len(row[0]) == 4 and not pd.isna(row[0]) + else ''), axis=1) + #create new dataframe + df = pd.concat([df1, df], ignore_index=True) + + if output_csv_filename is None or output_csv_filename == input_csv_filename: + while True: + overwrite_input = input(f"Overwrite '{input_csv_filename}'? (yes/no): ").strip().lower() + if overwrite_input == 'yes': + # Save the modified DataFrame back to the input CSV file, overriding it + df.to_csv(input_csv_filename, index=False, header=False) + print(f"Code descriptions added to '{input_csv_filename}'.") + break + elif overwrite_input == 'no': + break + else: + print("Please enter 'yes' or 'no'.") + + else: + # Save the modified DataFrame to the output CSV file + df.to_csv(output_csv_filename, index=False, header=False) + print(f"Code descriptions saved to '{output_csv_filename}'.") + + except Exception as e: + print(f"An error occurred: {e}") + + +# Add a new functions to handle decoding fuels and tech codes from the command line +def decode_fuel_from_cli(): + if len(sys.argv) != 3: + print("Usage: python script.py decode_fuel fuel_code") + else: + fuel_code = sys.argv[2] + result = decode_fuel(fuel_code) + print(result) + +def decode_tech_from_cli(): + if len(sys.argv) != 3: + print("Usage: python script.py decode_tech fuel_code") + else: + tech_code = sys.argv[2] + result = decode_tech(tech_code) + print(result) + +if __name__ == '__main__': + if len(sys.argv) < 2 or len(sys.argv) > 3: + print("Usage: python script.py input_file [output_file]") + else: + input_file = sys.argv[1] + output_file = sys.argv[2] if len(sys.argv) > 2 else None + add_code_descriptions_to_csv(input_file, output_file) + From d9b54af6598ec81ae605f357df9df106f0a25d76 Mon Sep 17 00:00:00 2001 From: Timon Renzelmann Date: Thu, 12 Oct 2023 14:52:34 +0200 Subject: [PATCH 08/11] minor improvements for clarification --- code_decipherer/code_decipherer.py | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/code_decipherer/code_decipherer.py b/code_decipherer/code_decipherer.py index 8a64210..656ff1c 100644 --- a/code_decipherer/code_decipherer.py +++ b/code_decipherer/code_decipherer.py @@ -6,15 +6,20 @@ The main function creates a new or updates a CSV file with appended decoded descriptions in a new column. Functions: -1. decode_tech(tech_code): Decodes a single 9/-digit technology code. -2. decode_fuel(fuel_code): Decodes a single 4-digit fuel code. +1. decode_code(code): decodes a single code of 4 or 9 digits for an emission or a technology. 3. add_code_descriptions_to_csv(input_csv_filename, output_csv_filename=None): creates a new csv file including a new column with code descriptions. -Example Usage: -input_csv_filename = 'input_data\\WP1_NetZero\\data\\FUEL.csv' -output_csv_filename = 'output_codes_fuel.csv' +Example Usages: +```python +code = 'DEBFCCH1' +print(decode_code(code)) +# Output: Germany (DE)| Biofuel (BF)| Combined cycle (CC)| Primary energy commodity (P)| age 1| size 1 +``` +or +```python add_code_descriptions_to_csv(input_csv_filename, output_csv_filename) - +# Output: Code descriptions saved to 'output_codes_fuel.csv'. +``` Author: Timon Renzelmann """ From 326e65e7af7c217cb850abb28593ffb96fa42921 Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 15:00:06 +0200 Subject: [PATCH 09/11] Update README.rst --- README.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 0b983b7..302b91b 100644 --- a/README.rst +++ b/README.rst @@ -45,6 +45,7 @@ Documentation ------------- A more detailed documentation of OSeMBE can be found here: https://osembe.readthedocs.io/en/latest/ +The OSEMBE codes of the different sets can be found in the documentation. The codes can also be decoded using the script code_decipherer. Licensing --------- @@ -56,4 +57,4 @@ Citation If you wish to use, extend or otherwise build upon the work contained within this repository, you are welcome to do so, provided you abide by the terms of the licenses detailed above. -Please cite this work using the **DOI** indicated above. \ No newline at end of file +Please cite this work using the **DOI** indicated above. From cbd5675c3b810a1dc0c1a6761e9b55b91920072e Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 15:01:07 +0200 Subject: [PATCH 10/11] Update README.rst --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 302b91b..3a47893 100644 --- a/README.rst +++ b/README.rst @@ -45,7 +45,7 @@ Documentation ------------- A more detailed documentation of OSeMBE can be found here: https://osembe.readthedocs.io/en/latest/ -The OSEMBE codes of the different sets can be found in the documentation. The codes can also be decoded using the script code_decipherer. +The OSEMBE naming convention for the codes of the different sets can be found in the documentation. The codes can also be decoded using the script code_decipherer. Licensing --------- From 7257ede445a1330db3d0bbd896239944288330e8 Mon Sep 17 00:00:00 2001 From: Timon-R <145573860+Timon-R@users.noreply.github.com> Date: Thu, 12 Oct 2023 15:01:54 +0200 Subject: [PATCH 11/11] Update README.rst adding the code_decipherer script --- README.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/README.rst b/README.rst index 3a47893..cc6d66e 100644 --- a/README.rst +++ b/README.rst @@ -45,6 +45,7 @@ Documentation ------------- A more detailed documentation of OSeMBE can be found here: https://osembe.readthedocs.io/en/latest/ + The OSEMBE naming convention for the codes of the different sets can be found in the documentation. The codes can also be decoded using the script code_decipherer. Licensing