Ce 2642 adding s3 bucket for mwaa etl scripts

[LBH-wgreeff]

Adding new s3 bucket for ETL Scripts

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Tian-2017]

@LBH-wgreeff Looks good. Ready to approve, but I suggest renaming mwaa_etl_scripts_bucket to ecs_etl_scripts_bucket. Since most of the ETL scripts are in the airflow/dags folder, Only the etl scripts requiring ECS compute resources should be placed here. Using mwaa_etl_scripts_bucket might confuse users, making them unsure where to store their ETL scripts.

Additionally, I will rename the etl_scripts folder in the dap-airflow repo to ecs_etl_scripts shortly. Apologies for the inconvenience.

[timburke-hackit]

I'm inclined to go a step further and say the name should be service agnostic. The logical grouping is the task / workflow not the service the script is going to run on in my mind.

[Tian-2017]

Hey Tim, do you mean keeping the name of the etl_scripts folder in the airflow repo the same as before and renaming the bucket to etl_scripts_bucket instead? @timburke-hackit

[timburke-hackit]

I meant the bucket name - I can quite easily envisage a world where either:

• we move compute platform - for example to kubernetes, or if Airflow sort multi-tenancy environments in v3
• workflows that require or use more than one type of compute - so a combination of ECS tasks, Glue / EMR jobs etc

I think, to your point about not confusing users by how things are named, it's much easier to have a series of scripts that are in a structure similar to:

etl_scripts/<department>/<workflow>/some_python_task.py
etl_scripts/<department>/<workflow>/some_spark_task.py
etl_scripts/<department>/<workflow>/sql/some_bit_of_sql.sql
...

Rather than having to sync them all to different buckets. This is particularly true while people are manually uploading files to stg for testing.

[Tian-2017]

@timburke-hackit sounds good. What you’ve said makes sense to me

[LBH-wgreeff]

Just to add, the Python script, you can specify the bucket and an optional object within the bucket, aka folder in the bucket.