New RotateAirflowLogs DAG for deleting empty log directories

[jabortell]

Links:

• Design: STAD-198 Add log rotation to Airflow logs
• Jira: STAD-198 Add log rotation to Airflow logs

Testing instructions:

In a stadium or stadium-like project, import the new DAG and instantiate it like:

from ea_airflow_util import RotateLogsDAG
globals()[rotate_logs_dag.dag.dag_id] = rotate_logs_dag.dag

This DAG support configuration via airflow_config.yml under the key rotate_logs_dag. For example, if your logs root directory is located someplace besides $AIRFLOW_HOME/logs, you can configure it like:

rotate_logs_dag:
  schedule_interval: null
  default_args: *default_task_args
  logs_dir: /some/other/airflow/logs

then update your instantiation code to load the config:

from util import io_helpers

# Or wherever your config is:
airflow_configs_file = "/home/airflow/airflow/configs/airflow_config.yml"

AIRFLOW_CONFIGS = io_helpers.safe_load_yaml(configs_dir, airflow_configs_file)
dag_args = AIRFLOW_CONFIGS.get("rotate_logs_dag", {})

rotate_logs_dag = RotateLogsDAG(dag_id="rotate_airflow_logs", **dag_args)
globals()[rotate_logs_dag.dag.dag_id] = rotate_logs_dag.dag

Finally, when you run the DAG, confirm that it runs successfully, that empty sub-directories in the logging directory are deleted, and non-empty directories are not deleted. The DAG uses os.rmdir(), which will throw an exception if called on a non-empty directory, so there should not be a risk of deleting non-empty directories.

[jabortell]

@rlittle08, Fyi @ea-mtenhoor commented in the Jira about wanting to discuss the scope of the design further.

[jabortell]

Pushed a change to scan for empty files, via the meeting/notes from the Jira. I also updated the directory scanner to be a loop, so that it handles nested empty directories.