To solve the oversized Airflow scheduler logs issue, a new DAG was introduced in my pull request.
The new DAG needs two environment variables to work. However, when you import your variables manually, their is_encrypted attributes are automatically set to True. Whenever you restart the container, a new Encryption Key is generated. Thus the old ones got Invalid.
So that, we should set AIRFLOW__CORE__FERNET_KEY Environment Variable in docker-compose.yml explicitly. Such as:
environment:
- LOAD_EX=n
- EXECUTOR=Local
- AIRFLOW__CORE__FERNET_KEY='81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs='