Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 6 additions & 12 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ on:
- '**.py'
- 'pyproject.toml'
- 'uv.lock'
- '**/ci.yaml'
- '!airflow_dbt_python/__version__.py'
tags:
- "v*"
Expand All @@ -17,6 +18,7 @@ on:
- '**.py'
- 'pyproject.toml'
- 'uv.lock'
- '**/ci.yaml'
- '!airflow_dbt_python/__version__.py'

jobs:
Expand All @@ -31,24 +33,16 @@ jobs:
- '3.11'
- '3.10'
airflow-version:
- '3.1.1'
- '3.1.5'
- '3.0.6'
- '2.10.4'
dbt-version:
- '1.10.13'
- '1.9.10'
- '1.11.2'
- '1.10.17'
exclude:
# Airflow added 3.13 support in >=3.1
- airflow-version: '3.0.6'
python-version: '3.13'

- airflow-version: '2.10.4'
python-version: '3.13'

# Dbt added 3.13 support in >=1.10
- dbt-version: '1.9.10'
python-version: '3.13'

runs-on: ubuntu-latest
steps:
- name: Harden Runner
Expand Down Expand Up @@ -105,7 +99,7 @@ jobs:

- name: Static type checking with mypy
# We only run mypy on the latest supported versions of Airflow & dbt,
if: matrix.python-version == '3.13' && matrix.airflow-version == '3.1.1' && matrix.dbt-version == '1.10.13'
if: matrix.python-version == '3.13' && matrix.airflow-version == '3.1.1' && matrix.dbt-version == '1.11.12'
run: uv run mypy .

- name: Code formatting with ruff
Expand Down
22 changes: 11 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[![Test coverage](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/tomasfarias/81ef37701aa088d18db8a58ce07c79c7/raw/covbadge.json)](https://github.com/tomasfarias/airflow-dbt-python/actions)
[![Documentation](https://readthedocs.org/projects/airflow-dbt-python/badge/?version=latest)](https://airflow-dbt-python.readthedocs.io/en/latest/?badge=latest)

A collection of [Airflow](https://airflow.apache.org/) operators, hooks, and utilities to execute [`dbt`](https://pypi.org/project/dbt-core/) commands.
A collection of [*Airflow*](https://airflow.apache.org/) operators, hooks, and utilities to execute [*dbt*](https://pypi.org/project/dbt-core/) commands.

Read the [documentation](https://airflow-dbt-python.readthedocs.io) for examples, installation instructions, and more details.

Expand All @@ -16,20 +16,20 @@ Read the [documentation](https://airflow-dbt-python.readthedocs.io) for examples

Before using *airflow-dbt-python*, ensure you meet the following requirements:
* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.8 or later.
* An Airflow environment using version 2.8 or later.
* An *Airflow* deployment using version 3.0 or later.

* If using any managed service, like AWS MWAA or GCP Cloud Composer 2/3, ensure your environment is created with a supported version of Airflow.
* If self-hosting, Airflow installation instructions can be found in their [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).
* If using any managed *Airflow* service, like [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) or [GCP Cloud Composer](https://cloud.google.com/composer), ensure your environment is created with a supported version of *Airflow*.
* If self-hosting, *Airflow* installation instructions can be found in their [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html).

* Running Python 3.9 or later in your Airflow environment.
* Python 3.10 or later.

> **Warning**
>
> Even though we don't impose any upper limits on versions of Airflow and *dbt*, it's possible that new versions are not supported immediately after release, particularly for *dbt*. We recommend testing the latest versions before upgrading and [reporting any issues](https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose).
> New versions of *Airflow* and *dbt* may introduce breaking changes. We recommend testing any new versions of *Airflow* and *dbt* before upgrading production systems; Please [report any issues](https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose) that may arise during testing so they can be addressed.

> **Note**
>
> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) and [GCP Cloud Composer 2/3](https://aws.amazon.com/managed-workflows-for-apache-airflow/).
> We only test *airflow-dbt-python* against a limited set of versions of *Airflow* and *dbt*, and try to keep up with the latest releases. For *Airflow*, our policy is to cover with tests the latest release of *Airflow*, the latest version available in [GCP Cloud Composer](https://docs.cloud.google.com/composer/docs/composer-versions), and the latest version available in [AWS MWAA](https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-versions). For *dbt*, our policy is to cover the last two minor versions.

## From PyPI

Expand All @@ -45,7 +45,7 @@ As a convenience, some *dbt* adapters can be installed by specifying extras. For
pip install airflow-dbt-python[redshift]
```

## From this repo
## Building from source

*airflow-dbt-python* can also be built from source by cloning this GitHub repository:

Expand All @@ -54,10 +54,10 @@ git clone https://github.com/tomasfarias/airflow-dbt-python.git
cd airflow-dbt-python
```

And installing with *uv*:
And build with *uv*:

``` shell
uv pip install .
uv build
```

## In AWS MWAA
Expand All @@ -70,7 +70,7 @@ Read the [documentation](https://airflow-dbt-python.readthedocs.io/en/latest/get

Add *airflow-dbt-python* to your PyPI packages list.

Read the [documentation](https://cloud.google.com/composer/docs/composer-2/install-python-dependencies#install-pypi) for more a more detailed GCP Cloud Composer 2 installation breakdown.
Refer to the [GCP Cloud Composer documentation](https://cloud.google.com/composer/docs/composer-3/install-python-dependencies#install-pypi) on how to do this.

## In other managed services

Expand Down
1 change: 1 addition & 0 deletions airflow_dbt_python/utils/configs.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,7 @@ class BaseConfig:
upgrade: bool = False

require_model_names_without_spaces: bool = False
require_ref_searches_node_package_before_root: bool = False
exclude_resource_types: list[str] = dataclasses.field(
default_factory=list, repr=False
)
Expand Down
27 changes: 17 additions & 10 deletions docs/development.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,19 +38,26 @@ The additional extras install dependencies required for testing. If testing a sp

uv pip install -e .

Support for different versions of *Airflow*
-------------------------------------------
Supported *Airflow* versions
----------------------------

*airflow-dbt-python* is tested against multiple versions of *Airflow*; as a policy, besides the latest version of Airflow, we test *airflow-dbt-python* against the latest version available in `AWS MWAA <https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-versions>`_, and the latest version available in `GCP Cloud Composer <https://docs.cloud.google.com/composer/docs/composer-versions>`_.

We are open to patches to improve backwards compatibility as long as they don't increase maintenance load significantly.

*airflow-dbt-python* supports and is tested with multiple versions of Airflow; as a general rule, besides the latest version of Airflow, we test *airflow-dbt-python* against the latest version available in `AWS MWAA <https://aws.amazon.com/managed-workflows-for-apache-airflow/>`_, which usually lags behind a few minor versions. We are open to patches to improve backwards compatibility as long as they don't increase maintenance load significantly.
If you wish to install a different version of *Airflow* for testing you may skip the *airflow-providers* extras of the previous section and use *pip* instead to install any versions of *apache-airflow* and required providers.

Supported *dbt* versions
----------------------------

If you wish to install a different version of Airflow for testing you may skip the *airflow-providers* extras of the previous section and use *pip* instead to install any versions of *apache-airflow* and required providers.
*airflow-dbt-python* is tested against multiple versions of *dbt*; as a policy, we cover the latest two minor versions with tests.

Modifying dependencies
----------------------

Apache Airflow is a package of significant size that requires a lot of dependencies. Together with *dbt-core*, it's common to find dependency conflicts all over the place. Ultimately, we allow users to figure these issues out themselves, as most of the dependency conflicts are harmless: We do not interact with the *dbt* CLI, so any conflicts with CLI-specific packages can be safely ignored, but these requirements are not optional for *dbt-core*.
*Apache Airflow* is a package of significant size that requires a lot of dependencies. Together with *dbt-core*, it's common to find dependency conflicts all over the place. Ultimately, we allow users to figure these issues out themselves, as most of the dependency conflicts are harmless: We do not interact with the *dbt* CLI, so any conflicts with CLI-specific packages can be safely ignored, but these requirements are not optional for *dbt-core*.

All being said, this presents a problem when we try to add dependencies or modify existing ones. Grabbing a constraints file from `Airflow <https://github.com/apache/airflow>`_ and adding it as an optional group in ``pyproject.toml`` can be a useful strategy.
All being said, this presents a problem when we try to add dependencies or modify existing ones. Grabbing a constraints file from `*Airflow* <https://github.com/apache/airflow>`_ and adding it as an optional group in ``pyproject.toml`` can be a useful strategy.

Pre-commit hooks
----------------
Expand Down Expand Up @@ -86,12 +93,12 @@ Tests are available for all operators, hooks, and utilities. That being said, on
.. note::
Unit tests (and *airflow-dbt-python*) assume *dbt* works correctly and do not assert the behavior of the *dbt* commands in depth.

Testing specific requirements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Requirements
^^^^^^^^^^^^

Unit tests interact with a `PostgreSQL <https://www.postgresql.org/>`_ database as a target to run dbt commands. This requires *PostgreSQL* to be installed in your local environment. Installation instructions for all major platforms can be found `here <https://www.postgresql.org/download/>`_.
Unit tests interact with a `PostgreSQL <https://www.postgresql.org/>`_ database as a target to run *dbt* commands. This requires *PostgreSQL* to be installed in your local environment. Installation instructions for all major platforms can be found `here <https://www.postgresql.org/download/>`_.

An Airflow database needs to be initialized in your local environment. This requires choosing a location for it, via the ``AIRFLOW_HOME`` environment variable. The same directory where *airflow-dbt-python* was cloned to can be used for this:
An *Airflow* database needs to be initialized in your local environment. This requires choosing a location for it, via the ``AIRFLOW_HOME`` environment variable. The same directory where *airflow-dbt-python* was cloned to can be used for this:

.. code-block:: shell

Expand Down
2 changes: 1 addition & 1 deletion docs/example_dags.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Example DAGs
This section contains a few DAGs showing off some dbt pipelines to get you going.

.. warning::
All example DAGs are tested against against ``apache-airflow==2.2.5``. Some changes, like modifying ``import`` statements or changing types, may be required for them to work in environments running other versions of Airflow.
All example DAGs are tested against a subset of *Airflow* versions. Some changes, like modifying ``import`` statements or changing types, may be required for them to work in environments running other versions of *Airflow*.

Basic DAG
^^^^^^^^^
Expand Down
47 changes: 28 additions & 19 deletions docs/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,23 +10,23 @@ Requirements

Before using *airflow-dbt-python*, ensure you meet the following requirements:
* A *dbt* project using `dbt-core <https://pypi.org/project/dbt-core/>`_ version 1.8 or later.
* An Airflow environment using version 2.8 or later.
* An *Airflow* environment using version 3.0 or later.

* If using any managed service, like AWS MWAA or GCP Cloud Composer 2/3, ensure your environment is created with a supported version of Airflow.
* If self-hosting, Airflow installation instructions can be found in their `official documentation <https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html>`_.
* If using any managed *Airflow* service, like `AWS MWAA <https://aws.amazon.com/managed-workflows-for-apache-airflow/>`_ or `GCP Cloud Composer <https://cloud.google.com/composer>`_, ensure your environment is created with a supported version of *Airflow*.
* If self-hosting, *Airflow* installation instructions can be found in their `official documentation <https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html>`_.

* Running Python 3.9 or later in your Airflow environment.
* Python 3.10 or later.

.. warning::
Even though we don't impose any upper limits on versions of Airflow and *dbt*, it's possible that new versions are not supported immediately after release, particularly for *dbt*. We recommend testing the latest versions before upgrading and `reporting any issues <https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose>`_.
New versions of *Airflow* and *dbt* may introduce breaking changes. We recommend testing any new versions of *Airflow* and *dbt* before upgrading production systems; Please `report any issues <https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose>`_ that may arise during testing so they can be addressed.

.. note::
Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by `AWS MWAA <https://aws.amazon.com/managed-workflows-for-apache-airflow/>`_ and `GCP Cloud Composer 2/3 <https://aws.amazon.com/managed-workflows-for-apache-airflow/>`_.
We only test *airflow-dbt-python* against a limited set of versions of *Airflow* and *dbt*, and try to keep up with the latest releases. For *Airflow*, our policy is to cover with tests the latest release of *Airflow*, the latest version available in `GCP Cloud Composer <https://docs.cloud.google.com/composer/docs/composer-versions>`_, and the latest version available in `AWS MWAA <https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-versions>`_. For *dbt*, our policy is to cover the last two minor versions.

Installation
------------

Your installation will vary according to your specific Airflow environment setup. These instructions cover a general approach by installing from PyPI or the GitHub repository, and how to install it in AWS MWAA. Other serviced offerings may require different steps, check the documentation of your managed service.
Your installation will vary according to your specific Airflow environment setup. These instructions cover installing from PyPI, building the GitHub repository directly, installing in AWS MWAA, and installing in GCP Cloud Composer. Other serviced offerings may require different steps, check the documentation of your managed service.

From PyPI
^^^^^^^^^
Expand Down Expand Up @@ -62,24 +62,18 @@ Building from source
git clone https://github.com/tomasfarias/airflow-dbt-python.git
cd airflow-dbt-python

And installing with *pip*:
And build with *uv*:

.. code-block:: shell

pip install .
uv build

Optionally, any *dbt* adapters can be installed by specifying extras:
In AWS MWAA
^^^^^^^^^^^

.. code-block:: shell

pip install .[postgres, redshift, bigquery, snowflake]

Installing in MWAA
^^^^^^^^^^^^^^^^^^
Add *airflow-dbt-python* to your ``requirements.txt`` file and edit your *Airflow* environment to use this new ``requirements.txt`` file, or upload it as a plugin.

*airflow-dbt-python* can be installed in an Airflow environment managed by AWS via their `Managed Workflows for Apache Airflow <https://aws.amazon.com/managed-workflows-for-apache-airflow/>`_ service.

To do so, include *airflow-dbt-python* in the *requirements.txt* file provided to MWAA, for example:
To do so, include *airflow-dbt-python* in the ``requirements.txt`` file provided to AWS MWAA, for example:

.. code-block:: shell
:caption: requirements.txt
Expand Down Expand Up @@ -110,6 +104,21 @@ The wheel file can now be added to your *plugins.zip*, and the requirements can

/usr/local/airflow/plugins/airflow_dbt_python-X.Y.Z-py3-none-any.whl

In GCP Cloud Composer
^^^^^^^^^^^^^^^^^^^^^

Add *airflow-dbt-python* to your PyPI packages list.

Refer to the `GCP Cloud Composer documentation <https://cloud.google.com/composer/docs/composer-3/install-python-dependencies#install-pypi>`_ on how to do this.

In other managed services
^^^^^^^^^^^^^^^^^^^^^^^^^

*airflow-dbt-python* should be compatible with most or all *Airflow* managed services. Consult the documentation specific to your provider.

If you notice an issue when installing *airflow-dbt-python* in a specific managed service, please open an `issue <https://github.com/tomasfarias/airflow-dbt-python/issues/new/choose>`_.


Accessing a *dbt* project
-------------------------

Expand Down
Loading