Standardize unioning macros to use fivetran_utils versions

CHANGELOG.md

@@ -1,3 +1,21 @@
+# dbt_linkedin v1.3.0
+
+[PR #63](https://github.com/fivetran/dbt_linkedin/pull/63) includes the following updates:
+
+## Schema/Data Changes (--full-refresh required after upgrading)
+**1 total change • 1 possible breaking change**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | --- | --- | ----- |
+| All models | `source_relation` column (when using a single LinkedIn Ads schema) | Empty string (`''`) | `<database>.<schema>` |  |
+
+## Feature Updates
+- Introduces the new (recommended) `linkedin_ads_sources` variable for more robust union data configuration. The old`linkedin_ads_union_schemas` and `linkedin_ads_union_databases` variables will still be supported. See the [README](https://github.com/fivetran/dbt_linkedin/tree/main#define-database-and-schema-variables) for specific details.
+
+## Under the Hood
+- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_linkedin/#source-casing-for-case-sensitive-destinations) section of the README for details.
+- Introduces `fivetran_utils.partition_by_source_relation` to conditionally include `source_relation` in partition clauses only when multiplesources are configured.
+
 # dbt_linkedin v1.2.0
 
 [PR #58](https://github.com/fivetran/dbt_linkedin/pull/58) includes the following updates:

README.md

@@ -75,7 +75,7 @@ Include the following Linkedin Ads package version in your `packages.yml` file:
 # packages.yml
 packages:
   - package: fivetran/linkedin
-    version: [">=1.2.0", "<1.3.0"]
+    version: [">=1.3.0", "<1.4.0"]
 ```
 
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/linkedin_source` in your `packages.yml` since this package has been deprecated.
@@ -89,28 +89,42 @@ dispatch:
 ```
 
 ### Define database and schema variables
-By default, this package runs using your destination and the `linkedin_ads` schema. If this is not where your Linkedin Ad Analytics data is (for example, if your Linkedin schema is named `linkedin_ads_fivetran`), add the following configuration to your root `dbt_project.yml` file:
+#### Option A: Single connection
+By default, this package runs using your destination and the `linkedin` schema. If this is not where your Linkedin data is (for example, if your Linkedin schema is named `linkedin_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
-# dbt_project.yml
 vars:
+    linkedin_ads_database: your_destination_name
     linkedin_ads_schema: your_schema_name
-    linkedin_ads_database: your_destination_name 
 ```
 
-### (Optional) Additional configurations
-#### Union multiple connections
-If you have multiple linkedin connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `linkedin_ads_union_schemas` OR `linkedin_ads_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
+#### Option B: Union multiple connections
+If you have multiple Linkedin Ads connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.
+
+To use this functionality, you will need to set the `linkedin_ads_sources` variable in your root `dbt_project.yml` file:
 
 ```yml
+# dbt_project.yml
+
 vars:
-    linkedin_ads_union_schemas: ['linkedin_usa','linkedin_canada'] # use this if the data is in different schemas/datasets of the same database/project
-    linkedin_ads_union_databases: ['linkedin_usa','linkedin_canada'] # use this if the data is in different databases/projects but uses the same schema name
+  linkedin:
+    linkedin_ads_sources:
+      - database: connection_1_destination_name # Required
+        schema: connection_1_schema_name # Required
+        name: connection_1_source_name # Required only if following the step in the following subsection
+
+      - database: connection_2_destination_name
+        schema: connection_2_schema_name
+        name: connection_2_source_name
 ```
-> NOTE: The native `src_linkedin.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `src_linkedin.yml`.
 
-To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
+> Previous versions of this package employed two separate, mutually exclusive variables for unioning: `linkedin_ads_union_schemas` and `linkedin_ads_union_databases`. While these variables are still supported, `linkedin_ads_sources` is the recommended variable to configure.
+
+#### Optional: Incorporate unioned sources into DAG
+
+If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Linkedin Ads connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_linkedin/blob/main/models/staging/src_linkedin.yml). Set the variable `has_defined_sources: true` under the Linkedin namespace in your `dbt_project.yml`. Otherwise, your Linkedin Ads connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.
 
+### (Optional) Additional configurations
 #### Disable Country and Region Reports
 This package leverages the `geo`, `monthly_ad_analytics_by_member_country`, and `monthly_ad_analytics_by_member_region` tables to help report on campaign performance by country and region. However, if you are not actively syncing these reports from your LinkedIn Ads connection, you may disable relevant transformations by adding the following variable configuration to your root `dbt_project.yml` file:
 ```yml
@@ -206,6 +220,14 @@ vars:
     linkedin_ads_<default_source_table_name>_identifier: your_table_name 
 ```
 
+#### Source casing for case-sensitive destinations
+By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:
+
+```yml
+vars:
+    fivetran_using_source_casing: true
+```
+
 ### (Optional) Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand for more details</summary>
 

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'linkedin'
-version: '1.2.0'
+version: '1.3.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 

integration_tests/ci/test_scenarios.yml

@@ -25,3 +25,9 @@ test_scenarios:
       linkedin_ads__using_monthly_ad_analytics_by_member_country: false
       linkedin_ads__using_monthly_ad_analytics_by_member_region: false
     include_incremental: false
+
+  - name: "MDLS"
+    vars:
+      fivetran_using_source_casing: true
+    include_incremental: false
+    include_dbt_seed: true

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'linkedin_integration_tests'
-version: '1.2.0'
+version: '1.3.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -30,6 +30,7 @@ models:
 seeds:
   +docs:
     show: false
+  +quote_columns: "{{ target.type == 'redshift' or var('fivetran_using_source_casing', false) }}"
   linkedin_integration_tests:
     linkedin_creative_history_data:
       +column_types:
@@ -67,4 +68,4 @@ dispatch:
 
 flags:
   send_anonymous_usage_stats: False
-  use_colors: True
+  use_colors: True

models/staging/src_linkedin.yml

@@ -9,7 +9,7 @@ sources:
 
     config:
       loaded_at_field: _fivetran_synced
-      enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) }}"
+      enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) and var('linkedin_ads_sources', []) == [] }}"
       freshness: 
         warn_after: {count: 48, period: hour}
         error_after: {count: 168, period: hour}
@@ -315,7 +315,7 @@ sources:
         description: Contains geographic data for LinkedIn Ads, including IDs and names of locations.
         identifier: "{{ var('linkedin_ads_geo_identifier', 'geo') }}"
         config:
-          enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) and var('linkedin_ads__using_geo', true) }}"
+          enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) and var('linkedin_ads__using_geo', true) and var('linkedin_ads_sources', []) == [] }}"
         columns:
           - name: id
             description: Unique identifier for the geographic entity, used for joins in reporting tables.
@@ -326,7 +326,7 @@ sources:
         description: Monthly LinkedIn Ads performance metrics aggregated by member country and campaign.
         identifier: "{{ var('linkedin_ads_monthly_ad_analytics_by_member_country_identifier', 'monthly_ad_analytics_by_member_country') }}"
         config:
-          enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) and var('linkedin_ads__using_monthly_ad_analytics_by_member_country', true) }}"
+          enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) and var('linkedin_ads__using_monthly_ad_analytics_by_member_country', true) and var('linkedin_ads_sources', []) == [] }}"
         columns:
           - name: campaign_id
             description: Identifier of the LinkedIn Ads campaign.
@@ -353,7 +353,7 @@ sources:
         description: Monthly LinkedIn Ads performance metrics aggregated by member region and campaign.
         identifier: "{{ var('linkedin_ads_monthly_ad_analytics_by_member_region_identifier', 'monthly_ad_analytics_by_member_region') }}"
         config:
-          enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) and var('linkedin_ads__using_monthly_ad_analytics_by_member_region', true) }}"
+          enabled: "{{ var('ad_reporting__linkedin_ads_enabled', true) and var('linkedin_ads__using_monthly_ad_analytics_by_member_region', true) and var('linkedin_ads_sources', []) == [] }}"
         columns:
           - name: campaign_id
             description: Identifier of the LinkedIn Ads campaign.

models/staging/stg_linkedin_ads__account_history.sql

@@ -15,10 +15,7 @@ with base as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 
@@ -34,7 +31,7 @@ with base as (
         type,
         cast(last_modified_time as {{ dbt.type_timestamp() }}) as last_modified_at,
         cast(created_time as {{ dbt.type_timestamp() }}) as created_at,
-        row_number() over (partition by id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_time desc) = 1 as is_latest_version
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='linkedin_ads') }} order by last_modified_time desc) = 1 as is_latest_version
 
     from macro
 

models/staging/stg_linkedin_ads__ad_analytics_by_campaign.sql

@@ -16,10 +16,7 @@ macro as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 ),
@@ -50,4 +47,4 @@ fields as (
 )
 
 select *
-from fields
+from fields

models/staging/stg_linkedin_ads__ad_analytics_by_creative.sql

@@ -15,10 +15,7 @@ with base as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 
@@ -49,4 +46,4 @@ with base as (
 )
 
 select *
-from fields
+from fields

models/staging/stg_linkedin_ads__campaign_group_history.sql

@@ -15,10 +15,7 @@ with base as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 
@@ -35,7 +32,7 @@ with base as (
         cast(run_schedule_end as {{ dbt.type_timestamp() }}) as run_schedule_end_at,
         cast(last_modified_time as {{ dbt.type_timestamp() }}) as last_modified_at,
         cast(created_time as {{ dbt.type_timestamp() }}) as created_at,
-        row_number() over (partition by id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_time desc) = 1 as is_latest_version
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='linkedin_ads') }} order by last_modified_time desc) = 1 as is_latest_version
 
     from macro
 

models/staging/stg_linkedin_ads__campaign_history.sql

@@ -15,10 +15,7 @@ with base as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 
@@ -50,7 +47,7 @@ with base as (
         cast(run_schedule_end as {{ dbt.type_timestamp() }}) as run_schedule_end_at,
         cast(last_modified_time as {{ dbt.type_timestamp() }}) as last_modified_at,
         cast(created_time as {{ dbt.type_timestamp() }}) as created_at,
-        row_number() over (partition by id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_time desc) = 1 as is_latest_version
+        row_number() over (partition by id {{ fivetran_utils.partition_by_source_relation(package_name='linkedin_ads') }} order by last_modified_time desc) = 1 as is_latest_version
 
     from macro
 

models/staging/stg_linkedin_ads__creative_history.sql

@@ -15,10 +15,7 @@ with base as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 
@@ -43,7 +40,7 @@ with base as (
 
     select 
         *,
-        row_number() over (partition by creative_id {{ ', source_relation' if (var('linkedin_ads_union_schemas', []) or var('linkedin_ads_union_databases', []) | length > 1) }} order by last_modified_at desc) = 1 as is_latest_version,
+        row_number() over (partition by creative_id {{ fivetran_utils.partition_by_source_relation(package_name='linkedin_ads') }} order by last_modified_at desc) = 1 as is_latest_version,
         {{ dbt.split_part('click_uri', "'?'", 1) }} as base_url,
         {{ dbt_utils.get_url_host('click_uri') }} as url_host,
         '/' || {{ dbt_utils.get_url_path('click_uri') }} as url_path,
@@ -57,5 +54,4 @@ with base as (
 )
 
 select *
-from url_fields
-
+from url_fields

models/staging/stg_linkedin_ads__geo.sql

@@ -16,10 +16,7 @@ macro as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 ),
@@ -34,4 +31,4 @@ fields as (
 )
 
 select *
-from fields
+from fields

models/staging/stg_linkedin_ads__monthly_ad_analytics_by_country.sql

@@ -17,10 +17,7 @@ macro as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 ),
@@ -52,4 +49,4 @@ fields as (
 )
 
 select *
-from fields
+from fields

models/staging/stg_linkedin_ads__monthly_ad_analytics_by_region.sql

@@ -16,10 +16,7 @@ macro as (
             )
         }}
 
-        {{ fivetran_utils.source_relation(
-            union_schema_variable='linkedin_ads_union_schemas', 
-            union_database_variable='linkedin_ads_union_databases') 
-        }}
+        {{ fivetran_utils.apply_source_relation(package_name='linkedin_ads') }}
 
     from base
 ),
@@ -51,4 +48,4 @@ fields as (
 )
 
 select *
-from fields
+from fields

models/staging/tmp/stg_linkedin_ads__account_history_tmp.sql

@@ -1,5 +1,7 @@
 {{ config(enabled=var('ad_reporting__linkedin_ads_enabled', True)) }}
 
+{% if var('linkedin_ads_union_schemas', []) | length > 0 or var('linkedin_ads_union_databases', []) | length > 0 %}
+
 {{
     fivetran_utils.union_data(
         table_identifier='account_history', 
@@ -11,4 +13,16 @@
         union_schema_variable='linkedin_ads_union_schemas',
         union_database_variable='linkedin_ads_union_databases'
     )
-}}
+}}
+
+{% else %}
+
+{{
+    fivetran_utils.union_connections(
+        connection_dictionary='linkedin_ads_sources',
+        single_source_name='linkedin_ads',
+        single_table_name='account_history'
+    )
+}}
+
+{% endif %}