Feature/add union data

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -11,15 +11,20 @@
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
 - [ ] dbt run –full-refresh && dbt test
-- [ ] dbt run (if incremental models are present)
+- [ ] dbt run (if incremental models are present) && dbt test
 
 Before marking this PR as "ready for review" the following have been applied:
 - [ ] The appropriate issue has been linked, tagged, and properly assigned
 - [ ] All necessary documentation and version upgrades have been applied
+    <!--- Be sure to update the package version in the dbt_project.yml, integration_tests/dbt_project.yml, and README if necessary. -->
 - [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
 - [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
 Please share any and all of your validation steps:
 <!--- Provide the steps you took to validate your changes below. -->
+
+### If you had to summarize this PR in an emoji, which would it be?
+<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
+:dancer:

.github/workflows/auto-release.yml

@@ -0,0 +1,13 @@
+name: 'auto release'
+on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - main
+
+jobs:
+  call-workflow-passing-data:
+    if: github.event.pull_request.merged
+    uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
+    secrets: inherit

CHANGELOG.md

@@ -1,3 +1,15 @@
+# dbt_netsuite_source v0.10.0
+
+## 🎉 Feature Update 🎉 
+- This release supports running the package on multiple Netsuite sources at once! See the [README](https://github.com/fivetran/dbt_netsuite_source/tree/main?tab=readme-ov-file#step-4-define-database-and-schema-variables) for details on how to leverage this feature ([PR #43](https://github.com/fivetran/dbt_netsuite_source/pull/43)).
+
+## 🛠️ Under the Hood 🛠️
+- Included auto-releaser GitHub Actions workflow to automate future releases.
+- Updated the maintainer PR template to resemble the most up to date format.
+
+## Contributors:
+- [@fivetran-poonamagate](https://github.com/fivetran-poonamagate) ([PR #43](https://github.com/fivetran/dbt_netsuite_source/pull/43))
+
 # dbt_netsuite_source v0.9.0
 [PR #42](https://github.com/fivetran/dbt_netsuite_source/pull/42) includes the following update:
 ## 🚨 Breaking Changes 🚨:

README.md

@@ -89,7 +89,7 @@ If you  are **not** using the [Netsuite transformation package](https://github.c
 ```yaml
 packages:
   - package: fivetran/netsuite_source
-    version: [">=0.9.0", "<0.10.0"]
+    version: [">=0.10.0", "<0.11.0"]
 ```
 
 ## Step 3: Define Netsuite.com or Netsuite2 Source
@@ -100,15 +100,62 @@ vars:
 ```
 
 ## Step 4: Define database and schema variables
+### Option 1: Single connector 💃
 By default, this package runs using your destination and the `netsuite` schema. If this is not where your Netsuite data is (for example, if your netsuite schema is named `netsuite_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 vars:
     netsuite_database: your_destination_name
     netsuite_schema: your_schema_name 
 ```
+> **Note**: If you are running the package on one source connector, each model will have a `source_relation` column that is just an empty string.
+
+### Option 2: Union multiple connectors (Netsuite2 only) 👯
+If you have multiple Netsuite2 connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `netsuite2_union_schemas` OR `netsuite2_union_databases` variables (cannot do both, though a more flexible approach is in the works...) in your root `dbt_project.yml` file:
+
+```yml
+# dbt_project.yml
+
+vars:
+    netsuite2_union_schemas: ['netsuite2_usa','netsuite2_canada'] # use this if the data is in different schemas/datasets of the same database/project
+    netsuite2_union_databases: ['netsuite2_usa','netsuite2_canada'] # use this if the data is in different databases/projects but uses the same schema name
+```
+
+#### Recommended: Incorporate unioned sources into DAG
+By default, this package defines one single-connector source, called `netsuite2`, which will be disabled if you are unioning multiple connectors. This means that your DAG will not include your Netsuite2 sources, though the package will run successfully.
+
+To properly incorporate all of your Netsuite2 connectors into your project's DAG:
+1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_netsuite2.yml` [file](https://github.com/fivetran/dbt_netsuite_source/blob/main/models/netsuite2/src_netsuite2.yml#L15-L607).
+
+```yml
+# a .yml file in your root project
+sources:
+  - name: <name> # ex: netsuite2_usa
+    schema: <schema_name> # one of var('netsuite2_union_schemas') if unioning schemas, otherwise just 'netsuite2'
+    database: <database_name> # one of var('netsuite2_union_databases') if unioning databases, otherwise whatever DB your netsuite2 schemas all live in
+    loader: fivetran
+    loaded_at_field: _fivetran_synced
+
+    freshness: # feel free to adjust to your liking
+      warn_after: {count: 72, period: hour}
+      error_after: {count: 168, period: hour}
+
+    tables: # copy and paste from models/netsuite2/src_netsuite2.yml 
+```
+
+> **Note**: If there are source tables you do not have (see [Step 4](https://github.com/fivetran/dbt_netsuite_source?tab=readme-ov-file#step-5-disable-models-for-non-existent-sources-netsuite2-only)), you may still include them, as long as you have set the right [variables](https://github.com/fivetran/dbt_netsuite_source?tab=readme-ov-file#step-5-disable-models-for-non-existent-sources-netsuite2-only) to `False`. Otherwise, you may remove them from your source definitions.
+
+2. Set the `has_defined_sources` variable (scoped to the `netsuite_source` package) to `True`, like such:
+```yml
+# dbt_project.yml
+vars:
+  netsuite_source:
+    has_defined_sources: true
+```
 
 ## Step 5: Disable models for non-existent sources (Netsuite2 only)
+> _This step is unnecessary (but still available for use) if you are unioning multiple connectors together in the previous step. That is, the `union_data` macro we use will create completely empty staging models for sources that are not found in any of your Netsuite2 schemas/databases. However, you can still leverage the below variables if you would like to avoid this behavior._
+
 It's possible that your Netsuite connector does not sync every table that this package expects. If your syncs exclude certain tables, it is because you either don't use that feature in Netsuite or actively excluded some tables from your syncs. To disable the corresponding functionality in the package, you must add the relevant variables. By default, all variables are assumed to be true. Add variables for only the tables you would like to disable:
 ```yml
 vars:
@@ -123,7 +170,7 @@ vars:
 > To determine if a table or field is activated by a feature, access the [Records Catalog](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/article_159367781370.html).
 
 ## (Optional) Step 6: Additional configurations
-<details><summary>Expand for configurations</summary>
+<details open><summary>Expand/collapse configurations</summary>
 
 ### Passing Through Additional Fields
 This package includes all source columns defined in the macros folder. You can add more columns using our pass-through column variables. These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:
@@ -170,13 +217,14 @@ models:
     netsuite_source:
       +schema: my_new_schema_name # leave blank for just the target_schema
 ```
-
-### Change the source table references
-If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
+### Change the source table references (only if using a single connector)
+If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable (and set `use_table_name_identifer_override` if using Netsuite2):
 > IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_netsuite_source/blob/main/dbt_project.yml) variable declarations to see the expected names.
 
 ```yml
 vars:
+    use_table_name_identifer_override: true # Netsuite2 users must set this to TRUE. default = false
+
     # For all Netsuite source tables
     netsuite_<default_source_table_name>_identifier: your_table_name 
 
@@ -223,4 +271,4 @@ We highly encourage and welcome contributions to this package. Check out [this d
 # 🏪 Are there any resources available?
 - If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_netsuite_source/issues/new/choose) section to find the right avenue of support for you.
 - If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
-- Have questions or want to be part of the community discourse? Create a post in the [Fivetran community](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) and our team along with the community can join in on the discussion!
+- Have questions or want to be part of the community discourse? Create a post in the [Fivetran community](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) and our team along with the community can join in on the discussion!

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 name: 'netsuite_source'
-version: '0.9.0'
+version: '0.10.0'
 
 models:
   netsuite_source:

integration_tests/dbt_project.yml

@@ -6,6 +6,7 @@ profile: 'integration_tests'
 config-version: 2
 vars:
   netsuite_source:
+    use_table_name_identifer_override: true # to use seed identifiers in integration tests
     netsuite_schema: netsuite_source_integration_tests_3
     netsuite_data_model_override: netsuite
     netsuite_accounting_books_identifier: "netsuite_accounting_books_data"

models/docs.md

@@ -97,7 +97,7 @@
 
 {% docs addr3 %} The associated address 3. {% enddocs %}
 
-{% docs addressee %} The individual associated with the address {% enddocs %}
+{% docs addressee %} The individual associated with the address. {% enddocs %}
 
 {% docs full_address %} The full address associated. {% enddocs %}
 

models/netsuite2/src_netsuite2.yml

@@ -7,7 +7,7 @@ sources:
     loader: fivetran
     loaded_at_field: _fivetran_synced
     config:
-      enabled: "{{ var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') }}"
+      enabled: "{{ var('netsuite_data_model', 'netsuite') == var('netsuite_data_model_override','netsuite2') and var('netsuite2_union_schemas', []) == [] and var('netsuite2_union_databases', []) == [] }}"
     freshness:
       warn_after: { count: 24, period: hour }
       error_after: { count: 48, period: hour }
@@ -197,7 +197,7 @@ sources:
         description: "{{ doc('consolidated_exchange_rates_table') }}"
         columns:
           - name: id
-            description:  unique identifier for the consolidated exchange rate.
+            description: Unique identifier for the consolidated exchange rate.
           - name: postingperiod
             description: "{{ doc('accounting_period_id') }}"
           - name: fromcurrency
@@ -291,7 +291,7 @@ sources:
         description: "{{ doc('entities_table') }}"
         columns:
           - name: id
-            description: Th "{{ doc('entity_id') }}"e unique identifier of the entity.
+            description: "{{ doc('entity_id') }}"
           - name: contact
             description: The unique identifier of the contact associated with the entity.
           - name: customer

models/netsuite2/stg_netsuite2__account_types.sql

@@ -15,6 +15,12 @@ fields as (
                 staging_columns=get_accounttype_columns()
             )
         }}
+
+        {{ fivetran_utils.source_relation(
+            union_schema_variable='netsuite2_union_schemas', 
+            union_database_variable='netsuite2_union_databases')
+        }}
+
     from base
 ),
 
@@ -34,10 +40,11 @@ final as (
         {%- else -%}
         left
         {%- endif -%} = 'T' as is_leftside,
-        longname as type_name
+        longname as type_name,
+        source_relation
 
     from fields
 )
 
 select *
-from final
+from final

models/netsuite2/stg_netsuite2__accounting_book_subsidiaries.sql

@@ -15,6 +15,12 @@ fields as (
                 staging_columns=get_accountingbooksubsidiaries_columns()
             )
         }}
+
+        {{ fivetran_utils.source_relation(
+            union_schema_variable='netsuite2_union_schemas', 
+            union_database_variable='netsuite2_union_databases')
+        }}
+
     from base
 ),
 
@@ -25,11 +31,12 @@ final as (
         _fivetran_synced,
         accountingbook as accounting_book_id,
         status,
-        subsidiary as subsidiary_id
+        subsidiary as subsidiary_id,
+        source_relation
 
     from fields
     where not coalesce(_fivetran_deleted, false)
 )
 
 select *
-from final
+from final

models/netsuite2/stg_netsuite2__accounting_books.sql

@@ -15,6 +15,12 @@ fields as (
                 staging_columns=get_netsuite2_accounting_books_columns()
             )
         }}
+
+        {{ fivetran_utils.source_relation(
+            union_schema_variable='netsuite2_union_schemas', 
+            union_database_variable='netsuite2_union_databases')
+        }}
+
     from base
 ),
 
@@ -30,10 +36,11 @@ final as (
         isconsolidated = 'T' as is_consolidated,
         contingentrevenuehandling as is_contingent_revenue_handling,
         isprimary = 'T' as is_primary,
-        twosteprevenueallocation as is_two_step_revenue_allocation
+        twosteprevenueallocation as is_two_step_revenue_allocation,
+        source_relation
     from fields
     where not coalesce(_fivetran_deleted, false)
 )
 
 select * 
-from final
+from final

models/netsuite2/stg_netsuite2__accounting_period_fiscal_cal.sql

@@ -15,6 +15,12 @@ fields as (
                 staging_columns=get_accountingperiodfiscalcalendars_columns()
             )
         }}
+
+        {{ fivetran_utils.source_relation(
+            union_schema_variable='netsuite2_union_schemas', 
+            union_database_variable='netsuite2_union_databases')
+        }}
+
     from base
 ),
 
@@ -25,10 +31,11 @@ final as (
         _fivetran_synced,
         accountingperiod as accounting_period_id,
         fiscalcalendar as fiscal_calendar_id,
-        parent as parent_id
+        parent as parent_id,
+        source_relation
     from fields
     where not coalesce(_fivetran_deleted, false)
 )
 
 select *
-from final
+from final

models/netsuite2/stg_netsuite2__accounting_periods.sql

@@ -16,6 +16,12 @@ fields as (
                 staging_columns=get_netsuite2_accounting_periods_columns()
             )
         }}
+
+        {{ fivetran_utils.source_relation(
+            union_schema_variable='netsuite2_union_schemas', 
+            union_database_variable='netsuite2_union_databases')
+        }}
+
     from base
 ),
 
@@ -36,10 +42,11 @@ final as (
         closed = 'T' as is_closed,
         alllocked = 'T' as is_all_locked,
         arlocked = 'T' as is_ar_locked,
-        aplocked = 'T' as is_ap_locked
+        aplocked = 'T' as is_ap_locked,
+        source_relation
     from fields
     where not coalesce(_fivetran_deleted, false)
 )
 
 select * 
-from final
+from final

models/netsuite2/stg_netsuite2__accounts.sql

@@ -15,6 +15,12 @@ fields as (
                 staging_columns=get_netsuite2_accounts_columns()
             )
         }}
+
+        {{ fivetran_utils.source_relation(
+            union_schema_variable='netsuite2_union_schemas', 
+            union_database_variable='netsuite2_union_databases')
+        }}
+
     from base
 ),
 
@@ -41,7 +47,8 @@ final as (
         isinactive = 'T' as is_inactive,
         issummary = 'T' as is_summary,
         eliminate = 'T' as is_eliminate,
-        _fivetran_deleted
+        _fivetran_deleted,
+        source_relation
 
         --The below macro adds the fields defined within your accounts_pass_through_columns variable into the staging model
         {{ fivetran_utils.fill_pass_through_columns('accounts_pass_through_columns') }}
@@ -50,4 +57,4 @@ final as (
 )
 
 select * 
-from final
+from final