Merge pull request #142 from fivetran/bugfix/compilation-error-contac…

…t-lists

[Bug] Compilation error on the end contact lists model

.buildkite/scripts/run_models.sh

@@ -20,9 +20,6 @@ dbt seed --target "$db" --full-refresh
 dbt compile --target "$db" --select hubspot # source does not compile at this time
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --target "$db" --vars '{hubspot_service_enabled: true}' --full-refresh
-dbt run --target "$db" --vars '{hubspot_service_enabled: true}'
-dbt test --target "$db"
 dbt run --vars '{hubspot_service_enabled: true, hubspot_marketing_enabled: true, hubspot_sales_enabled: false}' --target "$db" --full-refresh
 dbt run --vars '{hubspot_service_enabled: true, hubspot_marketing_enabled: true, hubspot_sales_enabled: false, hubspot_merged_deal_enabled: true}' --target "$db"
 dbt test --target "$db"

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -4,48 +4,27 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present) && dbt test
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
+- [ ] The appropriate issue has been linked, tagged, and properly assigned
+- [ ] All necessary documentation and version upgrades have been applied
+    <!--- Be sure to update the package version in the dbt_project.yml, integration_tests/dbt_project.yml, and README if necessary. -->
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
+- [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+Please share any and all of your validation steps:
 <!--- Provide the steps you took to validate your changes below. -->
 
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
 ### If you had to summarize this PR in an emoji, which would it be?
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+:dancer:

CHANGELOG.md

@@ -1,3 +1,15 @@
+# dbt_hubspot v0.17.2
+[PR #142](https://github.com/fivetran/dbt_hubspot/pull/142) includes the following updates:
+
+## 🪲 Bug Fixes 🪛
+- Fixed the `fivetran_utils.enabled_vars` conditional by adding the `hubspot_contact_list_member_enabled` variable in the `hubspot__contact_lists` model. This solves for compilation errors when the `contact_list` source table is not synced in the destination. If `hubspot_contact_list_member_enabled` is `true`, `int_hubspot__email_metrics__by_contact_list` is brought in as a dependency, and ignored otherwise. 
+
+## 🚘 Under the Hood 🚘
+- Updated the `integration_tests/dbt_project.yml` variables to be global to ensure more effective testing of our seed data.
+- Updated `property_closed_date` and `property_createdate` datatypes in `ticket_data` to cast as timestamp to fix datetime data type issues in BigQuery tests.
+- Updated the maintainer PR template to resemble the most up to date format.
+- Removed the check docs GitHub Action as it is no longer necessary.
+
 # dbt_hubspot v0.17.1
 [PR #140](https://github.com/fivetran/dbt_hubspot/pull/140) includes the following updates:
 

README.md

@@ -338,4 +338,3 @@ We highly encourage and welcome contributions to this package. Check out [this d
 # 🏪 Are there any resources available?
 - If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_hubspot/issues/new/choose) section to find the right avenue of support for you.
 - If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
-- Have questions or want to be part of the community discourse? Create a post in the [Fivetran community](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) and our team along with the community can join in on the discussion!

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'hubspot'
-version: '0.17.1'
+version: '0.17.2'
 
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: hubspot_integration_tests_55
+      schema: hubspot_integration_tests_57
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: hubspot_integration_tests_55
+      schema: hubspot_integration_tests_57
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: hubspot_integration_tests_55
+      schema: hubspot_integration_tests_57
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: hubspot_integration_tests_55
+      schema: hubspot_integration_tests_57
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: hubspot_integration_tests_55
+      schema: hubspot_integration_tests_57
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,65 +1,64 @@
 name: 'hubspot_integration_tests'
-version: '0.17.1'
+version: '0.17.2'
 
 profile: 'integration_tests'
 config-version: 2
 vars:
-  hubspot_schema: hubspot_integration_tests_55
-  hubspot_source:
-    hubspot_service_enabled: true
-    # hubspot_sales_enabled: true # enable when generating docs
-    # hubspot_marketing_enabled: true # enable when generating docs
-    # hubspot_contact_merge_audit_enabled: true # enable when generating docs
-    # hubspot_using_all_email_events: true # enable when generating docs
-    # hubspot_merged_deal_enabled: true # enable when generating docs  
-    hubspot_company_property_history_identifier: "company_property_history_data"
-    hubspot_company_identifier: "company_data"
-    hubspot_contact_identifier: "contact_data"
-    hubspot_contact_merge_audit_identifier: "contact_merge_audit_data"
-    hubspot_contact_list_member_identifier: "contact_list_member_data"
-    hubspot_contact_property_history_identifier: "contact_property_history_data"
-    hubspot_deal_identifier: "deal_data"
-    hubspot_deal_stage_identifier: "deal_stage_data"
-    hubspot_deal_company_identifier: "deal_company_data"
-    hubspot_deal_contact_identifier: "deal_contact_data"
-    hubspot_deal_pipeline_identifier: "deal_pipeline_data"
-    hubspot_deal_pipeline_stage_identifier: "deal_pipeline_stage_data"
-    hubspot_deal_property_history_identifier: "deal_property_history_data"
-    hubspot_email_campaign_identifier: "email_campaign_data"
-    hubspot_email_event_identifier: "email_event_data"
-    hubspot_email_event_bounce_identifier: "email_event_bounce_data"
-    hubspot_email_event_click_identifier: "email_event_click_data"
-    hubspot_email_event_deferred_identifier: "email_event_deferred_data"
-    hubspot_email_event_delivered_identifier: "email_event_delivered_data"
-    hubspot_email_event_forward_identifier: "email_event_forward_data"
-    hubspot_email_event_open_identifier: "email_event_open_data"
-    hubspot_email_event_print_identifier: "email_event_print_data"
-    hubspot_email_event_spam_report_identifier: "email_event_spam_report_data"
-    hubspot_email_event_status_change_identifier: "email_event_status_change_data"
-    hubspot_engagement_identifier: "engagement_data"
-    hubspot_engagement_call_identifier: "engagement_call_data"
-    hubspot_engagement_company_identifier: "engagement_company_data"
-    hubspot_engagement_contact_identifier: "engagement_contact_data"
-    hubspot_engagement_deal_identifier: "engagement_deal_data"
-    hubspot_engagement_email_identifier: "engagement_email_data"
-    hubspot_engagement_meeting_identifier: "engagement_meeting_data"
-    hubspot_engagement_note_identifier: "engagement_note_data"
-    hubspot_engagement_task_identifier: "engagement_task_data"
-    hubspot_owner_identifier: "owner_data"
-    hubspot_property_identifier: "property_data"
-    hubspot_property_option_identifier: "property_option_data"
-    hubspot_ticket_company_identifier: "ticket_company_data"
-    hubspot_ticket_contact_identifier: "ticket_contact_data"
-    hubspot_ticket_deal_identifier: "ticket_deal_data"
-    hubspot_ticket_engagement_identifier: "ticket_engagement_data"
-    hubspot_ticket_pipeline_identifier: "ticket_pipeline_data"
-    hubspot_ticket_pipeline_stage_identifier: "ticket_pipeline_stage_data"
-    hubspot_ticket_property_history_identifier: "ticket_property_history_data"
-    hubspot_ticket_identifier: "ticket_data"
-    hubspot_contact_list_identifier: "contact_list_data"
-    hubspot_email_event_sent_identifier: "email_event_sent_data"
-    hubspot_email_event_dropped_identifier: "email_event_dropped_data"
-    hubspot_merged_deal_identifier: "merged_deal_data"
+  hubspot_schema: hubspot_integration_tests_57
+  hubspot_service_enabled: true
+  # hubspot_sales_enabled: true # enable when generating docs
+  # hubspot_marketing_enabled: true # enable when generating docs
+  # hubspot_contact_merge_audit_enabled: true # enable when generating docs
+  # hubspot_using_all_email_events: true # enable when generating docs
+  # hubspot_merged_deal_enabled: true # enable when generating docs 
+  hubspot_company_property_history_identifier: "company_property_history_data"
+  hubspot_company_identifier: "company_data"
+  hubspot_contact_identifier: "contact_data"
+  hubspot_contact_merge_audit_identifier: "contact_merge_audit_data"
+  hubspot_contact_list_member_identifier: "contact_list_member_data"
+  hubspot_contact_property_history_identifier: "contact_property_history_data"
+  hubspot_deal_identifier: "deal_data"
+  hubspot_deal_stage_identifier: "deal_stage_data"
+  hubspot_deal_company_identifier: "deal_company_data"
+  hubspot_deal_contact_identifier: "deal_contact_data"
+  hubspot_deal_pipeline_identifier: "deal_pipeline_data"
+  hubspot_deal_pipeline_stage_identifier: "deal_pipeline_stage_data"
+  hubspot_deal_property_history_identifier: "deal_property_history_data"
+  hubspot_email_campaign_identifier: "email_campaign_data"
+  hubspot_email_event_identifier: "email_event_data"
+  hubspot_email_event_bounce_identifier: "email_event_bounce_data"
+  hubspot_email_event_click_identifier: "email_event_click_data"
+  hubspot_email_event_deferred_identifier: "email_event_deferred_data"
+  hubspot_email_event_delivered_identifier: "email_event_delivered_data"
+  hubspot_email_event_forward_identifier: "email_event_forward_data"
+  hubspot_email_event_open_identifier: "email_event_open_data"
+  hubspot_email_event_print_identifier: "email_event_print_data"
+  hubspot_email_event_spam_report_identifier: "email_event_spam_report_data"
+  hubspot_email_event_status_change_identifier: "email_event_status_change_data"
+  hubspot_engagement_identifier: "engagement_data"
+  hubspot_engagement_call_identifier: "engagement_call_data"
+  hubspot_engagement_company_identifier: "engagement_company_data"
+  hubspot_engagement_contact_identifier: "engagement_contact_data"
+  hubspot_engagement_deal_identifier: "engagement_deal_data"
+  hubspot_engagement_email_identifier: "engagement_email_data"
+  hubspot_engagement_meeting_identifier: "engagement_meeting_data"
+  hubspot_engagement_note_identifier: "engagement_note_data"
+  hubspot_engagement_task_identifier: "engagement_task_data"
+  hubspot_owner_identifier: "owner_data"
+  hubspot_property_identifier: "property_data"
+  hubspot_property_option_identifier: "property_option_data"
+  hubspot_ticket_company_identifier: "ticket_company_data"
+  hubspot_ticket_contact_identifier: "ticket_contact_data"
+  hubspot_ticket_deal_identifier: "ticket_deal_data"
+  hubspot_ticket_engagement_identifier: "ticket_engagement_data"
+  hubspot_ticket_pipeline_identifier: "ticket_pipeline_data"
+  hubspot_ticket_pipeline_stage_identifier: "ticket_pipeline_stage_data"
+  hubspot_ticket_property_history_identifier: "ticket_property_history_data"
+  hubspot_ticket_identifier: "ticket_data"
+  hubspot_contact_list_identifier: "contact_list_data"
+  hubspot_email_event_sent_identifier: "email_event_sent_data"
+  hubspot_email_event_dropped_identifier: "email_event_dropped_data"
+  hubspot_merged_deal_identifier: "merged_deal_data"
 
 seeds:
   hubspot_integration_tests:
@@ -175,6 +174,10 @@ seeds:
         created: timestamp
         obsoleted_by_created: timestamp
         sent_by_created: timestamp
+    ticket_data:
+      +column_types:
+        property_closed_date: timestamp
+        property_createdate: timestamp
     ticket_property_history_data:
       +column_types:
         timestamp_instant: timestamp

integration_tests/requirements.txt

@@ -3,5 +3,5 @@ dbt-bigquery>=1.3.0,<2.0.0
 dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
-dbt-spark[PyHive]>=1.3.0,<2.0.0
+dbt-spark[PyHive]>=1.3.0,<1.8.0
 dbt-databricks>=1.3.0,<1.7.13

integration_tests/seeds/ticket_data.csv

@@ -12,14 +12,14 @@ _fivetran_synced,id,is_deleted,property_closed_date,property_createdate,property
 2020-11-26 18:51:39,197316537,FALSE,2020-09-12 19:53:15,2020-09-12 19:53:15,,1486381,1706266,,,,28a8d366cc9182135385d79b15c7b7af,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,121723395,FALSE,2020-04-20 12:49:01,2020-04-20 12:49:01,,1486381,1706266,,,,87b17cc25902180b6572f3f35a059711,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,226600613,FALSE,2020-10-22 18:44:56,2020-10-22 18:44:56,,1486381,1706266,,,,98ff5402c937d4664bbbb00ec05cfe47,37a6259cc0c1dae299a7866489dff0bd
-2020-11-26 18:51:39,338430345,FALSE,2021-03-16 0:32:29,2021-03-16 0:32:29,,1486381,1706266,,,,d8254b6c7ad36c09cf05a464856dd19d,37a6259cc0c1dae299a7866489dff0bd
+2020-11-26 18:51:39,338430345,FALSE,2021-03-16 00:32:29,2021-03-16 00:32:29,,1486381,1706266,,,,d8254b6c7ad36c09cf05a464856dd19d,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,210718823,FALSE,2020-10-02 21:12:18,2020-10-02 21:12:18,,1486381,1706266,,,,5f4a993802657bf17e2d7016288b0bab,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,198968635,FALSE,2020-09-15 15:27:16,2020-09-15 15:27:16,,1486381,1706266,,,,d52dbfdbb2eb5c94c701cf7c63eea941,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,200415373,FALSE,2020-09-17 15:43:38,2020-09-17 15:43:38,,1486381,1706266,,,,5e05e38174cc749a7d0a04236be00fbc,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,287177375,FALSE,2021-01-20 18:36:51,2021-01-20 18:36:51,,1486381,1706266,,,,8c697d166a33fa3758f03c2097779d9f,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,234235597,FALSE,2020-11-03 14:24:28,2020-11-03 14:24:28,,1486381,1706266,,,,54e965e21077751164217935c380facb,37a6259cc0c1dae299a7866489dff0bd
-2020-11-26 18:51:39,280473935,FALSE,2021-01-11 3:42:30,2021-01-11 3:42:30,,1486381,1706266,,,,2f03edf45aaa4d282095e7e88330cecc,37a6259cc0c1dae299a7866489dff0bd
+2020-11-26 18:51:39,280473935,FALSE,2021-01-11 03:42:30,2021-01-11 03:42:30,,1486381,1706266,,,,2f03edf45aaa4d282095e7e88330cecc,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,201110733,FALSE,2020-09-18 13:30:40,2020-09-18 13:30:40,,1486381,1706266,,,,a7aaed53b33cdd85eee6308d6bc7a240,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,120338492,FALSE,2020-04-16 13:25:03,2020-04-16 13:25:03,,1486381,1706266,,,,e11b8d7610e2205e2c3cb233b926d56d,37a6259cc0c1dae299a7866489dff0bd
-2020-11-26 18:51:39,171529540,FALSE,2020-08-03 2:49:24,2020-08-03 2:49:24,,1486381,1706266,,,,6c5237eee681e041f8def82aff141375,37a6259cc0c1dae299a7866489dff0bd
+2020-11-26 18:51:39,171529540,FALSE,2020-08-03 02:49:24,2020-08-03 02:49:24,,1486381,1706266,,,,6c5237eee681e041f8def82aff141375,37a6259cc0c1dae299a7866489dff0bd
 2020-11-26 18:51:39,249658461,FALSE,2020-11-26 18:51:39,2020-11-26 18:51:39,,1486381,1706266,,,,f053faea765fc8b6632e6099f62b09f1,37a6259cc0c1dae299a7866489dff0bd

models/marketing/hubspot__contact_lists.sql

@@ -1,12 +1,11 @@
-
 {{ config(enabled=fivetran_utils.enabled_vars(['hubspot_marketing_enabled','hubspot_contact_list_enabled'])) }}
 
 with contact_lists as (
 
     select *
     from {{ var('contact_list') }}
 
-{% if fivetran_utils.enabled_vars(['hubspot_marketing_enabled','hubspot_email_event_enabled','hubspot_contact_enabled','hubspot_email_event_sent_enabled']) %}
+{% if fivetran_utils.enabled_vars(['hubspot_marketing_enabled','hubspot_email_event_enabled','hubspot_contact_enabled','hubspot_contact_list_member_enabled','hubspot_email_event_sent_enabled']) %}
 
 ), email_metrics as (