Skip to content

Commit

Permalink
fix(data-warehouse): Handle when we dont know the data type (#27980)
Browse files Browse the repository at this point in the history
  • Loading branch information
@Gilbert09 @adamleithp
Gilbert09 authored and adamleithp committed Jan 29, 2025
1 parent eb42f9d commit 07ca82f
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 16 deletions.
33 changes: 17 additions & 16 deletions mypy-baseline.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -338,12 +338,12 @@ posthog/hogql/query.py:0: error: Subclass of "SelectQuery" and "SelectSetQuery"
posthog/api/organization.py:0: error: Incompatible return value type (got "int | None", expected "Level | None") [return-value]
posthog/api/capture.py:0: error: Module has no attribute "utc" [attr-defined]
posthog/queries/person_query.py:0: error: Incompatible type for lookup 'pk': (got "str | int | list[str]", expected "str | int") [misc]
posthog/api/action.py:0: error: Argument 1 to <tuple> has incompatible type "*tuple[str, ...]"; expected "type[BaseRenderer]" [arg-type]
posthog/queries/event_query/event_query.py:0: error: Incompatible type for lookup 'pk': (got "str | int | list[str]", expected "str | int") [misc]
posthog/hogql_queries/sessions_timeline_query_runner.py:0: error: Statement is unreachable [unreachable]
posthog/hogql_queries/hogql_query_runner.py:0: error: Statement is unreachable [unreachable]
posthog/hogql_queries/hogql_query_runner.py:0: error: Incompatible return value type (got "SelectQuery | SelectSetQuery", expected "SelectQuery") [return-value]
posthog/hogql_queries/events_query_runner.py:0: error: Statement is unreachable [unreachable]
posthog/api/action.py:0: error: Argument 1 to <tuple> has incompatible type "*tuple[str, ...]"; expected "type[BaseRenderer]" [arg-type]
posthog/queries/breakdown_props.py:0: error: Argument 1 to "translate_hogql" has incompatible type "str | int"; expected "str" [arg-type]
posthog/queries/breakdown_props.py:0: error: Incompatible type for lookup 'pk': (got "str | None", expected "str | int") [misc]
posthog/queries/breakdown_props.py:0: error: Incompatible return value type (got "str | None", expected "str") [return-value]
Expand Down Expand Up @@ -483,16 +483,6 @@ posthog/tasks/exports/test/test_csv_exporter_renders.py:0: error: Item "memoryvi
posthog/tasks/exports/test/test_csv_exporter_renders.py:0: error: Item "None" of "bytes | memoryview | None" has no attribute "decode" [union-attr]
posthog/tasks/exports/test/test_csv_exporter_renders.py:0: error: Item "memoryview" of "bytes | memoryview | None" has no attribute "decode" [union-attr]
posthog/tasks/exports/test/test_csv_exporter_renders.py:0: error: Item "None" of "bytes | memoryview | None" has no attribute "decode" [union-attr]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Unused "type: ignore" comment [unused-ignore]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Unused "type: ignore" comment [unused-ignore]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Argument 3 to "get_value" of "ConfigProvider" has incompatible type "str | None"; expected "str" [arg-type]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Unused "type: ignore" comment [unused-ignore]
posthog/management/commands/sync_persons_to_clickhouse.py:0: error: Argument 4 to "create_person_override" has incompatible type "int | None"; expected "int" [arg-type]
posthog/management/commands/sync_persons_to_clickhouse.py:0: error: Argument "group_type_index" to "raw_create_group_ch" has incompatible type "int"; expected "Literal[0, 1, 2, 3, 4]" [arg-type]
posthog/management/commands/migrate_team.py:0: error: Incompatible types in assignment (expression has type "None", variable has type "BatchExport") [assignment]
posthog/management/commands/migrate_team.py:0: error: "BatchExportDestination" has no attribute "exclude_events" [attr-defined]
posthog/management/commands/migrate_team.py:0: error: "BatchExportDestination" has no attribute "include_events" [attr-defined]
posthog/management/commands/fix_future_person_created_at.py:0: error: Argument "version" to "create_person" has incompatible type "int | None"; expected "int" [arg-type]
posthog/hogql_queries/test/test_query_runner.py:0: error: Variable "TestQueryRunner" is not valid as a type [valid-type]
posthog/hogql_queries/test/test_query_runner.py:0: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
posthog/hogql_queries/test/test_query_runner.py:0: error: Invalid base class "TestQueryRunner" [misc]
Expand Down Expand Up @@ -585,10 +575,6 @@ posthog/api/notebook.py:0: error: Incompatible types in assignment (expression h
posthog/year_in_posthog/year_in_posthog.py:0: error: Item "None" of "dict[Any, Any] | None" has no attribute "get" [union-attr]
posthog/year_in_posthog/year_in_posthog.py:0: error: Argument 1 to "stats_for_user" has incompatible type "dict[Any, Any] | None"; expected "dict[Any, Any]" [arg-type]
posthog/year_in_posthog/year_in_posthog.py:0: error: Item "None" of "dict[Any, Any] | None" has no attribute "get" [union-attr]
posthog/warehouse/external_data_source/source.py:0: error: Incompatible types in assignment (expression has type "int", target has type "str") [assignment]
posthog/warehouse/external_data_source/source.py:0: error: Incompatible types in assignment (expression has type "int", target has type "str") [assignment]
posthog/warehouse/external_data_source/source.py:0: error: Incompatible types in assignment (expression has type "dict[str, Collection[str]]", variable has type "StripeSourcePayload") [assignment]
posthog/warehouse/external_data_source/source.py:0: error: Argument 1 to "_create_source" has incompatible type "StripeSourcePayload"; expected "dict[Any, Any]" [arg-type]
posthog/warehouse/data_load/validate_schema.py:0: error: Incompatible types in assignment (expression has type "object", variable has type "DataWarehouseCredential | Combinable | None") [assignment]
posthog/warehouse/data_load/validate_schema.py:0: error: Incompatible types in assignment (expression has type "object", variable has type "str | int | Combinable") [assignment]
posthog/warehouse/data_load/validate_schema.py:0: error: Incompatible types in assignment (expression has type "dict[str, dict[str, str | bool]] | dict[str, str]", variable has type "dict[str, dict[str, str]]") [assignment]
Expand All @@ -603,6 +589,7 @@ posthog/temporal/data_imports/pipelines/pipeline_sync.py:0: error: Incompatible
posthog/temporal/data_imports/pipelines/pipeline_sync.py:0: error: Incompatible types in assignment (expression has type "object", variable has type "str | int | Combinable") [assignment]
posthog/temporal/data_imports/pipelines/pipeline_sync.py:0: error: Incompatible types in assignment (expression has type "dict[str, dict[str, str | bool]] | dict[str, str]", variable has type "dict[str, dict[str, str]]") [assignment]
posthog/temporal/data_imports/pipelines/pipeline_sync.py:0: error: Item "None" of "dict[str, str] | None" has no attribute "get" [union-attr]
posthog/temporal/data_imports/pipelines/pipeline/utils.py:0: error: Statement is unreachable [unreachable]
posthog/taxonomy/property_definition_api.py:0: error: Item "AnonymousUser" of "User | AnonymousUser" has no attribute "organization" [union-attr]
posthog/taxonomy/property_definition_api.py:0: error: Item "None" of "Organization | Any | None" has no attribute "is_feature_available" [union-attr]
posthog/taxonomy/property_definition_api.py:0: error: Item "ForeignObjectRel" of "Field[Any, Any] | ForeignObjectRel | GenericForeignKey" has no attribute "cached_col" [union-attr]
Expand Down Expand Up @@ -642,6 +629,16 @@ posthog/session_recordings/session_recording_api.py:0: error: Argument "team_id"
posthog/session_recordings/session_recording_api.py:0: error: Value of type variable "SupportsRichComparisonT" of "sorted" cannot be "str | None" [type-var]
posthog/session_recordings/session_recording_api.py:0: error: Argument 1 to "get" of "dict" has incompatible type "str | None"; expected "str" [arg-type]
posthog/models/test/test_organization_model.py:0: error: Module "django.utils.timezone" does not explicitly export attribute "timedelta" [attr-defined]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Unused "type: ignore" comment [unused-ignore]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Unused "type: ignore" comment [unused-ignore]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Argument 3 to "get_value" of "ConfigProvider" has incompatible type "str | None"; expected "str" [arg-type]
posthog/management/commands/warehouse_last_incremental_value_sync.py:0: error: Unused "type: ignore" comment [unused-ignore]
posthog/management/commands/sync_persons_to_clickhouse.py:0: error: Argument 4 to "create_person_override" has incompatible type "int | None"; expected "int" [arg-type]
posthog/management/commands/sync_persons_to_clickhouse.py:0: error: Argument "group_type_index" to "raw_create_group_ch" has incompatible type "int"; expected "Literal[0, 1, 2, 3, 4]" [arg-type]
posthog/management/commands/migrate_team.py:0: error: Incompatible types in assignment (expression has type "None", variable has type "BatchExport") [assignment]
posthog/management/commands/migrate_team.py:0: error: "BatchExportDestination" has no attribute "exclude_events" [attr-defined]
posthog/management/commands/migrate_team.py:0: error: "BatchExportDestination" has no attribute "include_events" [attr-defined]
posthog/management/commands/fix_future_person_created_at.py:0: error: Argument "version" to "create_person" has incompatible type "int | None"; expected "int" [arg-type]
posthog/hogql/test/test_resolver.py:0: error: Item "None" of "JoinExpr | None" has no attribute "next_join" [union-attr]
posthog/hogql/test/test_resolver.py:0: error: Item "None" of "JoinExpr | Any | None" has no attribute "constraint" [union-attr]
posthog/hogql/test/test_resolver.py:0: error: Item "None" of "JoinConstraint | Any | None" has no attribute "constraint_type" [union-attr]
Expand Down Expand Up @@ -733,6 +730,10 @@ ee/clickhouse/views/experiments.py:0: error: Argument 4 to "ClickhouseTrendExper
ee/clickhouse/views/experiments.py:0: error: Argument 4 to "ClickhouseFunnelExperimentResult" has incompatible type "datetime | None"; expected "datetime" [arg-type]
ee/clickhouse/views/experiments.py:0: error: Argument 4 to "ClickhouseSecondaryExperimentResult" has incompatible type "datetime | None"; expected "datetime" [arg-type]
ee/clickhouse/views/experiments.py:0: error: Item "None" of "User | None" has no attribute "email" [union-attr]
posthog/warehouse/external_data_source/source.py:0: error: Incompatible types in assignment (expression has type "int", target has type "str") [assignment]
posthog/warehouse/external_data_source/source.py:0: error: Incompatible types in assignment (expression has type "int", target has type "str") [assignment]
posthog/warehouse/external_data_source/source.py:0: error: Incompatible types in assignment (expression has type "dict[str, Collection[str]]", variable has type "StripeSourcePayload") [assignment]
posthog/warehouse/external_data_source/source.py:0: error: Argument 1 to "_create_source" has incompatible type "StripeSourcePayload"; expected "dict[Any, Any]" [arg-type]
posthog/warehouse/api/external_data_schema.py:0: error: Incompatible return value type (got "str | None", expected "SyncType | None") [return-value]
posthog/warehouse/api/external_data_schema.py:0: error: Argument 1 to "get_sql_schemas_for_source_type" has incompatible type "str"; expected "Type" [arg-type]
posthog/warehouse/api/external_data_schema.py:0: error: No overload variant of "get" of "dict" matches argument type "str" [call-overload]
Expand Down Expand Up @@ -853,4 +854,4 @@ posthog/api/test/batch_exports/test_update.py:0: error: Dict entry 1 has incompa
posthog/api/test/batch_exports/test_update.py:0: error: Value of type "BatchExport" is not indexable [index]
posthog/api/test/batch_exports/test_update.py:0: error: Value of type "BatchExport" is not indexable [index]
posthog/api/test/batch_exports/test_update.py:0: error: Value of type "BatchExport" is not indexable [index]
posthog/api/test/batch_exports/test_pause.py:0: error: "batch_export_delete_schedule" does not return a value (it only ever returns None) [func-returns-value]
posthog/api/test/batch_exports/test_pause.py:0: error: "batch_export_delete_schedule" does not return a value (it only ever returns None) [func-returns-value]
3 changes: 3 additions & 0 deletions posthog/temporal/data_imports/pipelines/pipeline/utils.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,9 @@ def _handle_null_columns_with_definitions(table: pa.Table, resource: DltResource
return table

for field_name, data_type in column_hints.items():
if data_type is None:
continue

normalized_field_name = normalize_column_name(field_name)
# If the table doesn't have all fields, then add a field with all Nulls and the correct field type
if normalized_field_name not in table.schema.names:
Expand Down

0 comments on commit 07ca82f

Please sign in to comment.