Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix array presenting columns to not match single element arrays to scalars for equality #15503

Merged

Conversation

clintropolis
Copy link
Member

Description

The equality filter allows automatic coercion of scalar typed columns to array columns to help assist schema migration between mvds and arrays, as well as scalars to arrays, where the scalar inputs are treated as single element arrays for the purposes of matching.

However, this had an unintended side-effect of allowing the reverse that impacts only native queries, allowing using an equality filter with a scalar match value to match single element arrays of array columns, which is pretty strange (and json_value using SQL).

This PR modifies the equality filter (and array indexes) to no longer allow matching single element arrays in columns with scalar values, while preserving the opposite, where scalar columns can match single element arrays.

This has a side-effect that strictly impacts native query equality filter on mixed type 'auto' columns which contain arrays, that they must now be interacted with as their presenting type, so if any rows are arrays, e.g. the segment metadata and information_schema reports the type as some array type, then the native queries must also filter as if they are some array type. This does not impact SQL, which already has this limitation due to how the type presents itself, and again only impacts mixed type 'auto' columns.

Release note

Native query equals filter on mixed type 'auto' columns which contain arrays must now be filtered as their presenting type, so if any rows are arrays (e.g. the segment metadata and information_schema reports the type as some array type), then the native queries must also filter as if they are some array type. This does not impact SQL, which already has this limitation due to how the type presents itself, and again only impacts mixed type 'auto' columns which contain both scalars and arrays.


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

@clintropolis clintropolis merged commit 1eafe98 into apache:master Dec 8, 2023
83 checks passed
@clintropolis clintropolis deleted the fix-array-scalar-equality-filtering branch December 8, 2023 09:22
@LakshSingla LakshSingla added this to the 29.0.0 milestone Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants