-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add Spark array_append function #12043
base: main
Are you sure you want to change the base?
Conversation
✅ Deploy Preview for meta-velox canceled.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your contribution. We can simply register this Presto function to Spark if their semantics are exactly the same. If any change on the implementation is needed for the reuse, we could move it to the 'velox/functions/lib' folder and make the necessary changes. Please also add documentation for the new registered Spark function.
4263b6c
to
dbdc5ab
Compare
@rui-mo Updated as your suggestion, can you review again? |
dbdc5ab
to
5ec415e
Compare
5ec415e
to
f404b3f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: it would be nice to include Spark's implementation to the PR description.
@rui-mo updated, can you review again? thanks |
f404b3f
to
207104f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
207104f
to
e15fbeb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Looks good overall.
@@ -15,6 +15,15 @@ Array Functions | |||
|
|||
SELECT array(1, 2, 3); -- [1,2,3] | |||
|
|||
.. spark:function:: array_append(array(E), element) -> array(E) | |||
Add the ``element`` at the end of the ``array`` passed as first argument. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: array passed as first argument -> input array
.. spark:function:: array_append(array(E), element) -> array(E) | ||
Add the ``element`` at the end of the ``array`` passed as first argument. | ||
Type of element should be same to type of the elements of the ``array``. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use reference to element.
.. spark:function:: array_append(array(E), element) -> array(E) | ||
Add the ``element`` at the end of the ``array`` passed as first argument. | ||
Type of element should be same to type of the elements of the ``array``. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: same -> the same
type of the elements of the array -> the type of elements in the array
Add the ``element`` at the end of the ``array`` passed as first argument. | ||
Type of element should be same to type of the elements of the ``array``. | ||
NULL element is also appended into the array. But if the ``array`` passed is NULL output is NULL. :: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But if the
array
passed is NULL output is NULL.
Comment suggestion: Returns NULL when the input ``array`` is NULL.
{7, 8, 9, 33}, | ||
{10, 20, std::nullopt, std::nullopt}, | ||
}); | ||
testExpression( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would you add a test when the input array and element are both null?
Add Spark array_append function, spark function code: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L1747