-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CALCITE-6805] Support hex function for Hive and Spark Library #4172
Conversation
5328c0f
to
5fe1251
Compare
core/src/main/java/org/apache/calcite/sql/fun/SqlLibraryOperators.java
Outdated
Show resolved
Hide resolved
5fe1251
to
97f7e9b
Compare
@rubenada Thanks for your review, all comments had addressed, PTAL |
Thanks @xuzifu666 . I left another minor comment, but overall lgtm. |
97f7e9b
to
ad73c8e
Compare
Very Thanks for @rubenada review , had addressed comments and fix the title of pr and commit, PTAL~ |
ad73c8e
to
5fca000
Compare
"abc", | ||
"VARCHAR NOT NULL"); | ||
f.checkString("unhex('')", "", "VARCHAR NOT NULL"); | ||
f.checkNull("unhex(cast(null as varbinary))"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to add more tests
e.g.
scala> val df = spark.sql("select unhex(null)")
df: org.apache.spark.sql.DataFrame = [unhex(NULL): binary]
scala> df.show()
+-----------+
|unhex(NULL)|
+-----------+
| null|
+-----------+
scala> val df = spark.sql("select unhex(0)")
df: org.apache.spark.sql.DataFrame = [unhex(0): binary]
scala> df.show()
+--------+
|unhex(0)|
+--------+
| [00]|
+--------+
scala> val df = spark.sql("select unhex(-1)")
df: org.apache.spark.sql.DataFrame = [unhex(-1): binary]
scala> df.show()
+---------+
|unhex(-1)|
+---------+
| null|
+---------+
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you sure that unhex(0) is equal to null? I tested it in Spark and the result is [00]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, unhex maybe had some problem in hive and spark side, after discussion I remove the implement of unhex from this pr, this pr change to support hex only and issue also changed too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xuzifu666 feel free to open a separate Jira ticket (related to this one) to cover the unhex
implementation, even if you don't plan to work on it right away.
6b93d9b
to
3e88ead
Compare
@caicancai had addressed comments,PTAL |
3e88ead
to
5e96887
Compare
@@ -364,6 +364,21 @@ public static String toHex(ByteString byteString) { | |||
return Hex.encodeHexString(byteString.getBytes()); | |||
} | |||
|
|||
/** SQL HEX(varchar) function. */ | |||
public static String hex(String value) { | |||
return Hex.encodeHexString(value.getBytes(UTF_8)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if it's UTF-8, but I'd accept it if it's compatible with all tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, UTF-8 seems ok for all, I had consider other funciton such md5/base32.. function also use UTF8 and CI is OK.
5e96887
to
4d3531e
Compare
LGTM, @caicancai waiting for you approval too before finalize the merge. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to add some tests
4d3531e
to
11ab8ba
Compare
Thanks for reminder, I had add more tests for it~ @caicancai |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Just a small suggestion, you don't need to compress and submit every time, just compress and submit after the final approval. |
Currently Spark Hive hex and unhex function not introduce for calcite, need to support it.
jira: https://issues.apache.org/jira/browse/CALCITE-6805