Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PPL command expression implementation for geoip #3228

Merged
merged 78 commits into from
Jan 30, 2025
Merged

PPL command expression implementation for geoip #3228

merged 78 commits into from
Jan 30, 2025

Conversation

andy-k-improving
Copy link
Contributor

@andy-k-improving andy-k-improving commented Jan 1, 2025 •
edited by acarbonetto
Loading

Description

Introduce a new PPL command expression geoip, to perform geo-spatial information lookup with the provided IPv4 || IPv6 addresses, result of the lookup is formatted into a tuple with attribute as key and location detail as value.

In this particular setting, SQL plugin will act as a thin client, by relaying the IPEnrichment request to OpenSearch Geo-Spatial plugin, WITHIN the same cluster.
Detail implementation and interface that exposed on Geo-Spatial side can be found:
opensearch-project/geospatial#700

Internally this functionality is achieved by:

  • Adding an no-op OpenSearchFunctionExpression marker to identify this is an expression has no default implement on other runtime (Ex: Prometheus)
  • Update OpenSearchIndex in order to provide an OpenSearch specific handler for eval operator and its expressions, when OS being used as the storage engine.

During runtime, all eval expressions, will being passed to OpenSearchIndex.visitEval( ), then OpenSearchEvalOperator class will pick up the call, by evaluating all eval expression as it is, and then handle all occasion of OpenSearchFunctionExpression separately, by reading the function name and argument, and execute the appropriate business logic.

Marker class OpenSearchFunctionExpression is being used in this case because the actual implementation require runtime OpenSearch client connectivity, however core module is mean to be generic, hence this workaround is being deployed, by tagging it as OpenSearchFunctionExpression on core and only handle it on the opensearch Cradle module .

Related Issues

Resolves: #3037

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@kenrickyap kenrickyap mentioned this pull request Jan 10, 2025
Closed
7 tasks
@andy-k-improving andy-k-improving mentioned this pull request Jan 10, 2025
Closed
@andy-k-improving andy-k-improving marked this pull request as ready for review January 10, 2025 23:35
@andy-k-improving
Copy link
Contributor Author

As per the offline discussion, I have separated out the integration related changes into #3244, in order to minimise the diff.

acarbonetto
acarbonetto reviewed Jan 13, 2025
View reviewed changes
.github/workflows/integ-tests-with-geo.yml Outdated Show resolved Hide resolved
.github/workflows/integ-tests-with-geo.yml Outdated Show resolved Hide resolved
.github/workflows/integ-tests-with-geo.yml Outdated Show resolved Hide resolved
integ-test/build.gradle Outdated Show resolved Hide resolved
integ-test/build.gradle Outdated Show resolved Hide resolved
ppl/src/main/antlr/OpenSearchPPLParser.g4 Outdated Show resolved Hide resolved
acarbonetto
acarbonetto reviewed Jan 22, 2025
View reviewed changes
docs/user/ppl/functions/geoip.rst Outdated Show resolved Hide resolved
docs/user/ppl/functions/geoip.rst Outdated Show resolved Hide resolved
docs/user/ppl/functions/geoip.rst Outdated Show resolved Hide resolved
docs/user/ppl/functions/geoip.rst Outdated Show resolved Hide resolved
core/src/main/java/org/opensearch/sql/expression/ip/GeoIPFunctions.java Outdated Show resolved Hide resolved
integ-test/src/test/java/org/opensearch/sql/ppl/GeoIpFunctionsIT.java Outdated Show resolved Hide resolved
integ-test/src/test/java/org/opensearch/sql/ppl/GeoIpFunctionsIT.java Outdated Show resolved Hide resolved
integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java Outdated Show resolved Hide resolved
...rch/src/main/java/org/opensearch/sql/opensearch/planner/physical/OpenSearchEvalOperator.java Outdated Show resolved Hide resolved
...rch/src/main/java/org/opensearch/sql/opensearch/planner/physical/OpenSearchEvalOperator.java Outdated Show resolved Hide resolved
@andy-k-improving andy-k-improving mentioned this pull request Jan 23, 2025
Open
@andy-k-improving
Update grammar structure
de5f3a5 
Signed-off-by: Andy Kwok <[email protected]>
andy-k-improving and others added 8 commits January 24, 2025 12:45
@andy-k-improving @acarbonetto
Doc fix
7019452 
Co-authored-by: Andrew Carbonetto <[email protected]>
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Simplify Integtest
41e99c6 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update doc
a826375 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Spotless
11f8bad 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update comment
9b735aa 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update unitTest
c74f3fd 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update existing integTest
d981332 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Spotless
241ea63 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Doc fix
66e17ce 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Address code reivews
3d79741 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update diff
c3a9348 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Minimise diff
89f33e5 
Signed-off-by: Andy Kwok <[email protected]>
normanj-bitquill
normanj-bitquill approved these changes Jan 28, 2025
View reviewed changes
Copy link
Contributor

@normanj-bitquill normanj-bitquill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

andy-k-improving and others added 3 commits January 28, 2025 15:14
@andy-k-improving
Update test-cases
11abfd6 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving @acarbonetto
Update docs/user/ppl/functions/ip.rst
b77d557 
Co-authored-by: Andrew Carbonetto <[email protected]>
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving @acarbonetto
Update opensearch/src/main/java/org/opensearch/sql/opensearch/planner…
1b41d5f 
…/physical/OpenSearchEvalOperator.java

Co-authored-by: Andrew Carbonetto <[email protected]>
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update spotless
694b08c 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update coverage
ebc451a 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Spotless
fbcaf0d 
Signed-off-by: Andy Kwok <[email protected]>
@andy-k-improving
Update log
96cc6f6 
Signed-off-by: Andy Kwok <[email protected]>
acarbonetto
acarbonetto approved these changes Jan 29, 2025
View reviewed changes
Copy link
Collaborator

@acarbonetto acarbonetto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@acarbonetto acarbonetto merged commit 3bf19ef into opensearch-project:main Jan 30, 2025
15 checks passed
@acarbonetto acarbonetto deleted the ft-ak-iplocation-cmd branch January 30, 2025 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@acarbonetto acarbonetto acarbonetto approved these changes

@normanj-bitquill normanj-bitquill normanj-bitquill approved these changes

@YANG-DB YANG-DB YANG-DB approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@LantaoJin LantaoJin Awaiting requested review from LantaoJin LantaoJin is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE]Add iplocation function to PPL for IP address geolocation
4 participants
@andy-k-improving @YANG-DB @normanj-bitquill @acarbonetto