Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DeleteEntry Processor support for parsing json object #3119

Open
dinujoh opened this issue Aug 7, 2023 · 0 comments
Open

DeleteEntry Processor support for parsing json object #3119

dinujoh opened this issue Aug 7, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@dinujoh
Copy link
Member

dinujoh commented Aug 7, 2023

Is your feature request related to a problem? Please describe.
The DeleteEntry processor uses GenericExpressionEvaluator and it doesn't support deleting key with DeleteEntry processor when the expression path refers to json object.

log-pipeline:
  source:
    http:
      path: "/test/path"
  processor:
    - delete_entries:
        with_keys: [ "responseElements" ]
        delete_when: '/responseElements == null'
    - date:
        from_time_received: true
        destination: "@timestamp"
  sink:
    - stdout:

 curl -X POST http://localhost:2021/test/path -H 'Content-Type: application/json; charset=utf-8' -d '[{"log":"sample log", "responseElements": {"a":"a", "b": "b" }}]'
2023-08-07T11:10:58,081 [log-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.pipeline.ProcessWorker - Encountered exception during pipeline log-pipeline processing
org.opensearch.dataprepper.expression.ExpressionEvaluationException: Unable to evaluate statement "/responseElements == null"
        at org.opensearch.dataprepper.expression.GenericExpressionEvaluator.evaluate(GenericExpressionEvaluator.java:41) ~[data-prepper-expression-2.3.2.jar:?]
        at org.opensearch.dataprepper.expression.ExpressionEvaluator.evaluateConditional(ExpressionEvaluator.java:28) ~[data-prepper-api-2.3.2.jar:?]
        at org.opensearch.dataprepper.plugins.processor.mutateevent.DeleteEntryProcessor.doExecute(DeleteEntryProcessor.java:40) ~[mutate-event-processors-2.3.2.jar:?]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.lambda$execute$0(AbstractProcessor.java:54) ~[data-prepper-api-2.3.2.jar:?]
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:69) ~[micrometer-core-1.10.5.jar:1.10.5]
        at org.opensearch.dataprepper.model.processor.AbstractProcessor.execute(AbstractProcessor.java:54) ~[data-prepper-api-2.3.2.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:115) ~[data-prepper-core-2.3.2.jar:?]
        at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:50) [data-prepper-core-2.3.2.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

Describe the solution you'd like
Add support for parsing Json object in LiteralTypeConversionsConfiguration that is used by GenericExpressionEvaluator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

No branches or pull requests

2 participants