elastalert not sending all the hits

[antoniopafundi-yv]

Hi,
elastalert send only some alert, but it doesn't send all the hits.
If I run elastalert-test-rule --config config/elastalert_config.yaml rules/slack_alert_f1_sniff.yaml I get instead all the hits.
Not sure is a problem with slack, because if I check elastalert _index in elk I don't have all the hits neither.

This is my rule:

name: my_name

type: any

realert:
  minutes: 0

# (Required)
# Index to search, wildcard supported
index: my_index

filter:
- term:
    report_tags.keyword: "my_tag"
- term:
    fail: 1

# (Required)
# The alert is use when a match is found
alert:
- "slack"

slack:
# prod slack
slack_webhook_url: "mywebhook"

slack_emoji_override: ':warning:'
slack_parse_override: full

alert_subject: "Dashboard link: mylink"
alert_text: |
    Failure test: {0}
    @timestamp: {1}
    Box: {2} {3} {4} {5}
    Failure reason: {6}
    Slot: myslot
    Report tags: {8}

alert_text_type: alert_text_only
alert_text_args: ["test_name","@timestamp", "box_oem", "box_type", "box_model","box_ps_version", "failure_reason","slot","report_tags"]

Any idea what could be wrong?
Thanks

[jertel]

When you say it's not sending all the hits do you mean it's not alerting on every rule match? Can you give some more specific examples with more timing details? Keep in mind that Elastalert only checks Elasticsearch once every minute, by default. So if you're expecting to see multiple alerts fire for the same rule within a 60 second window that would require customizing the run_every config value.

[antoniopafundi-yv]

Hi,
thanks for the answer.
Keeping in mind for example 6th of October I had one failure and didn't send any.
7th of October I had 3 failures at:
October 7th 2020, 14:35:07.463
October 7th 2020, 13:05:24.861
October 7th 2020, 12:50:42.066

If I look for elastalert _index in kibana (same for the alert on slack) I have
October 7th 2020, 14:47:49.315 time when failure registered on elastalert _index
Match time = October 7th 2020, 14:35:07.463 (only last failure has been detected).
I didn't run that day the test rule, but I imagine it would have got all the hits (as it did in the past).

For completeness I share my config

# number of replicas to run
replicaCount: 1

# number of helm release revisions to retain
revisionHistoryLimit: 5

# Default internal between alert checks against the elasticsearch datasource, in minutes
runIntervalMins: 1

# Default rule buffer duration, in minutes
bufferTimeMins: 15

# Amount of time to retry and deliver failed alerts (1440 minutes per day)
alertRetryLimitMins: 2880

# Default time before realerting, in minutes
realertIntervalMins: ""

# For ES 5: The name of the index which stores elastalert's statuses, typically elastalert_status
# For ES 6: The prefix of the names of indices which store elastalert's statuses, typicaly elastalert
#
# See https://github.com/Yelp/elastalert/commit/c250100b7be07c68a53789569a86f87193ec37f4 for more details about this differentiation.
#
writebackIndex: elastalert

image:
  # docker image
  repository: jertel/elastalert-docker
  # docker image tag
  tag: 0.2.4
  pullPolicy: IfNotPresent

resources: {}

# Annotations to be added to pods
podAnnotations: {}

elasticsearch:
  # elasticsearch endpoint e.g. (svc.namespace||svc)
  host: my_elk
  # elasticsearch port
  port: 9201
  # whether or not to connect to es_host using TLS
  useSsl: "True"
  # Username if authenticating to ES with basic auth
  username: ""
  # Password if authenticating to ES with basic auth
  password: ""

[jertel]

I agree, those timings should have caused an alert to trigger, provided the filter conditions were met, which I cannot confirm from what has been provided. You will need to look at the elastalert logs and the elastalert_status index, specifically in the minutes before and after the missing alert (Ex: 12:49 - 12-52 for the 12:50 occurrence). The index data will show you that Elastalert was indeed running and connecting to ES for that rule's filter, and the matches/hits values will tell you whether it succeeded in detecting an eligible match.

[antoniopafundi-yv]

If I run kubectl logs elastalert-747b6f47dd-ljk9g
i have a bunch of errors..

    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))


During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 77, in perform_request
    response = self.session.send(prepared_request, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_config_change (trigger: interval[0:01:00], next run at: 2020-10-09 14:02:55 UTC)" was missed by 0:00:02.403049
WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_rule_execution (trigger: interval[0:01:00], next run at: 2020-10-09 14:39:23 UTC)" was missed by 0:00:01.157457
WARNING:apscheduler.executors.default:Run time of job "ElastAlerter.handle_config_change (trigger: interval[0:01:00], next run at: 2020-10-09 15:02:55 UTC)" was missed by 0:00:01.431624

[jertel]

That could explain what you're seeing. Check your Elasticsearch logs for the corresponding connection issue on the ES side to find any clues as to why the connection is being dropped. If you have a multi-node kubernetes cluster it's possible there's a network problem in between the nodes.

[antoniopafundi-yv]

Hi,
thank you for all the insights. So I discovered that the @timestamp in elasticsearch was the starting time of my tests, so by the time the results were loaded in ELK the bufferTimeMins of 15 was already expired. Increasing that value seems until now to have solved the problem.