SO Elastic Integration w/ OpenCTI for enrichment.

[Slipperyclock]

Hello SO community, I am looking to get some help or at least pointed in the right direction to get this moving. I have been using SO since ...wow 16.04, GEEZ we came a long way. Good work you guys. I digress, I am looking to setup threat intel enrichment with either MISP or OpenCTI intel. Right now I have an OpenCTI server up, my SO instance is ingesting the data and it shows up in a number of locations. After reading and watching some videos, I am a bit confused on the difference and coordination between a .ds and logs-. First all, I have read https://glue.ghost.io/leveraging-threat-intel-for-event-enrichment-in-security-onion/, #12665, https://www.elastic.co/guide/en/integrations/current/ti_opencti.html and watched https://www.youtube.com/watch?v=4TwstDJBiVw&list=PLljFlTO9rB168zCHTwjrxUJPvc-o8RtwO [Enrich Your Data and Your Life SO 2023]. After all of that I have come up with the following.

Setup: SO 2.4.120 Distributed.

1. Create the enrichment Policy, 2x one for IP and other for Domain/hostname.

• This execution works as expected and returns successful.
• I did notice a discrepancy where the video which is newer than the article from Wes, uses .ds, where as Wes uses the so-threatintel-* index.

PUT /_enrich/policy/opencti-ip-enrichment-policy
{
 "match":{
 "indices":".ds-logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.ip",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

PUT /_enrich/policy/opencti-domain-enrichment-policy
{
 "match":{
 "indices":".ds-logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.url.domain",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

2. Execute Policy

POST /_enrich/policy/opencti-ip-enrichment-policy/_execute
POST /_enrich/policy/opencti-domain-enrichment-policy/_execute

• When I execute these it says the match does not work and I get the below error for both "threat.indicator.ip" and "threat.indicator.url.domain".

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "Match field 'threat.indicator.url.domain' doesn't have a correct mapping type for policy type 'match'"
      }
    ],
    "type": "exception",
    "reason": "Match field 'threat.indicator.url.domain' doesn't have a correct mapping type for policy type 'match'"
  },
  "status": 500
}

• If I recreate the policies and use 'logs-ti_opencti_latest.dest_indicator-' instead of 'ds-logs-ti_opencti.indicator-default-' the command executes successfully.

3. Check that the enrich index is created.

• This does return results and shows Yellow to Green.
  grep opencti-domain-enrichment-policy /opt/so/log/elasticsearch/securityonion.log

• When attempt the check from Wes' documentation nothing shows up, but if I go into the UI and follow though stack management, I can see .enrich- in "Index Management" > Indicies "Check Included hidden indicies".
  sudo so-elasticsearch-query _cat/indices | grep enrich

4 Create Ingest Pipeline

• I then create this file.
• The video doesn't reference any action to be taken other than restart elasticsearch. Wes' documentation says you need to the final pipeline for the indexes you want to enrich.

{ 
 "description" : "Threat Enrichment",
 "processors" : [
  { "enrich":{ "description": "Enrich IP address info with threat intel indicators", "policy_name": "opencti-ip-enrichment-policy", "target_field": "enrichment", "field": "destination.ip", "ignore_failure": true }},
  { "enrich":{ "description": "Enrich IP address info with threat intel indicators", "policy_name": "opencti-ip-enrichment-policy", "target_field": "enrichment", "field": "destination.address", "ignore_failure": true }},  
  { "enrich":{ "description": "Enrich DNS queries info with threat intel indicators", "policy_name": "opencti-domain-enrichment-policy", "target_field": "enrichment", "field": "dns.query.name", "ignore_failure": true }},
  { "enrich":{ "description": "Enrich SSL host info with threat intel indicators", "policy_name": "opencti-domain-enrichment-policy", "target_field": "enrichment", "field": "ssl.server_name", "ignore_failure": true }},
  { "enrich":{ "description": "Enrich HTTP traffic info with threat intel indicators", "policy_name": "opencti-domain-enrichment-policy", "target_field": "enrichment", "field": "http.virtual_host", "ignore_failure": true }}
 ]
}

5. Add to the final pipelines.

• Since the file referenced is no longer available '/opt/so/saltstack/local/pillar/global.sls', I did it in "Administration > Configuration > Elasticsearch" and added to the import template then reviewed where the change was made and found it in "/opt/so/saltstack/local/pillar/elasticsearch/soc_elasticsearch.sls" and added another setting for 'so-zeek'.

elasticsearch:
  index_settings:
    so-import:
      index_template:
        template:
          settings:
            index:
              final_pipeline: enrich.misp
              number_of_shards: 1
      warm: 7
      close: 45
      delete: 365

My setup

    index_settings:
        so-import:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich
        so-logs-pfsense_x_log:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich
        so-zeek:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich

When that is all done i reboot and look in the index "logs-ti_opencti_latest.dest_indicator-*" index for an IP and host. I test with a ping or a lookup. I see the Zeek data but nothing ever get attached to enrich the data. Since then, I made a grave mistake in one of my instances. I thought maybe if i delete the OpenCTI Index Template and Component Template, that might fix the match error. It did not. I assumed if I removed and reinstalled the Fleet integration it would put them back and it didn't.

1. Is what I am trying to accomplish, enrich my Zeek data with threat intel indicators possible in SO 2.4.120+?
2. What am I missing or doing wrong here?
3. Is there a way to get the Templates back?

Thanks as always all.

[reyesj2]

The .ds indices are part of a datastream. Just think of it as part of an overall index. Try using the datastream name in your enrich policy like

PUT /_enrich/policy/opencti-ip-enrichment-policy
{
 "match":{
 "indices":"logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.ip",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

Execute the policy and you should have an enrich index created, make sure it has docs in it not just empty (if its empty make sure the policy executed with an OK status)

If you set ignore failure to false do you get errors showing the enrich pipeline is being run, but just data is malformed in someway?

To get the templates back for opencti if you go to the integrations page and click settings there is a 'reinstall' option

[Slipperyclock]

Thank you for the advice. So I went back and put the policies in place as below. When I go to execute the polices I get the same error. Thanks for the heads up on "reinstall" that restored everything.

Policy creation: both return OK.

PUT /_enrich/policy/opencti-ip-enrichment-policy
{
 "match":{
 "indices":"logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.ip",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

PUT /_enrich/policy/opencti-domain-enrichment-policy
{
 "match":{
 "indices":"logs-ti_opencti.indicator-default-*",
 "match_field":"threat.indicator.url.domain",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.confidence",
  "opencti.indicator.score",
  "tags"
  ]
 }
}

Execution of Policy

POST /_enrich/policy/opencti-ip-enrichment-policy/_execute

POST /_enrich/policy/opencti-domain-enrichment-policy/_execute

[Slipperyclock]

For a follow up I did the same settings for AbuseCH and it worked as expected for Domains. It seems to be an issue with the way SO does the mappings for threat.indicator.ip and threat.indicator.url.domain for opencti integration. I tested this and the intel data does get enriched in Zeek data.

AbuseCH Policy

PUT /_enrich/policy/ti-abusech-url-domain-policy
{
 "match":{
 "indices":".ds-logs-ti_abusech.url-*",
 "match_field":"threat.indicator.url.domain",
 "enrich_fields":[
  "threat.indicator.provider",
  "threat.indicator.type",
  "threat.indicator.url.original",
  "threat.indicator.reference",
  "tags"
  ]
 }
}

Execution (Worked as expected)

POST /_enrich/policy/ti-abusech-url-domain-policy/_execute

Pipeline

{ 
 "description" : "Threat Enrichment",
 "processors" : [
  { "enrich":{ "description": "AbuseCH Enrich DNS queries info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "dns.query.name", "ignore_failure": true }},
  { "enrich":{ "description": "AbuseCH Enrich SSL host info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "ssl.server_name", "ignore_failure": true }},
  { "enrich":{ "description": "AbuseCH Enrich Meraki log info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "url.domain", "ignore_failure": true }},  
  { "enrich":{ "description": "AbuseCH Enrich HTTP traffic info with threat intel indicators", "policy_name": "ti-abusech-url-domain-policy", "target_field": "enrichment", "field": "http.virtual_host", "ignore_failure": true }}
 ]
}

Final Pipeline Settings

        so-logs-cisco_meraki_x_log:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich
        so-zeek:
            index_template:
                template:
                    settings:
                        index:
                            final_pipeline: threat.enrich