[Security Solution] Error when importing a duplicated Endpoint Security rule with default exceptions

[banderror]

Related to: #143864

Summary

If you try to import a duplicated Endpoint Security rule (so, it means it's a custom one) that has an exception item added to its default exception list (not the endpoint list), then overall the import fails, and:

• the rule gets imported
• the exception list doesn't

Steps to reproduce:

1. Install the prebuilt Endpoint Security rule.
2. Duplicate it. Delete the original Endpoint Security rule. Rename the duplicate from Endpoint Security [Duplicate] to Endpoint Security (not sure if the renaming matters).
3. At this point, you have a custom rule called Endpoint Security.
4. Add an exception item to its default exception list (Rule Details page -> Rule exceptions tab).
5. Export this rule. This will generate an ndjson file containing the rule + its default exception list + it's exception item.
6. Delete the Endpoint Security rule.
7. Try to import the ndjson file.
8. See the error.

Details

Use this minimal ndjson to reproduce the bug:

{"id":"ed9732ef-5a40-4f9c-8f91-524598b59b59","updated_at":"2024-10-31T02:56:54.261Z","updated_by":"elastic","created_at":"2024-10-31T02:55:36.680Z","created_by":"elastic","name":"Endpoint Security","tags":["Data Source: Elastic Defend"],"interval":"5m","enabled":false,"revision":3,"description":"Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":"","meta":{"from":"5m","kibana_siem_app_url":""},"rule_name_override":"message","timestamp_override":"event.ingested","timestamp_override_fallback_disabled":false,"author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"e456b162-a64f-4b62-a344-ca3cd589cf6d","max_signals":10000,"risk_score_mapping":[{"field":"event.risk_score","operator":"equals","value":""}],"severity_mapping":[{"field":"event.severity","operator":"equals","severity":"low","value":"21"},{"field":"event.severity","operator":"equals","severity":"medium","value":"47"},{"field":"event.severity","operator":"equals","severity":"high","value":"73"},{"field":"event.severity","operator":"equals","severity":"critical","value":"99"}],"threat":[],"to":"now","references":[],"version":103,"exceptions_list":[{"id":"endpoint_list","list_id":"endpoint_list","type":"endpoint","namespace_type":"agnostic"},{"id":"2618dcb5-7bb2-4436-9c68-fd75208feaa4","list_id":"3fa90098-d556-4c38-863f-644c99766126","type":"rule_default","namespace_type":"single"}],"immutable":false,"rule_source":{"type":"internal"},"related_integrations":[{"package":"endpoint","version":"^8.2.0"}],"required_fields":[],"setup":"## Setup\n\nThis rule is configured to generate more **Max alerts per run** than the default 1000 alerts per run set for all rules. This is to ensure that it captures as many alerts as possible.\n\n**IMPORTANT:** The rule's **Max alerts per run** setting can be superseded by the `xpack.alerting.rules.run.alerts.max` Kibana config setting, which determines the maximum alerts generated by _any_ rule in the Kibana alerting framework. For example, if `xpack.alerting.rules.run.alerts.max` is set to 1000, this rule will still generate no more than 1000 alerts even if its own **Max alerts per run** is set higher.\n\nTo make sure this rule can generate as many alerts as it's configured in its own **Max alerts per run** setting, increase the `xpack.alerting.rules.run.alerts.max` system setting accordingly.\n\n**NOTE:** Changing `xpack.alerting.rules.run.alerts.max` is not possible in Serverless projects.","type":"query","language":"kuery","index":["logs-endpoint.alerts-*"],"query":"event.kind:alert and event.module:(endpoint and not endgame)\n","filters":[],"actions":[]}
{"_version":"WzE1MjQsMV0=","created_at":"2024-10-31T02:56:12.355Z","created_by":"elastic","description":"Exception list containing exceptions for rule with id: ed9732ef-5a40-4f9c-8f91-524598b59b59","id":"2618dcb5-7bb2-4436-9c68-fd75208feaa4","immutable":false,"list_id":"3fa90098-d556-4c38-863f-644c99766126","name":"Exceptions for rule - Endpoint Security [Duplicate]","namespace_type":"single","os_types":[],"tags":["default_rule_exception_list"],"tie_breaker_id":"5bc67024-9c58-4912-b78d-c1962593f7dd","type":"rule_default","updated_at":"2024-10-31T02:56:12.355Z","updated_by":"elastic","version":1}
{"_version":"WzE1MjUsMV0=","comments":[],"created_at":"2024-10-31T02:56:12.878Z","created_by":"elastic","description":"Exception list item","entries":[{"type":"match","field":"test","value":"test","operator":"included"}],"id":"3f478fb9-39b2-421d-b64d-689ed3c77503","item_id":"ee0b2657-e4bf-4d02-9528-d92e5eb8aa26","list_id":"3fa90098-d556-4c38-863f-644c99766126","name":"test","namespace_type":"single","os_types":[],"tags":[],"tie_breaker_id":"f1391be9-a2a9-4134-bbad-035a91cdc9f9","type":"simple","updated_at":"2024-10-31T02:56:12.878Z","updated_by":"elastic"}
{"exported_count":3,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":1,"exported_exception_list_item_count":1,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0,"exported_action_connector_count":0,"missing_action_connection_count":0,"missing_action_connections":[],"excluded_action_connection_count":0,"excluded_action_connections":[]}

When you try to import it, it will call the following import API endpoint which will respond with the following error:

POST /api/detection_engine/rules/_import?overwrite=false&overwrite_exceptions=false&overwrite_action_connectors=false

{
    "exceptions_success": false,
    "exceptions_success_count": 0,
    "exceptions_errors": [
        {
            "list_id": "3fa90098-d556-4c38-863f-644c99766126",
            "error": {
                "status_code": 409,
                "message": "Found that list_id: \"3fa90098-d556-4c38-863f-644c99766126\" already exists. Import of list_id: \"3fa90098-d556-4c38-863f-644c99766126\" skipped."
            }
        },
        {
            "list_id": "3fa90098-d556-4c38-863f-644c99766126",
            "item_id": "ee0b2657-e4bf-4d02-9528-d92e5eb8aa26",
            "error": {
                "status_code": 409,
                "message": "Found that item_id: \"ee0b2657-e4bf-4d02-9528-d92e5eb8aa26\" already exists. Import of item_id: \"ee0b2657-e4bf-4d02-9528-d92e5eb8aa26\" skipped."
            }
        }
    ],
    "rules_count": 1,
    "success": false,
    "success_count": 1,
    "errors": [
        {
            "rule_id": "e456b162-a64f-4b62-a344-ca3cd589cf6d",
            "error": {
                "status_code": 400,
                "message": "Rule with rule_id: \"e456b162-a64f-4b62-a344-ca3cd589cf6d\" references a non existent exception list of list_id: \"endpoint_list\". Reference has been removed."
            }
        },
        {
            "rule_id": "e456b162-a64f-4b62-a344-ca3cd589cf6d",
            "error": {
                "status_code": 400,
                "message": "Rule with rule_id: \"e456b162-a64f-4b62-a344-ca3cd589cf6d\" references a non existent exception list of list_id: \"3fa90098-d556-4c38-863f-644c99766126\". Reference has been removed."
            }
        }
    ],
    "action_connectors_errors": [],
    "action_connectors_warnings": [],
    "action_connectors_success": true,
    "action_connectors_success_count": 0
}

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)