Limit stix_ids explosion by rewriting the standard_id in client python

[richard-julien]

Use case

Limit stix_ids explosion by rewriting the standard_id in client python.

Context

As described in OpenCTI-Platform/opencti#6774 by @ckane , because of remote system or bad designed connectors, a lot of stix ids can be cumulated in an element.

This seems to be able to result in a long sequence of entries in redis, during re-ingestion of the same entity (such as a malware STIX type) with new randomly-assigned STIX ids. My observation is that these get added to the stream once for each new STIX id encountered for the same item, and the list of stix_ids in the added/updated entity can grow infinitely. This is the source of one of the reasons why redis memory consumption can grow so high, under some circumstances, despite implementing a low TRIMMING size (such as 100000).

Solution

We need to generate the standard id on the client python and replace dynamically all the ids with the predicted ones.
By default, drop the base id of the source but add an option to configure in the connector if you want to consolidate them.

• a cleanup script for existing platform to remove all the bad stix_ids already written

Why not only a task in client python?

Its also a good moment to check and enforce the ids generation on opencti side

See OpenCTI-Platform/opencti#7158