Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backend] add exclusion cache handling (#8941) #9079

Merged
merged 4 commits into from
Nov 27, 2024

Conversation

ValentinBouzinFiligran
Copy link
Member

@ValentinBouzinFiligran ValentinBouzinFiligran commented Nov 19, 2024

Proposed changes

  • add cache file containing local exclusion list cache
  • add cache init at start of platform, reading current cache from redis if it exists
  • add cache build manager, building new cache when an exclusion list is changed and pushing it to redis
  • add cache sync manager, reading cache status on redis and updating local cache if it is outdated

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case (coverage and e2e)
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

@ValentinBouzinFiligran ValentinBouzinFiligran added the filigran team use to identify PR from the Filigran team label Nov 19, 2024
@JeremyCloarec JeremyCloarec changed the base branch from master to release/6.5.0 November 20, 2024 14:40
@JeremyCloarec JeremyCloarec force-pushed the issue/8941-chunk3bis branch 2 times, most recently from f35b66c to 6ce404a Compare November 22, 2024 13:42
@JeremyCloarec JeremyCloarec changed the title [backend] add exclusionListCacheBuildManager [backend] add exclusion cache handling Nov 22, 2024
@JeremyCloarec JeremyCloarec marked this pull request as ready for review November 22, 2024 15:16
Copy link

codecov bot commented Nov 25, 2024

Codecov Report

Attention: Patch coverage is 48.90830% with 117 lines in your changes missing coverage. Please review.

Project coverage is 65.63%. Comparing base (402d6bc) to head (79957d3).
Report is 14 commits behind head on release/6.5.0.

Files with missing lines Patch % Lines
...aphql/src/manager/exclusionListCacheSyncManager.ts 0.00% 36 Missing ⚠️
...phql/src/manager/exclusionListCacheBuildManager.ts 0.00% 33 Missing ⚠️
.../src/modules/exclusionList/exclusionList-domain.ts 48.38% 16 Missing ⚠️
...opencti-graphql/src/database/exclusionListCache.ts 77.41% 14 Missing ⚠️
...cti-platform/opencti-graphql/src/database/redis.ts 73.68% 5 Missing ⚠️
...-graphql/src/modules/indicator/indicator-domain.ts 58.33% 5 Missing ⚠️
...cti-platform/opencti-graphql/src/initialization.js 0.00% 3 Missing ⚠️
...ncti-platform/opencti-graphql/src/manager/index.ts 0.00% 2 Missing ⚠️
...rc/modules/exclusionList/exclusionList-resolver.ts 50.00% 2 Missing ⚠️
...c/modules/exclusionList/exclusionList-converter.ts 0.00% 1 Missing ⚠️
Additional details and impacted files
@@                Coverage Diff                @@
##           release/6.5.0    #9079      +/-   ##
=================================================
+ Coverage          64.97%   65.63%   +0.66%     
=================================================
  Files                623      626       +3     
  Lines              59496    59885     +389     
  Branches            6577     7195     +618     
=================================================
+ Hits               38656    39308     +652     
+ Misses             20840    20577     -263     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JeremyCloarec JeremyCloarec changed the title [backend] add exclusion cache handling [backend] add exclusion cache handling (#8941) Nov 25, 2024

export const buildCacheFromAllExclusionLists = async (context: AuthContext) => {
const exclusionLists: BasicStoreEntityExclusionList[] = await listAllEntities(context, SYSTEM_USER, [ENTITY_TYPE_EXCLUSION_LIST]);
const enabledExclusionLists = exclusionLists.filter((l) => l.enabled);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filter by enabled

import { getIsCacheInitialized, syncExclusionListCache } from '../database/exclusionListCache';

const EXCLUSION_LIST_CACHE_SYNC_MANAGER_LOCK_KEY = conf.get('exclusion_list_cache_sync_manager:lock_key') || 'exclusion_list_cache_sync_manager_lock';
const EXCLUSION_LIST_CACHE_SYNC_MANAGER_NODE_LOCK_KEY = `${EXCLUSION_LIST_CACHE_SYNC_MANAGER_LOCK_KEY}_${NODE_INSTANCE_ID}`;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for next version : try to remove the lock

@SouadHadjiat
Copy link
Member

missing translations (list of managers in parameters page) :

image

@SouadHadjiat
Copy link
Member

I have this error when trying to access "created_at" on an exclusion list by API
Cannot return null for non-nullable field ExclusionList.created_at

@JeremyCloarec JeremyCloarec merged commit 0042e77 into release/6.5.0 Nov 27, 2024
5 checks passed
@JeremyCloarec JeremyCloarec deleted the issue/8941-chunk3bis branch November 27, 2024 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filigran team use to identify PR from the Filigran team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants