feat: Add more settings and tunability to snapshot table and dictionary

[tkaemming]

Problem

If max_memory_usage is set on the cluster client settings, the INSERT INTO statement in the populate step can fail due to the large aggregation needed to ensure we have the latest value for each override in the overrides table.

Changes

optimize_aggregation_in_order reduces the memory (which we don't have enough of) at the cost of some time (which we have plenty of.)

This also allows providing a max_memory_usage parameter to the dictionary as an additional safeguard against excessive memory usage on the cluster during job execution.

Does this work well for both Cloud and self-hosted?

Yes

How did you test this code?

Already covered by existing test, and ensured optimize_aggregation_in_order improved the peak memory usage of the relevant query in production

[greptile-apps]

PR Summary

This PR adds memory optimization settings to the person overrides squashing process to prevent out-of-memory errors during large data operations.

• Added optimize_aggregation_in_order=1 in dags/person_overrides.py to reduce memory usage during snapshot table population
• Added max_memory_usage parameter to dictionary creation for controlling memory consumption
• Added max_execution_time parameter to dictionary creation for timeout control
• Improved memory efficiency by trading execution speed for lower memory usage during aggregation steps

_{1 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile}