Implement dynamic loading rather than static intervals on data page

[TeachMeTW]

Feature Request: Implement Dynamic Loading on Data Page

Description

Currently, the data page refreshes at a static interval of n seconds after updates to the UUID table. However, this approach is inefficient and may cause issues, especially as the environment scales.

• Local Processing Time: ~10 seconds, currently set to 20 seconds for leeway
• Staging/Production Processing Time: Potentially longer, which increases the risk of unnecessary refreshes and unsafe data handling.

Problem

A static interval refresh is not adaptive to the actual processing time. This can lead to inefficient use of resources, especially when processing times vary across environments (local, staging, and production). By the time the static interval kicks in:

• In local environments, refresh occurs much later than needed, wasting time.
• In staging/production, the refresh may happen too early, leading to incomplete data refresh and possibly unstable application behavior.

Proposed Solution

Implement dynamic loading based on actual completion of data processing rather than a fixed time interval. This approach would ensure:

1. Efficiency: Only refreshing the data when it’s actually updated.
2. Safer Operations: Minimizing the risk of refreshing during incomplete processing in more complex environments.

Benefits

• Optimized Performance: Tailored refresh times based on processing speed.
• Scalability: More adaptable for larger datasets and varying processing times across different environments.
• Reduced Resource Usage: Avoid unnecessary refresh cycles.

Tasks

• Investigate appropriate triggers for dynamic refresh (e.g., based on completion of updates in the UUID table).
• Implement the dynamic refresh logic, replacing the current static n-second interval.
• Test across environments to ensure proper functionality and performance.

[TeachMeTW]

@shankari This may be one of the issues staging/prod would run into regarding the batch loading from #124 . If it does NOT work as intended, this may be a possible cause and would be a high priority issue -- otherwise, if things perform normally, then what would you say about this?

[TeachMeTW]

Currently, loading can take 10-50s depending on server state; dynamic loading is important.