[Discussion and action required] ec2 data collection workflows (need links to Kepler)

[sunya-ch]

What would you like to be added?

We need a bot to check the change on Kepler whether it has significant changes (affecting exported metrics) merged to the Kepler repository. When that happens, we need to recollect the data from our machines (currently on ec2 spot instances).

Since the machine can be costly, we need to properly activate the collection.
Here is my thought. Let's put yours and discuss here. I would like push this by this quarter if possible.

• Push notification on slack of status unsync when detected the significant change waiting to activate the workflow once a week or two weeks in routine. So, the collect workflow will be activate only once a week (or two weeks) in the case of there is potential of unsync metrics.

@rootfs @SamYuan1990 @sthaha @KaiyiLiu1234 @vimalk78

---

For the workflow, I have initiated one in as a dispatch workflow: https://github.com/sustainable-computing-io/kepler-model-server/actions/workflows/collect-train.yml

We still need to:

• investigate the feasibility of the instance profile and add to the list (reusing https://github.com/sustainable-computing-io/kepler-model-training-playbook)
• add CI to push the updated model to the model db (currently is GitHub repository but considering to use large file storage: Need of large file storage  kepler-model-db#18)
• need to archive the models when release.

Why is this needed?

We should provide the common power model up to date to the latest exported metrics.

[sunya-ch]

Moved to track in sustainable-computing-io/kepler#1910