Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFE] Rather than Day of the week can I specify a date instead #203

Open
SteveWoodTi2 opened this issue Apr 30, 2024 · 7 comments
Open

[RFE] Rather than Day of the week can I specify a date instead #203

SteveWoodTi2 opened this issue Apr 30, 2024 · 7 comments

Comments

@SteveWoodTi2
Copy link

Current situation

I'd like to have my non prod clusters run the just published updates before they are in prod

Impact

According the docs updates are around the 16th. So I'd like to specifcy my reboot window like so -reboot-window-start 16th 23:00 and for prod 30th 23:00

Ideal future situation

This means no untested version updates are running in prod.

Implementation options

See above

Additional information

[ Please Add any information that does not fit into any of the above sections here ]

@invidian
Copy link
Member

I think that sounds reasonable. We should be able to extend our periodic system with that if we agree on some reasonable notation.

@SteveWoodTi2
Copy link
Author

The simplicity of this approach should mean that any OS breaking updates will only effect my non prod clusters and give us enough time to revert to the prod cluster back to the previous version before it causes a unplanned outage which tend to get noticed. Also makes planning and general management of updates a whole lot easier.

@jepio
Copy link
Member

jepio commented Apr 30, 2024

@SteveWoodTi2 have you thought of running beta nodes in non-production? That would help you catch issues ahead of when they hit stable.

That's the recommendation we generally give users.

@pothos
Copy link
Member

pothos commented May 2, 2024

I think what Cincinnati has is also interesting, it was something like a number that defines the order for clients to update by having them wait longer. I think it was implemented server-side but one can also do a client side variant of this where we would have a delay that you can define. The request in this issue here about hardcoded dates wouldn't really help because when the release is delayed, both groups would still update at the same time.

In the current way the update operators work they only delay the reboot. Defining a minimum reboot wait time instead of the reboot window could be an easy addition to get an order. Best would be if we would make update-engine have an update delay and update windows, so that we don't even stage the update before that. Currently you'll still end up with updated prod if you happen to reboot before the scheduled time.

@pothos
Copy link
Member

pothos commented May 2, 2024

That's the recommendation we generally give users.

This, and that one can also run an own Nebraska server if fine grained control is needed.

@SteveWoodTi2
Copy link
Author

Our non prod is in fact classified as prod as it delivers services to developers not customers like prod prod does so using the beta channel has been vetoed by people above my pay grade.

@SteveWoodTi2
Copy link
Author

Just to add, that if we could specify a date rather than a day then even if the update is late and we haven't got an alert about our non prod nodes rebooting before prod is due then it's just a case of scaling down the reboot operator then scale it backup after the date the reboot window has been set at. Seems pretty simple to me unless I'm missing something ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants