-
Notifications
You must be signed in to change notification settings - Fork 717
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide mechanism for cleanup after failed deployment to enable re-deployment #1240
Comments
Hi @mromascanu123 , can you help me understand more about your desired outcome, and in what scenario you want to use this script? Is it something that isn't already addressed by using the helper script to automate the manual steps of deploying with Cloud Build, then destroying? While there are some flaky errors that require unpicking state like #1187, they are specific enough that I don't recommend creating a script to directly modify terraform state. (Usually modifying terraform state files by any method other than |
Hi @eeaton. One of the issues I've seen is the persistence of the ** resource "random_string" ** in the tfstate when the "plan" decides to delete and recreate a project following a failed deployment. The project-factory module will attempt to recreate the roject but the resulting id will be the same as the one of the just-deleted project and obviously will fail. I was able to reproduce the issue at least once by aborting a tf-wrapper apply with a Ctrl-C then re-planning and re-applying. But a failed deployment may occur for many other reasons. The resource "random_string" is being used in many places and it persists after deletion of the actual resource using it tgo generate a suffix but afaik the projects and KMS keystores persist after being deleted and their name / id can't be reused in tf-wrapper.sh "list" and "remove" operations could be added to list the resource IDs in tfstate and e.g. selectively delete as necessary the random_string resources which served to generate IDs for resources deleted but still zombified |
Another issue seen repeatedly is described in #1228 - even when retrying with tf-wrapper plan and then apply and apparently succeeding, in reality this ends up in tfstate corruption for that particular job (e.g.development under 3-nhas). In this case, for safer recovery must delete all created resources by that job and also the corresponding resource IDs in tfstate and redo the plan and apply, obviously crossing fingers not to hit the snag again. |
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days |
TL;DR
Need to selectively remove from the environment and from tfstate the already created resources after a failed deployment .This has two sides:
For the first item this would be a script replicating the manual steps below
For the second item add in tf-wrapper.sh 2 options :
(will have to extract from the resulting list the resources ID to clean-up)
Terraform Resources
Detailed design
See TL;DR* above
Additional information
Related #1238
The text was updated successfully, but these errors were encountered: