Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: App environment not reliably updated after restart #3217

Closed
georgethebeatle opened this issue Apr 17, 2024 · 0 comments
Closed

Bug: App environment not reliably updated after restart #3217

georgethebeatle opened this issue Apr 17, 2024 · 0 comments
Assignees
@danail-branekov @georgethebeatle
Labels
bug Something isn't working flake

Comments

@georgethebeatle
Copy link
Member

georgethebeatle commented Apr 17, 2024 •
edited by danail-branekov
Loading

We are occasionally seeing the following flake in e2e tests:

| 29d   | main/run-e2es-eks-pr             | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-eks-pr/builds/1549
| 40d   | main/run-e2es-eks-pr             | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-eks-pr/builds/1521
| 38d   | main/run-e2es-eks-periodic       | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-eks-periodic/builds/6842
| 43d   | main/run-e2es-eks-pr             | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-eks-pr/builds/1513
| 31d   | main/run-e2es-periodic           | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-periodic/builds/14271
| 54d   | main/run-e2es-pr                 | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-pr/builds/2489
| 59d   | main/run-e2es-periodic           | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-periodic/builds/13764
| 61d   | main/run-e2es-main               | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-main/builds/1058
| 66d   | main/run-e2es-periodic           | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-periodic/builds/13645
| 73d   | main/run-e2es-periodic           | https://ci.korifi.cf-app.com/teams/main/pipelines/main/jobs/run-e2es-periodic/builds/13547

The error looks like this:

  [FAILED] in [It] - /tmp/build/3e139e9d/korifi/tests/e2e/apps_test.go:625 @ 03/08/24 10:28:56.244
  << Timeline

  [FAILED] Expected
      <map[string]interface {} | len:1>: {
          "d4e30b9d-1f45-4c6e-942d-c11315a11590": <map[string]interface {} | len:3>{
              "foo": <string>"bar",
              "type": <string>"user-provided",
              "baz": <string>"qux",
          },
      }
  to have length 2
  In [It] at: /tmp/build/3e139e9d/korifi/tests/e2e/apps_test.go:625 @ 03/08/24 10:28:56.244

In #3036 we make the app restart synchronous. However it is possible that the app is started and ready, but the servicebinding.io resources are not yet in the Ready state (meanging that the pod env is not updated).

We think that introducing Ready states in cfservicebinding, cfapp and making the relevant controllers synchronize on the Ready state of their dependent resouces could fix this issue. Related to #2098
@georgethebeatle georgethebeatle added bug Something isn't working flake labels Apr 17, 2024
@github-project-automation github-project-automation bot moved this to 🧊 Icebox in Korifi - Backlog Apr 17, 2024
@danail-branekov danail-branekov changed the title Flake/Bug: App environment not updated after restart Bug: App environment not reliably updated after restart Apr 17, 2024
@georgethebeatle georgethebeatle moved this from 🧊 Icebox to 🇪🇺 To do in Korifi - Backlog Apr 17, 2024
@danail-branekov danail-branekov moved this from 🇪🇺 To do to 🔄 In progress in Korifi - Backlog Apr 23, 2024
danail-branekov added a commit that referenced this issue Apr 24, 2024
@danail-branekov @georgethebeatle
Introduce Ready status condition for CFServiceBinding
99cbb2a 
* The CFServiceBinding becomes ready when the underlying
  servicebinding.io bindings become ready
* The rest of the CFServiceBinding status conditions have been removed
  (as other controllers and API should only care whether the binding is
  ready or not)
* Introduce a reusable `ReadyConditionBuilder` utility used to set the
  object status

issue: #3217

Co-authored-by: Georgi Sabev <[email protected]>
georgethebeatle added a commit that referenced this issue Apr 25, 2024
@danail-branekov @georgethebeatle
Introduce Ready status condition for CFServiceBinding
3080f03 
* The CFServiceBinding becomes ready when the underlying
  servicebinding.io bindings become ready
* The rest of the CFServiceBinding status conditions have been removed
  (as other controllers and API should only care whether the binding is
  ready or not)
* Introduce a reusable `ReadyConditionBuilder` utility used to set the
  object status

issue: #3217

Co-authored-by: Georgi Sabev <[email protected]>
@georgethebeatle georgethebeatle mentioned this issue Apr 25, 2024
Merged
danail-branekov added a commit that referenced this issue Apr 25, 2024
@danail-branekov @georgethebeatle
Introduce Ready status condition for CFServiceBinding
bc80f7c 
* The CFServiceBinding becomes ready when the underlying
  servicebinding.io bindings become ready
* The rest of the CFServiceBinding status conditions have been removed
  (as other controllers and API should only care whether the binding is
  ready or not)
* Introduce a reusable `ReadyConditionBuilder` utility used to set the
  object status

issue: #3217

Co-authored-by: Georgi Sabev <[email protected]>
danail-branekov added a commit that referenced this issue Apr 25, 2024
@danail-branekov @georgethebeatle
Introduce Ready status condition for CFServiceBinding
f78f69c 
* The CFServiceBinding becomes ready when the underlying
  servicebinding.io bindings become ready
* The rest of the CFServiceBinding status conditions have been removed
  (as other controllers and API should only care whether the binding is
  ready or not)
* Introduce a reusable `ReadyConditionBuilder` utility used to set the
  object status

issue: #3217

Co-authored-by: Georgi Sabev <[email protected]>
danail-branekov added a commit that referenced this issue Apr 30, 2024
@danail-branekov
App stop/start waits for service bindings readiness
a0717cc 
* The app controller sets the ready state if all of the criteria below
  are met:
  - app service bindings (if any) become ready
  - the app actual state is equal to the desired state
  - the current droplet is set
* The app repository waits for the app `Ready` state in
  `SetAppDesiredState` to ensure synchronous app stop/start
* The condition awaiter verifies that the condition observed generation
  matches the object generation. Thus we ensure that the condition is
  not outdated.

Co-authored-by: Georgi Sabev <[email protected]>

fixes: #3217
danail-branekov added a commit that referenced this issue Apr 30, 2024
@danail-branekov
App stop/start waits for service bindings readiness
1f22141 
* The app controller sets the ready state if all of the criteria below
  are met:
  - app service bindings (if any) become ready
  - the app actual state is equal to the desired state
  - the current droplet is set
* The app repository waits for the app `Ready` state in
  `SetAppDesiredState` to ensure synchronous app stop/start
* The condition awaiter verifies that the condition observed generation
  matches the object generation. Thus we ensure that the condition is
  not outdated.

Co-authored-by: Georgi Sabev <[email protected]>

fixes: #3217
@georgethebeatle georgethebeatle mentioned this issue Apr 30, 2024
Merged
danail-branekov added a commit that referenced this issue Apr 30, 2024
@danail-branekov
App stop/start waits for service bindings readiness
c5de5f6 
* The app controller sets the ready state if all of the criteria below
  are met:
  - app service bindings (if any) become ready
  - the app actual state is equal to the desired state
  - the current droplet is set
* The app repository waits for the app `Ready` state in
  `SetAppDesiredState` to ensure synchronous app stop/start
* The condition awaiter verifies that the condition observed generation
  matches the object generation. Thus we ensure that the condition is
  not outdated.

Co-authored-by: Georgi Sabev <[email protected]>

fixes: #3217
@github-project-automation github-project-automation bot moved this from 🔄 In progress to ✅ Done in Korifi - Backlog Apr 30, 2024
marsteg pushed a commit to marsteg/korifi that referenced this issue May 6, 2024
@danail-branekov @marsteg
App stop/start waits for service bindings readiness
9fb2dad 
* The app controller sets the ready state if all of the criteria below
  are met:
  - app service bindings (if any) become ready
  - the app actual state is equal to the desired state
  - the current droplet is set
* The app repository waits for the app `Ready` state in
  `SetAppDesiredState` to ensure synchronous app stop/start
* The condition awaiter verifies that the condition observed generation
  matches the object generation. Thus we ensure that the condition is
  not outdated.

Co-authored-by: Georgi Sabev <[email protected]>

fixes: cloudfoundry#3217
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@danail-branekov danail-branekov

@georgethebeatle georgethebeatle

Labels
bug Something isn't working flake
Projects
Korifi - Backlog
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@danail-branekov @georgethebeatle