Occurence of the object has been modified on Stack resource

[ringods]

What happened?

Customer sees this error after applying a Stack resource to the cluster:

pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 2025-02-10T21:16:44.756Z	ERROR	Reconciler error	{"controller": "stack-controller", "namespace": "c1c824c-staging", "name": "s1-pko", "reconcileID": "8c3e1cd7-54f5-4291-9a1e-b947e2f96ea6", "error": "Operation cannot be fulfilled on stacks.pulumi.com \"s1-pko\": the object has been modified; please apply your changes to the latest version and try again"}
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 2025-02-10T21:16:44.786Z	ERROR	updating status	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"s1-pko","namespace":"c1c824c-staging"}, "namespace": "c1c824c-staging", "name": "s1-pko", "reconcileID": "1dc1b287-5cef-4865-aa1f-5289aec8cbcf", "revision": "171742985", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"s1-pko\": the object has been modified; please apply your changes to the latest version and try again"}
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile.func1
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/operator/internal/controller/auto/workspace_controller.go:113
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager github.com/pulumi/pulumi-kubernetes-operator/v2/operator/internal/controller/auto.(*WorkspaceReconciler).Reconcile
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/operator/internal/controller/auto/workspace_controller.go:198
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 2025-02-10T21:16:44.786Z	ERROR	Reconciler error	{"controller": "workspace-controller", "controllerGroup": "auto.pulumi.com", "controllerKind": "Workspace", "Workspace": {"name":"s1-pko","namespace":"c1c824c-staging"}, "namespace": "c1c824c-staging", "name": "s1-pko", "reconcileID": "1dc1b287-5cef-4865-aa1f-5289aec8cbcf", "error": "Operation cannot be fulfilled on workspaces.auto.pulumi.com \"s1-pko\": the object has been modified; please apply your changes to the latest version and try again"}
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
pko-c1c824c-staging-controller-manager-6967cb6f75-pjcjt manager 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

Example

Customer applied this Stack:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: s1-pko
  namespace: c1c824c-staging
spec:
  serviceAccountName: pulumi-mateus
  branch: main
  envRefs:
    PULUMI_CONFIG_PASSPHRASE:
      literal:
        value: ""
      type: Literal
  gitAuth:
    accessToken:
      secret:
        key: password
        name: github-app-cloud-provisioning-bot-token
        namespace: c1c824c-staging
      type: Secret
  projectRepo: https://github.com/<redacted>/cloud-provisioning-recipes.git
  repoDir: /random-pet
  resyncFrequencySeconds: 300
  serviceAccountName: random-yaml
  stack: organization/random-pet/s1

Output of pulumi about

Customer installed the operator via Helm:

      chart: "./deploy/helm/pulumi-operator"
      git:
        tag: "v2.0.0-beta.3"

Additional context

Related:

• Clean up "object has already been modified" warnings #700 #717

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[EronWright]

Thanks for the report.

[mateusvtt]

The problem was because we were migrating from V1 with a scenario involving two Pulumi Operators separated by namespaces. In V2, this is no longer allowed:

With the 2.0 release, the supported installation modes are limited to a cluster-wide installation. You’ll need to be a cluster admin to install the Operator. The Operator is able to handle stacks across all the namespaces of your cluster.

Another point of attention for those using Helm and S3 as backend state and are migrating to V2 is that previously, in our values.yaml, we had:

extraEnv:
  - name: PULUMI_BACKEND_URL
    value: "<s3-bucket-path>"

However, this is not propagated to the new workspace pods. In our case, it was necessary to include the environment variable in the Stack objects like this:

envRefs:
  PULUMI_BACKEND_URL:
    literal:
      value: "<s3-bucket-path>""

Additionally, the method for setting the log level has changed. It took us some time to realize this, which delayed our debugging efforts. Previously, we used:

controller:
  args: ["--zap-level=debug"]

Now, it has changed to:

controller:
  logLevel: debug

[EronWright]

Thanks @mateusvtt, I took some time to incorporate your feedback into the migration guide.

Since beta 3, some of the causes of the "object has been modified" message were fixed. The message doesn't necessarily represent a problem, just a symptom. I'm going to close this now, and please re-open if you have more details after updating to the latest build.