helm.v4.Chart: hook support

[awoimbee]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

helm.v3.Chart ignored the helm.sh/hook* annotations but still deployed the resources.
The helm.v4.Chart resource skips the hooks. This is mentioned in the blog but not in the docs BTW.

I think users should have a way to make v4.Chart deploy hooked resources like v3 did, because it's useful and becasue it makes migrating from v3 to v4 less painful.

[EronWright]

I think the issue is that hook resources have special lifecycle/ordering requirements. Basically would they work as expected?

[awoimbee]

Yep, I have multiple charts that stopped working when moving from v3 to v4:

• https://charts.min.io/ minio (the create bucket job did not run anymore, I'm using the old v4 chart)
• https://kubernetes.github.io/ingress-nginx ingress-nginx (the patch job for the admission webhook)
• https://kubernetes-sigs.github.io/aws-efs-csi-driver/ efs-csi-driver
• https://charts.wiz.io/ wiz-kubernetes-integration
• https://trow-registry.github.io/trow trow (the patch job just like ingress-nginx)

The issue I have with just removing the hooks is that I can't get the templated yaml out of pulumi.
If I want to remove a hook, I can do it myself, using something like (this v3 transform):

export const ignoreResourcesHelmTransformation = (...resources: string[]) => {
  const stack = pulumi.getStack();
  return (o: any, opts: pulumi.CustomResourceOptions) => {
    const resourceName = `${o.apiVersion}:${o.kind}:${o.metadata?.namespace || ""}:${o.metadata?.name || ""}`;
    if (resources.includes(resourceName)) {
      console.log(`Skip: ${resourceName}`);
      const dir = `../ignored-resources-${stack}`;
      if (!fsSync.existsSync(dir)) {
        fsSync.mkdirSync(dir);
      }
      fsSync.writeFileSync(
        `${dir}/${resourceName.replaceAll("/", "-")}.yaml`,
        yaml.dump(o)
      );
      for (const key of Object.keys(o)) {
        delete o[key];
      }
      o.apiVersion = "v1";
      o.kind = "List";
      o.items = [];
    }
  };
};

[EronWright]

(here's an outline of the technical challenge)

The Helm documentation outlines the various hook behaviors that Pulumi would be expected to emulate:
https://helm.sh/docs/topics/charts_hooks/#the-available-hooks

In v3, Pulumi simply treated the hooks as normal resources, which can lead to incorrect behavior.

As we can see, hooks generally affect the order of operations, and also vary based on whether the chart is being installed, upgraded, or deleted. The Pulumi engine provides the dependsOn option to control order-of-operations between resources, which is likely sufficient to support the pre/post aspect. It is less clear how to support "upgrade" hooks, since the Chart/v4 resource is stateless, it cannot distinguish between install and upgrade. Also unclear how to support "delete" hooks, since the Chart/v4 resource is a component resource and there's no opportunity to "create" a hook resource during component deletion.

[EronWright]

A low-hanging improvement here would be to emit a clear warning message if the chart did contain a hook. The provider does have a warning like this for Chart/v3 but isn't effective in Chart/v4.

[yann-soubeyrand]

Hello,

In my opinion, Argo CD does the right thing regarding hooks: https://argo-cd.readthedocs.io/en/latest/user-guide/helm/#helm-hooks. It makes almost every chart work as expected. I think we could achieve nearly the same with Pulumi. The only challenge is see is around the hook deletion policy, where we might have to do clever things like replaceOnChanges(*) and deleteBeforeReplace(true). However, it involves a little bit more work to implement than simply considering hooks as normal resources. @EronWright what do you think about this approach?

[EronWright]

Thanks for the information, it says that Argo CD emulates the behavior of some hooks, with caveats. I agree that, where there's opportunity to support a type of hook, we should. But, like Argo, Pulumi may struggle to distinguish installs and upgrades, etc.

[yann-soubeyrand]

Sure, the solution cannot be perfect, it’s infeasible. However, Argo CD emulation proved to be satisfying in a lot of use cases and, this is pure conjecture from myself, I think they are sufficiently big to make chart authors considering modifying their charts so that they work with Argo CD.

My question is, would you be open to consider a PR bringing the same behaviour than Argo CD to Pulumi (I anticipate this PR to be a fair amount of work, so I prefer knowing you position before trying to work on it 🙂)?