-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: MakeDefaultConfig once #5082
Conversation
Hi @chlunde. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR looks good to me. I have some minor feedback.
/ok-to-test |
This PR has multiple commits, and the default merge method is: merge. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
rebased for Go 1.20/CI fix |
LGTM +1 |
/kind cleanup |
Hi @chlunde |
@koba1t sorry for the delay, I removed the extra blank line now |
@natasha41575 is there anything I should do to move this forward? |
Thank you for your hard work 🎉 /lgtm |
/unassign |
@natasha41575 Could you rerun CI? |
@chlunde can you rebase to rerun CI? I think once all the tests run & pass we will be good to go here |
This shaves of another 2 seconds (62%) of the remaining execution time for a kustomization tree with 4000 documents, reducing the execution time from 4.79s to 1.82s 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.MakeDefaultConfig before: (pprof) top30 -cum Showing nodes accounting for 1.82s, 22.75% of 8s total Dropped 408 nodes (cum <= 0.04s) Showing top 30 nodes out of 308 flat flat% sum% cum cum% 0 0% 0% 4.79s 59.88% github.com/spf13/cobra.(*Command).Execute 0 0% 0% 4.79s 59.88% github.com/spf13/cobra.(*Command).ExecuteC 0 0% 0% 4.79s 59.88% github.com/spf13/cobra.(*Command).execute 0 0% 0% 4.79s 59.88% main.main 0 0% 0% 4.79s 59.88% runtime.main 0 0% 0% 4.79s 59.88% sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1 0 0% 0% 4.22s 52.75% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run 0 0% 0% 4.18s 52.25% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline) 0 0% 0% 4.18s 52.25% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap 0 0% 0% 4.06s 50.75% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget 0 0% 0% 4.06s 50.75% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources 0 0% 0% 4.06s 50.75% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget 0 0% 0% 4.05s 50.62% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateDirectory 0 0% 0% 3.22s 40.25% runtime.systemstack 0 0% 0% 3.03s 37.88% runtime.gcBgMarkWorker 0 0% 0% 3.03s 37.88% runtime.gcBgMarkWorker.func2 0.11s 1.38% 1.38% 3.03s 37.88% runtime.gcDrain 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.MakeDefaultConfig 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.MakeTransformerConfig 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.makeTransformerConfigFromBytes 0 0% 1.38% 2.97s 37.12% sigs.k8s.io/yaml.yamlUnmarshal 0 0% 1.38% 2.91s 36.38% sigs.k8s.io/yaml.Unmarshal (inline) 1.34s 16.75% 18.12% 2.87s 35.88% runtime.scanobject 0 0% 18.12% 2.53s 31.62% sigs.k8s.io/yaml.yamlToJSON 0 0% 18.12% 1.91s 23.88% gopkg.in/yaml%2ev2.unmarshal 0 0% 18.12% 1.86s 23.25% gopkg.in/yaml%2ev2.Unmarshal (partial-inline) 0 0% 18.12% 1.43s 17.88% gopkg.in/yaml%2ev2.(*parser).parse 0.01s 0.12% 18.25% 1.27s 15.88% gopkg.in/yaml%2ev2.(*parser).document 0.01s 0.12% 18.38% 1.25s 15.62% gopkg.in/yaml%2ev2.(*parser).mapping 0.35s 4.38% 22.75% 1.21s 15.12% runtime.mallocgc after: (pprof) top30 -cum Showing nodes accounting for 0.84s, 24.42% of 3.44s total Dropped 225 nodes (cum <= 0.02s) Showing top 30 nodes out of 345 flat flat% sum% cum cum% 0 0% 0% 1.82s 52.91% github.com/spf13/cobra.(*Command).Execute 0 0% 0% 1.82s 52.91% github.com/spf13/cobra.(*Command).ExecuteC 0 0% 0% 1.82s 52.91% github.com/spf13/cobra.(*Command).execute 0 0% 0% 1.82s 52.91% main.main 0 0% 0% 1.82s 52.91% runtime.main 0 0% 0% 1.82s 52.91% sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1 0 0% 0% 1.57s 45.64% runtime.systemstack 0 0% 0% 1.49s 43.31% runtime.gcBgMarkWorker 0 0% 0% 1.49s 43.31% runtime.gcBgMarkWorker.func2 0.03s 0.87% 0.87% 1.49s 43.31% runtime.gcDrain 0.62s 18.02% 18.90% 1.45s 42.15% runtime.scanobject 0 0% 18.90% 1.25s 36.34% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run 0 0% 18.90% 1.21s 35.17% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline) 0 0% 18.90% 1.21s 35.17% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap 0 0% 18.90% 1.08s 31.40% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget 0 0% 18.90% 1.08s 31.40% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources 0 0% 18.90% 1.08s 31.40% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget 0 0% 18.90% 1.07s 31.10% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateDirectory 0 0% 18.90% 0.57s 16.57% sigs.k8s.io/kustomize/api/resmap.(*resWrangler).AsYaml 0 0% 18.90% 0.57s 16.57% sigs.k8s.io/kustomize/api/resource.(*Resource).AsYAML 0.11s 3.20% 22.09% 0.48s 13.95% runtime.mallocgc 0 0% 22.09% 0.45s 13.08% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateFile 0 0% 22.09% 0.45s 13.08% sigs.k8s.io/kustomize/api/resmap.(*Factory).FromFile 0 0% 22.09% 0.33s 9.59% sigs.k8s.io/kustomize/kyaml/yaml.(*RNode).MarshalJSON 0.08s 2.33% 24.42% 0.32s 9.30% runtime.greyobject 0 0% 24.42% 0.30s 8.72% sigs.k8s.io/kustomize/api/resmap.(*Factory).NewResMapFromBytes 0 0% 24.42% 0.27s 7.85% sigs.k8s.io/yaml.JSONToYAML 0 0% 24.42% 0.25s 7.27% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).runTransformers 0 0% 24.42% 0.25s 7.27% sigs.k8s.io/kustomize/api/resource.(*Factory).RNodesFromBytes 0 0% 24.42% 0.25s 7.27% sigs.k8s.io/kustomize/api/resource.(*Factory).SliceFromBytes
Document updated code in MakeDefaultConfig. Add unit tests to ensure DeepCopy works. Add hints to other code to ensure DeepCopy is kept up to date.
@natasha41575 rebased, no conflicts so no new changes since last time. I can squash if you would like. |
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: chlunde, natasha41575 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This change has also significantly improved performance in Component-specific tests (using multiple invocations of kustomize binary), which is very nice:
Now only if we could improve patchesStrategicMerge performance... :) |
* perf: MakeDefaultConfig once This shaves of another 2 seconds (62%) of the remaining execution time for a kustomization tree with 4000 documents, reducing the execution time from 4.79s to 1.82s 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.MakeDefaultConfig before: (pprof) top30 -cum Showing nodes accounting for 1.82s, 22.75% of 8s total Dropped 408 nodes (cum <= 0.04s) Showing top 30 nodes out of 308 flat flat% sum% cum cum% 0 0% 0% 4.79s 59.88% github.com/spf13/cobra.(*Command).Execute 0 0% 0% 4.79s 59.88% github.com/spf13/cobra.(*Command).ExecuteC 0 0% 0% 4.79s 59.88% github.com/spf13/cobra.(*Command).execute 0 0% 0% 4.79s 59.88% main.main 0 0% 0% 4.79s 59.88% runtime.main 0 0% 0% 4.79s 59.88% sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1 0 0% 0% 4.22s 52.75% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run 0 0% 0% 4.18s 52.25% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline) 0 0% 0% 4.18s 52.25% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap 0 0% 0% 4.06s 50.75% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget 0 0% 0% 4.06s 50.75% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources 0 0% 0% 4.06s 50.75% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget 0 0% 0% 4.05s 50.62% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateDirectory 0 0% 0% 3.22s 40.25% runtime.systemstack 0 0% 0% 3.03s 37.88% runtime.gcBgMarkWorker 0 0% 0% 3.03s 37.88% runtime.gcBgMarkWorker.func2 0.11s 1.38% 1.38% 3.03s 37.88% runtime.gcDrain 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.MakeDefaultConfig 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.MakeTransformerConfig 0 0% 1.38% 2.98s 37.25% sigs.k8s.io/kustomize/api/internal/plugins/builtinconfig.makeTransformerConfigFromBytes 0 0% 1.38% 2.97s 37.12% sigs.k8s.io/yaml.yamlUnmarshal 0 0% 1.38% 2.91s 36.38% sigs.k8s.io/yaml.Unmarshal (inline) 1.34s 16.75% 18.12% 2.87s 35.88% runtime.scanobject 0 0% 18.12% 2.53s 31.62% sigs.k8s.io/yaml.yamlToJSON 0 0% 18.12% 1.91s 23.88% gopkg.in/yaml%2ev2.unmarshal 0 0% 18.12% 1.86s 23.25% gopkg.in/yaml%2ev2.Unmarshal (partial-inline) 0 0% 18.12% 1.43s 17.88% gopkg.in/yaml%2ev2.(*parser).parse 0.01s 0.12% 18.25% 1.27s 15.88% gopkg.in/yaml%2ev2.(*parser).document 0.01s 0.12% 18.38% 1.25s 15.62% gopkg.in/yaml%2ev2.(*parser).mapping 0.35s 4.38% 22.75% 1.21s 15.12% runtime.mallocgc after: (pprof) top30 -cum Showing nodes accounting for 0.84s, 24.42% of 3.44s total Dropped 225 nodes (cum <= 0.02s) Showing top 30 nodes out of 345 flat flat% sum% cum cum% 0 0% 0% 1.82s 52.91% github.com/spf13/cobra.(*Command).Execute 0 0% 0% 1.82s 52.91% github.com/spf13/cobra.(*Command).ExecuteC 0 0% 0% 1.82s 52.91% github.com/spf13/cobra.(*Command).execute 0 0% 0% 1.82s 52.91% main.main 0 0% 0% 1.82s 52.91% runtime.main 0 0% 0% 1.82s 52.91% sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1 0 0% 0% 1.57s 45.64% runtime.systemstack 0 0% 0% 1.49s 43.31% runtime.gcBgMarkWorker 0 0% 0% 1.49s 43.31% runtime.gcBgMarkWorker.func2 0.03s 0.87% 0.87% 1.49s 43.31% runtime.gcDrain 0.62s 18.02% 18.90% 1.45s 42.15% runtime.scanobject 0 0% 18.90% 1.25s 36.34% sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run 0 0% 18.90% 1.21s 35.17% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline) 0 0% 18.90% 1.21s 35.17% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap 0 0% 18.90% 1.08s 31.40% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget 0 0% 18.90% 1.08s 31.40% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources 0 0% 18.90% 1.08s 31.40% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget 0 0% 18.90% 1.07s 31.10% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateDirectory 0 0% 18.90% 0.57s 16.57% sigs.k8s.io/kustomize/api/resmap.(*resWrangler).AsYaml 0 0% 18.90% 0.57s 16.57% sigs.k8s.io/kustomize/api/resource.(*Resource).AsYAML 0.11s 3.20% 22.09% 0.48s 13.95% runtime.mallocgc 0 0% 22.09% 0.45s 13.08% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateFile 0 0% 22.09% 0.45s 13.08% sigs.k8s.io/kustomize/api/resmap.(*Factory).FromFile 0 0% 22.09% 0.33s 9.59% sigs.k8s.io/kustomize/kyaml/yaml.(*RNode).MarshalJSON 0.08s 2.33% 24.42% 0.32s 9.30% runtime.greyobject 0 0% 24.42% 0.30s 8.72% sigs.k8s.io/kustomize/api/resmap.(*Factory).NewResMapFromBytes 0 0% 24.42% 0.27s 7.85% sigs.k8s.io/yaml.JSONToYAML 0 0% 24.42% 0.25s 7.27% sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).runTransformers 0 0% 24.42% 0.25s 7.27% sigs.k8s.io/kustomize/api/resource.(*Factory).RNodesFromBytes 0 0% 24.42% 0.25s 7.27% sigs.k8s.io/kustomize/api/resource.(*Factory).SliceFromBytes * Tests and comments for MakeDefaultConfig perf work Document updated code in MakeDefaultConfig. Add unit tests to ensure DeepCopy works. Add hints to other code to ensure DeepCopy is kept up to date.
This shaves of another 2 seconds (62%) of the remaining execution time for a kustomization tree with 4000 documents, reducing the execution time from 4.79s to 1.82s
before:
after: