Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve applyOrdering by avoid call to GetByCurrentId #5079

Merged
merged 1 commit into from
Apr 6, 2024

Conversation

chlunde
Copy link
Contributor

@chlunde chlunde commented Mar 7, 2023

This shaves of 14 seconds (one third) of the execution time for a kustomization tree with 4000 documents, from 40.68s to 27.41s

        0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder
        0     0%  5.44%     18.40s 40.52%  sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering

before

(pprof) top20 -cum
Showing nodes accounting for 5.85s, 12.88% of 45.41s total
Dropped 622 nodes (cum <= 0.23s)
Showing top 20 nodes out of 157
    flat  flat%   sum%        cum   cum%
        0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).Execute
        0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).ExecuteC
        0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).execute
        0     0%     0%     40.68s 89.58%  main.main
        0     0%     0%     40.68s 89.58%  runtime.main
        0     0%     0%     40.68s 89.58%  sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1
        0     0%     0%     40.12s 88.35%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run
    0.51s  1.12%  1.12%     33.20s 73.11%  sigs.k8s.io/kustomize/api/resource.(*Resource).CurId
        0     0%  1.12%     26.95s 59.35%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId
    0.35s  0.77%  1.89%     26.95s 59.35%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById
    0.07s  0.15%  2.05%     25.53s 56.22%  sigs.k8s.io/kustomize/api/resmap.GetCurrentId
        0     0%  2.05%     21.68s 47.74%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline)
        0     0%  2.05%     21.68s 47.74%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap
    0.54s  1.19%  3.24%     19.75s 43.49%  sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline)
        1s  2.20%  5.44%     19.21s 42.30%  sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode
        0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/internal/builtins.(*SortOrderTransformerPlugin).Transform
        0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder
        0     0%  5.44%     18.40s 40.52%  sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering
    0.87s  1.92%  7.36%     16.55s 36.45%  sigs.k8s.io/kustomize/kyaml/yaml.visitMappingNodeFields
    2.51s  5.53% 12.88%     15.68s 34.53%  sigs.k8s.io/kustomize/kyaml/yaml.visitFieldsWhileTrue

after

(pprof) top20 -cum
Showing nodes accounting for 1.23s, 3.85% of 31.98s total
Dropped 584 nodes (cum <= 0.16s)
Showing top 20 nodes out of 184
    flat  flat%   sum%        cum   cum%
        0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).Execute
        0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).ExecuteC
        0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).execute
        0     0%     0%     27.41s 85.71%  main.main
        0     0%     0%     27.41s 85.71%  runtime.main
        0     0%     0%     27.41s 85.71%  sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1
        0     0%     0%     26.85s 83.96%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run
        0     0%     0%     22.07s 69.01%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline)
        0     0%     0%     22.07s 69.01%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap
    0.38s  1.19%  1.19%     20.69s 64.70%  sigs.k8s.io/kustomize/api/resource.(*Resource).CurId
        0     0%  1.19%     13.64s 42.65%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).Append
        0     0%  1.19%     13.55s 42.37%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId (inline)
    0.12s  0.38%  1.56%     13.55s 42.37%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById
    0.01s 0.031%  1.59%     12.67s 39.62%  sigs.k8s.io/kustomize/api/resmap.GetCurrentId
    0.21s  0.66%  2.25%     12.49s 39.06%  sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline)
    0.51s  1.59%  3.85%     12.28s 38.40%  sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode
        0     0%  3.85%     11.52s 36.02%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).IgnoreLocal
        0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget
        0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources
        0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 7, 2023
@k8s-ci-robot
Copy link
Contributor

Welcome @chlunde!

It looks like this is your first PR to kubernetes-sigs/kustomize 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kustomize has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Mar 7, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @chlunde. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 7, 2023
@koba1t
Copy link
Member

koba1t commented Mar 8, 2023

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 8, 2023
@koba1t
Copy link
Member

koba1t commented Mar 8, 2023

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 8, 2023
@natasha41575
Copy link
Contributor

@yanniszark Do you have time to take a look at this one?

Copy link
Contributor

@yanniszark yanniszark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chlunde thanks a lot for your perf PRs, they are greatly needed and will massively improve the user experience.
I left 2 comments for small changes but overall if this passes tests it should be good to go.

api/internal/builtins/SortOrderTransformer.go Outdated Show resolved Hide resolved
api/internal/builtins/SortOrderTransformer.go Show resolved Hide resolved
@koba1t
Copy link
Member

koba1t commented Mar 22, 2023

/triage accepted
/kind feature
/priority important-longterm

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. kind/feature Categorizes issue or PR as related to a new feature. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Mar 22, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 20, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle rotten
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 20, 2023
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 30, 2023
This shaves of 14 seconds (one third) of the execution time for a
kustomization tree with 4000 documents, from 40.68s to 27.41s

            0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder
            0     0%  5.44%     18.40s 40.52%  sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering
before

    (pprof) top20 -cum
    Showing nodes accounting for 5.85s, 12.88% of 45.41s total
    Dropped 622 nodes (cum <= 0.23s)
    Showing top 20 nodes out of 157
        flat  flat%   sum%        cum   cum%
            0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).Execute
            0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).ExecuteC
            0     0%     0%     40.68s 89.58%  github.com/spf13/cobra.(*Command).execute
            0     0%     0%     40.68s 89.58%  main.main
            0     0%     0%     40.68s 89.58%  runtime.main
            0     0%     0%     40.68s 89.58%  sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1
            0     0%     0%     40.12s 88.35%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run
        0.51s  1.12%  1.12%     33.20s 73.11%  sigs.k8s.io/kustomize/api/resource.(*Resource).CurId
            0     0%  1.12%     26.95s 59.35%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId
        0.35s  0.77%  1.89%     26.95s 59.35%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById
        0.07s  0.15%  2.05%     25.53s 56.22%  sigs.k8s.io/kustomize/api/resmap.GetCurrentId
            0     0%  2.05%     21.68s 47.74%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline)
            0     0%  2.05%     21.68s 47.74%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap
        0.54s  1.19%  3.24%     19.75s 43.49%  sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline)
            1s  2.20%  5.44%     19.21s 42.30%  sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode
            0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/internal/builtins.(*SortOrderTransformerPlugin).Transform
            0     0%  5.44%     18.42s 40.56%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).applySortOrder
            0     0%  5.44%     18.40s 40.52%  sigs.k8s.io/kustomize/api/internal/builtins.applyOrdering
        0.87s  1.92%  7.36%     16.55s 36.45%  sigs.k8s.io/kustomize/kyaml/yaml.visitMappingNodeFields
        2.51s  5.53% 12.88%     15.68s 34.53%  sigs.k8s.io/kustomize/kyaml/yaml.visitFieldsWhileTrue

after

    (pprof) top20 -cum
    Showing nodes accounting for 1.23s, 3.85% of 31.98s total
    Dropped 584 nodes (cum <= 0.16s)
    Showing top 20 nodes out of 184
        flat  flat%   sum%        cum   cum%
            0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).Execute
            0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).ExecuteC
            0     0%     0%     27.41s 85.71%  github.com/spf13/cobra.(*Command).execute
            0     0%     0%     27.41s 85.71%  main.main
            0     0%     0%     27.41s 85.71%  runtime.main
            0     0%     0%     27.41s 85.71%  sigs.k8s.io/kustomize/kustomize/v5/commands/build.NewCmdBuild.func1
            0     0%     0%     26.85s 83.96%  sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run
            0     0%     0%     22.07s 69.01%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap (inline)
            0     0%     0%     22.07s 69.01%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap
        0.38s  1.19%  1.19%     20.69s 64.70%  sigs.k8s.io/kustomize/api/resource.(*Resource).CurId
            0     0%  1.19%     13.64s 42.65%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).Append
            0     0%  1.19%     13.55s 42.37%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).GetMatchingResourcesByCurrentId (inline)
        0.12s  0.38%  1.56%     13.55s 42.37%  sigs.k8s.io/kustomize/api/resmap.(*resWrangler).filteredById
        0.01s 0.031%  1.59%     12.67s 39.62%  sigs.k8s.io/kustomize/api/resmap.GetCurrentId
        0.21s  0.66%  2.25%     12.49s 39.06%  sigs.k8s.io/kustomize/api/resource.(*Resource).GetGvk (inline)
        0.51s  1.59%  3.85%     12.28s 38.40%  sigs.k8s.io/kustomize/kyaml/resid.GvkFromNode
            0     0%  3.85%     11.52s 36.02%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).IgnoreLocal
            0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget
            0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateResources
            0     0%  3.85%     10.53s 32.93%  sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 30, 2023
@chlunde
Copy link
Contributor Author

chlunde commented Oct 30, 2023

I believe I've addressed the comments, can you take another look? Thanks!

@chlunde
Copy link
Contributor Author

chlunde commented Nov 13, 2023

@natasha41575 or @yanniszark - can we get this merged? I'd like some input on how to structure the next PR #5427 but since it creates merge conflicts with this PR it would be better if this is merged first.

@chlunde
Copy link
Contributor Author

chlunde commented Nov 22, 2023

Hi, @koba1t can we get this merged? Thanks for taking a look :)

@koba1t
Copy link
Member

koba1t commented Nov 22, 2023

@natasha41575
Could you recheck this PR?
I think you reviewed this PR and I want to hear your opinions.

@chlunde
Copy link
Contributor Author

chlunde commented Jan 8, 2024

@natasha41575 could you check this again or delegate to someone else?

@chlunde
Copy link
Contributor Author

chlunde commented Jan 19, 2024

@chlunde thanks a lot for your perf PRs, they are greatly needed and will massively improve the user experience. I left 2 comments for small changes but overall if this passes tests it should be good to go.

@yanniszark I believe I've addressed your comments, could we get this merged now?

@yanniszark
Copy link
Contributor

Hi @chlunde and sorry for the late reply!
I don't have any merging capabilities unfortunately.
The current approvers are @varshaprasad96 @koba1t @KnVerey

@koba1t
Copy link
Member

koba1t commented Feb 6, 2024

HI @yanniszark
If you think your commented point was fixed, Please type the /lgtm command.
#5079 (review)
That mark means you think this PR is ready to merge.

@chlunde
Copy link
Contributor Author

chlunde commented Feb 22, 2024

@koba1t could we get another reviewer here please? Soon 1 year birthday 😄 🍰 (a lot of that time is on me)

@yanniszark
Copy link
Contributor

/lgtm

@k8s-ci-robot
Copy link
Contributor

@yanniszark: changing LGTM is restricted to collaborators

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@koba1t
Copy link
Member

koba1t commented Mar 6, 2024

@chlunde
Thanks, and I'm sorry for the delay in the review.
It looks like all test was passed; I think this PR is good to me!

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chlunde, koba1t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 6, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Reopen this PR with /reopen
  • Mark this PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Reopen this PR with /reopen
  • Mark this PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@koba1t
Copy link
Member

koba1t commented Apr 6, 2024

/reopen

@k8s-ci-robot k8s-ci-robot reopened this Apr 6, 2024
@k8s-ci-robot
Copy link
Contributor

@koba1t: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@koba1t
Copy link
Member

koba1t commented Apr 6, 2024

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 6, 2024
@k8s-ci-robot k8s-ci-robot merged commit 82ee768 into kubernetes-sigs:master Apr 6, 2024
15 checks passed
@koba1t
Copy link
Member

koba1t commented Apr 6, 2024

Sorry. Maybe I missed merging the last time.

@chlunde
Copy link
Contributor Author

chlunde commented Apr 12, 2024

@koba1t thanks! 👍

@chlunde chlunde deleted the perf-1 branch April 12, 2024 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants