Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve atmos list stacks #979

Open
wants to merge 67 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
8311350
feat: add output format options for `atmos list stacks` command
Cerebrovinny Jan 28, 2025
faeec51
feat(list): enhance stack listing with configurable formats and columns
Cerebrovinny Jan 28, 2025
80d540c
Add stack list columns configuration to atmos.yaml
Cerebrovinny Jan 28, 2025
baa42e5
refactor: improve table formatting for non-TTY output
Cerebrovinny Jan 28, 2025
060d49b
Add list format and columns configuration to stacks config
Cerebrovinny Jan 28, 2025
c29f1e8
docs: update list-stacks command documentation and simplify default c…
Cerebrovinny Jan 30, 2025
d53379f
test: add test case for FilterAndListStacks function
Cerebrovinny Jan 30, 2025
9e79f60
Update website/docs/cli/commands/list/list-stacks.mdx
Cerebrovinny Jan 30, 2025
6bbe5b7
Update website/docs/cli/commands/list/list-stacks.mdx
Cerebrovinny Feb 2, 2025
1611285
Update website/docs/cli/commands/list/list-stacks.mdx
Cerebrovinny Feb 2, 2025
8690e35
Move stack list columns configuration to quick-start-advanced example
Cerebrovinny Feb 2, 2025
d7718bc
Update website/docs/cli/commands/list/list-stacks.mdx
Cerebrovinny Feb 2, 2025
6c33077
test: enhance FilterAndListStacks test with realistic stack configura…
Cerebrovinny Feb 2, 2025
1a871fa
fix: use quick-start-simple path for stacks when tenant info is incom…
Cerebrovinny Feb 2, 2025
7ad8f6e
refactor: simplify stack listing and improve output formatting
Cerebrovinny Feb 3, 2025
856c0eb
refactor: extract stack info creation and improve JSON output handling
Cerebrovinny Feb 3, 2025
d66c1c7
fix: use stack file from atmos configuration instead of hardcoded value
Cerebrovinny Feb 3, 2025
6b257de
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 5, 2025
c976cf3
optimize: pre-parse templates for better list performance
Cerebrovinny Feb 5, 2025
c8dd900
Add atmos_stack_file to stack info and update column template
Cerebrovinny Feb 5, 2025
14e477f
Merge branch 'main' into DEV-2804
osterman Feb 6, 2025
98f5dfe
feat: enhance stack listing with variable extraction and dynamic columns
Cerebrovinny Feb 6, 2025
da45ed2
feat: improve JSON and CSV output formatting for empty values
Cerebrovinny Feb 6, 2025
aab2f73
Merge branch 'main' into DEV-2804
osterman Feb 6, 2025
6ead9fb
Remove unused list configuration from stacks section
Cerebrovinny Feb 7, 2025
90345ad
update describe
Cerebrovinny Feb 9, 2025
e348298
update snapshot
Cerebrovinny Feb 9, 2025
c0d58e0
test: add Stage field to list stacks test cases
Cerebrovinny Feb 9, 2025
7d23cf2
update tsv code
Cerebrovinny Feb 9, 2025
7468739
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 9, 2025
e69f7af
Consider the website as documentation, not programming (#1041)
Nuru Feb 9, 2025
de25bdb
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 9, 2025
4fcbaca
add test characters and improvements for list stacks
Cerebrovinny Feb 9, 2025
72fa8a0
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 9, 2025
e63a8d0
Merge branch 'main' into DEV-2804
aknysh Feb 10, 2025
d25dd4b
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 11, 2025
07d2a69
simplify loops
Cerebrovinny Feb 12, 2025
4ed7c5f
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 12, 2025
61e6383
update docs
Cerebrovinny Feb 12, 2025
049a2ed
clean code
Cerebrovinny Feb 12, 2025
d6c7f91
added fixes for stack empty
Cerebrovinny Feb 12, 2025
7429e5c
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 13, 2025
3167c4f
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 13, 2025
8e3dd3b
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 13, 2025
b669f45
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 13, 2025
bac6f41
update tty
Cerebrovinny Feb 13, 2025
0a0ad12
singles func
Cerebrovinny Feb 13, 2025
53bd07c
Update website/docs/cli/commands/list/list-stacks.mdx
osterman Feb 14, 2025
21a2de6
add nested values support
Cerebrovinny Feb 15, 2025
22154ad
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 15, 2025
35710f0
clean code
Cerebrovinny Feb 15, 2025
a2fce86
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 15, 2025
8ee545c
clean code
Cerebrovinny Feb 15, 2025
0de25ea
Filter stack names by include/exclude paths; improve logic
Cerebrovinny Feb 15, 2025
f1b682b
Revert "Filter stack names by include/exclude paths; improve logic"
Cerebrovinny Feb 15, 2025
43fcbaa
Update pkg/list/list_stacks.go
Cerebrovinny Feb 15, 2025
42a0938
Refactor delimiter logic, simplify table formatting
Cerebrovinny Feb 15, 2025
132d9a0
Refactor stack metadata handling and template checks
Cerebrovinny Feb 16, 2025
7bee512
Add filterStackNames func & unit tests for stack filtering
Cerebrovinny Feb 16, 2025
80c577b
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 16, 2025
7b403ba
Fix header alignment by changing row index check
Cerebrovinny Feb 16, 2025
1eeb293
Fix CSV output condition; add table format for TTY output
Cerebrovinny Feb 16, 2025
7934498
Merge remote-tracking branch 'origin/main' into DEV-2804
Cerebrovinny Feb 17, 2025
53a10b7
Merge branch 'main' into DEV-2804
osterman Feb 18, 2025
108ad37
Merge branch 'main' into DEV-2804
Cerebrovinny Feb 19, 2025
d6e3919
Merge branch 'main' into DEV-2804
osterman Feb 20, 2025
21c21f1
Merge branch 'main' into DEV-2804
osterman Feb 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion atmos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -394,4 +394,3 @@ version:
enabled: true
timeout: 1000 # ms
frequency: 1h

28 changes: 26 additions & 2 deletions cmd/cmd_utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -704,12 +704,36 @@
u.PrintInvalidUsageErrorAndExit(errors.New(details), suggestion)
}

// filterStackNames filters out file paths and returns valid stack names

Check failure on line 707 in cmd/cmd_utils.go

View workflow job for this annotation

GitHub Actions / golangci

[golangci] cmd/cmd_utils.go#L707

Comment should end in a period (godot)
Raw output
cmd/cmd_utils.go:707:1: Comment should end in a period (godot)
// filterStackNames filters out file paths and returns valid stack names
^
func filterStackNames(stacksMap map[string]any, basePath string) []string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s revise this function name. We want to add proper filtering later (not this PR), but this is filtering on paths. Also, it’s not clear to me why we need this. Can you add a comment explaining why we need this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this wold get replaced once we refactor the full stacks path on the discussion bellow

stackNames := make([]string, 0)
for stackName := range stacksMap {
// Skip if it's a file path (starts with base path)
if basePath != "" && strings.HasPrefix(stackName, basePath+"/") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is not right either. Stacks can exist anywhere, what we include and exclude is entirely config driven. We are not consulting that configuration, and deriving an implementation that doesn’t correspond to the documentation. We should have a centralized function for this and it relates to stacks not general utils, IMO.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so I think on this case would be best having that function inside stacks pkg and then reuse FindAllStackConfigsInPaths from config to find the paths

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aknysh please let me know your opinion of whats the best approach to refactor this, I see that we also have getConfigAndStacksInfo in utils

continue
}
stackNames = append(stackNames, stackName)
}
return stackNames
}

func stackFlagCompletion(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
output, err := listStacks(cmd)
configAndStacksInfo := schema.ConfigAndStacksInfo{}
atmosConfig, err := cfg.InitCliConfig(configAndStacksInfo, true)
if err != nil {
return nil, cobra.ShellCompDirectiveNoFileComp
}

// Get all stacks
stacksMap, err := e.ExecuteDescribeStacks(atmosConfig, "", nil, nil, nil, false, true, true, false, nil)
if err != nil {
return nil, cobra.ShellCompDirectiveNoFileComp
}
return output, cobra.ShellCompDirectiveNoFileComp

// Filter stack names using configured base path
stackNames := filterStackNames(stacksMap, atmosConfig.Stacks.BasePath)

return stackNames, cobra.ShellCompDirectiveNoFileComp
}

func AddStackCompletion(cmd *cobra.Command) {
Expand Down
59 changes: 59 additions & 0 deletions cmd/cmd_utils_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package cmd

import (
"testing"

"github.com/stretchr/testify/assert"
)

func TestFilterStackNames(t *testing.T) {
tests := []struct {
name string
stacksBasePath string
stacksMap map[string]any
expectedStacks []string
}{
{
name: "filters out paths with configured base path",
stacksBasePath: "stacks",
stacksMap: map[string]any{
"stacks/test1.yaml": nil,
"test1": nil,
"test2": nil,
"stacks/test2": nil,
},
expectedStacks: []string{"test1", "test2"},
},
{
name: "handles custom base path",
stacksBasePath: "custom/path",
stacksMap: map[string]any{
"custom/path/test1.yaml": nil,
"test1": nil,
"test2": nil,
"custom/path/test2": nil,
},
expectedStacks: []string{"test1", "test2"},
},
{
name: "handles empty base path",
stacksBasePath: "",
stacksMap: map[string]any{
"test1.yaml": nil,
"test1": nil,
"test2": nil,
},
expectedStacks: []string{"test1", "test2", "test1.yaml"},
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
// Run the filter function
stacks := filterStackNames(tt.stacksMap, tt.stacksBasePath)

// Verify the results
assert.ElementsMatch(t, tt.expectedStacks, stacks)
})
}
}
19 changes: 14 additions & 5 deletions cmd/list_stacks.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,10 @@ var listStacksCmd = &cobra.Command{
Short: "List all Atmos stacks or stacks for a specific component",
Long: "This command lists all Atmos stacks, or filters the list to show only the stacks associated with a specified component.",
Example: "atmos list stacks\n" +
"atmos list stacks -c <component>",
"atmos list stacks -c <component>\n" +
"atmos list stacks --format json\n" +
"atmos list stacks --format csv --delimiter ','\n" +
"atmos list stacks --format table",
FParseErrWhitelist: struct{ UnknownFlags bool }{UnknownFlags: false},
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
Expand All @@ -37,22 +40,28 @@ var listStacksCmd = &cobra.Command{

func init() {
listStacksCmd.DisableFlagParsing = false
listStacksCmd.PersistentFlags().StringP("component", "c", "", "atmos list stacks -c <component>")
listStacksCmd.PersistentFlags().StringP("component", "c", "", "Filter stacks by component")
listStacksCmd.PersistentFlags().StringP("format", "f", "", "Output format (table, json, csv)")
listStacksCmd.PersistentFlags().StringP("delimiter", "d", "\t", "Delimiter for table and csv formats")
listCmd.AddCommand(listStacksCmd)
}

func listStacks(cmd *cobra.Command) ([]string, error) {
componentFlag, _ := cmd.Flags().GetString("component")
formatFlag, _ := cmd.Flags().GetString("format")
delimiterFlag, _ := cmd.Flags().GetString("delimiter")

configAndStacksInfo := schema.ConfigAndStacksInfo{}
atmosConfig, err := config.InitCliConfig(configAndStacksInfo, true)
if err != nil {
return nil, fmt.Errorf("Error initializing CLI config: %v", err)
}
stacksMap, err := e.ExecuteDescribeStacks(atmosConfig, "", nil, nil, nil, false, false, false, false, nil)

stacksMap, err := e.ExecuteDescribeStacks(atmosConfig, "", nil, nil, nil, false, true, true, false, nil)
if err != nil {
return nil, fmt.Errorf("Error describing stacks: %v", err)
}

output, err := l.FilterAndListStacks(stacksMap, componentFlag)
return output, err
output, err := l.FilterAndListStacks(stacksMap, componentFlag, atmosConfig.Stacks.List, formatFlag, delimiterFlag)
return []string{output}, err
}
10 changes: 10 additions & 0 deletions examples/quick-start-advanced/atmos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,16 @@ stacks:
- "**/_defaults.yaml"
# Can also be set using 'ATMOS_STACKS_NAME_PATTERN' ENV var
name_pattern: "{tenant}-{environment}-{stage}"
list:
columns:
- name: Stack
value: '{{ .atmos_stack }}'
- name: Tenant
value: '{{ index .vars "tenant" }}'
- name: Environment
value: '{{ index .vars "environment" }}'
- name: File
value: '{{ .atmos_stack_file }}'

workflows:
# Can also be set using 'ATMOS_WORKFLOWS_BASE_PATH' ENV var, or '--workflows-dir' command-line argument
Expand Down
33 changes: 21 additions & 12 deletions internal/exec/describe_stacks.go
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@
if !u.MapKeyExists(finalStacksMap, stackName) {
finalStacksMap[stackName] = make(map[string]any)
finalStacksMap[stackName].(map[string]any)["components"] = make(map[string]any)
finalStacksMap[stackName].(map[string]any)["atmos_stack_file"] = stackFileName
}

if componentsSection, ok := stackSection.(map[string]any)["components"].(map[string]any); ok {
Expand Down Expand Up @@ -307,15 +308,22 @@
if err != nil {
return nil, err
}
} else {
} else if atmosConfig.Stacks.NamePattern != "" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aknysh please check this part. We need to ensure that both name pattern and name templates work.

context = cfg.GetContextFromVars(varsSection)
configAndStacksInfo.Context = context
stackName, err = cfg.GetContextPrefix(stackFileName, context, GetStackNamePattern(atmosConfig), stackFileName)
if err != nil {
return nil, err
}
} else {
// If no name pattern or template is configured, use the stack file name
stackName = stackFileName
}

// Update the component section with the final stack name
configAndStacksInfo.ComponentSection["atmos_stack"] = stackName
configAndStacksInfo.ComponentSection["stack"] = stackName

if filterByStack != "" && filterByStack != stackFileName && filterByStack != stackName {
continue
}
Expand All @@ -327,18 +335,15 @@
// Only create the stack entry if it doesn't exist
if !u.MapKeyExists(finalStacksMap, stackName) {
finalStacksMap[stackName] = make(map[string]any)
finalStacksMap[stackName].(map[string]any)["components"] = make(map[string]any)
finalStacksMap[stackName].(map[string]any)["atmos_stack_file"] = stackFileName
}

configAndStacksInfo.ComponentSection["atmos_component"] = componentName
configAndStacksInfo.ComponentSection["atmos_stack"] = stackName
configAndStacksInfo.ComponentSection["stack"] = stackName
configAndStacksInfo.ComponentSection["atmos_stack_file"] = stackFileName
configAndStacksInfo.ComponentSection["atmos_manifest"] = stackFileName

if len(components) == 0 || u.SliceContainsString(components, componentName) || u.SliceContainsString(derivedComponents, componentName) {
if !u.MapKeyExists(finalStacksMap[stackName].(map[string]any), "components") {
finalStacksMap[stackName].(map[string]any)["components"] = make(map[string]any)
}
if !u.MapKeyExists(finalStacksMap[stackName].(map[string]any)["components"].(map[string]any), "terraform") {
finalStacksMap[stackName].(map[string]any)["components"].(map[string]any)["terraform"] = make(map[string]any)
}
Expand Down Expand Up @@ -516,15 +521,22 @@
if err != nil {
return nil, err
}
} else {
} else if atmosConfig.Stacks.NamePattern != "" {
context = cfg.GetContextFromVars(varsSection)
configAndStacksInfo.Context = context
stackName, err = cfg.GetContextPrefix(stackFileName, context, GetStackNamePattern(atmosConfig), stackFileName)
if err != nil {
return nil, err
}
} else {
// If no name pattern or template is configured, use the stack file name
stackName = stackFileName
}

// Update the component section with the final stack name
configAndStacksInfo.ComponentSection["atmos_stack"] = stackName
configAndStacksInfo.ComponentSection["stack"] = stackName

Check failure on line 538 in internal/exec/describe_stacks.go

View workflow job for this annotation

GitHub Actions / golangci

[golangci] internal/exec/describe_stacks.go#L538

add-constant: string literal "stack" appears, at least, 4 times, create a named constant for it (revive)
Raw output
internal/exec/describe_stacks.go:538:44: add-constant: string literal "stack" appears, at least, 4 times, create a named constant for it (revive)
						configAndStacksInfo.ComponentSection["stack"] = stackName
						                                     ^

if filterByStack != "" && filterByStack != stackFileName && filterByStack != stackName {
continue
}
Expand All @@ -536,18 +548,15 @@
// Only create the stack entry if it doesn't exist
if !u.MapKeyExists(finalStacksMap, stackName) {
finalStacksMap[stackName] = make(map[string]any)
finalStacksMap[stackName].(map[string]any)["components"] = make(map[string]any)
finalStacksMap[stackName].(map[string]any)["atmos_stack_file"] = stackFileName
}

configAndStacksInfo.ComponentSection["atmos_component"] = componentName
configAndStacksInfo.ComponentSection["atmos_stack"] = stackName
configAndStacksInfo.ComponentSection["stack"] = stackName
configAndStacksInfo.ComponentSection["atmos_stack_file"] = stackFileName
configAndStacksInfo.ComponentSection["atmos_manifest"] = stackFileName

if len(components) == 0 || u.SliceContainsString(components, componentName) || u.SliceContainsString(derivedComponents, componentName) {
if !u.MapKeyExists(finalStacksMap[stackName].(map[string]any), "components") {
finalStacksMap[stackName].(map[string]any)["components"] = make(map[string]any)
}
if !u.MapKeyExists(finalStacksMap[stackName].(map[string]any)["components"].(map[string]any), "helmfile") {
finalStacksMap[stackName].(map[string]any)["components"].(map[string]any)["helmfile"] = make(map[string]any)
}
Expand Down
Loading
Loading