Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce Output in Atmos Tests #923

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a lot of tests were unintentionally deleted from this file.

Copy link
Collaborator Author

@Cerebrovinny Cerebrovinny Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did that to test the output of robherley/go-test-action@v0 in actions, its added back

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw this is not very good, not able to make it work properly so for now I recommend we stick with our main tests only

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"this is not very good"

Not sure what it refers to.

Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ jobs:

- name: Acceptance tests
timeout-minutes: 10
run: make testacc
run: make testacc TEST_VERBOSITY=quiet

docker:
name: "Docker Lint"
Expand Down Expand Up @@ -465,4 +465,4 @@ jobs:
with:
publish: false
format: binary
secrets: inherit
secrets: inherit
15 changes: 12 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,17 @@ version-windows: build-windows
deps:
go mod download

# Run acceptance tests
testacc: get
go test $(TEST) -v $(TESTARGS) -timeout 2m
# Test verbosity levels: quiet, normal, verbose
TEST_VERBOSITY ?= normal

.PHONY: testacc
testacc: ## Run all tests
@if [ "$(TEST_VERBOSITY)" = "quiet" ]; then \
ATMOS_TEST_VERBOSITY=$(TEST_VERBOSITY) go test ./... -timeout 20m; \
elif [ "$(TEST_VERBOSITY)" = "verbose" ]; then \
ATMOS_TEST_VERBOSITY=$(TEST_VERBOSITY) go test -v -count=1 ./... -timeout 20m; \
else \
ATMOS_TEST_VERBOSITY=$(TEST_VERBOSITY) go test -v ./... -timeout 20m; \
fi
Comment on lines +60 to +66
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what benefit this translation brings. Why not just set the variable needed, rather than translate it?


.PHONY: lint get build version build-linux build-windows build-macos deps version-linux version-windows version-macos testacc
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,3 +222,25 @@ Copyright © 2017-2024 [Cloud Posse, LLC](https://cpco.io/copyright)
<a href="https://cloudposse.com/readme/footer/link?utm_source=github&utm_medium=readme&utm_campaign=cloudposse/atmos&utm_content=readme_footer_link"><img alt="README footer" src="https://cloudposse.com/readme/footer/img"/></a>

<img alt="Beacon" width="0" src="https://ga-beacon.cloudposse.com/UA-76589703-4/cloudposse/atmos?pixel&cs=github&cm=readme&an=atmos"/>

## Testing

### Test Verbosity Control

The test suite supports three verbosity levels to control output:

- `quiet`: Minimal output, only shows failures
- `normal`: Standard test output (default)
- `verbose`: Detailed test output with additional debugging information

You can control the verbosity level using the `TEST_VERBOSITY` environment variable:

```bash
# Run tests with minimal output
make test TEST_VERBOSITY=quiet

# Run acceptance tests with verbose output
make testacc TEST_VERBOSITY=verbose
```

This is particularly useful when running tests in CI environments or when debugging specific test failures.
9 changes: 9 additions & 0 deletions cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"github.com/cloudposse/atmos/internal/tui/templates"
tuiUtils "github.com/cloudposse/atmos/internal/tui/utils"
cfg "github.com/cloudposse/atmos/pkg/config"
"github.com/cloudposse/atmos/pkg/logger"
"github.com/cloudposse/atmos/pkg/schema"
u "github.com/cloudposse/atmos/pkg/utils"
)
Expand All @@ -39,6 +40,14 @@ var RootCmd = &cobra.Command{
cmd.SilenceUsage = true
cmd.SilenceErrors = true
}

// Validate logs-level flag if provided
if cmd.Flags().Changed("logs-level") {
logsLevel, _ := cmd.Flags().GetString("logs-level")
if _, err := logger.ParseLogLevel(logsLevel); err != nil {
u.LogErrorAndExit(schema.AtmosConfiguration{}, err)
}
}
},
Run: func(cmd *cobra.Command, args []string) {
// Check Atmos configuration
Expand Down
75 changes: 51 additions & 24 deletions pkg/describe/describe_stacks_test.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package describe

import (
"os"
"testing"

"github.com/stretchr/testify/assert"
Expand All @@ -10,6 +11,11 @@ import (
u "github.com/cloudposse/atmos/pkg/utils"
)

// shouldLogVerbose returns true if test verbosity is set to verbose
func shouldLogVerbose() bool {
return os.Getenv("ATMOS_TEST_VERBOSITY") == "verbose"
}

func TestDescribeStacks(t *testing.T) {
configAndStacksInfo := schema.ConfigAndStacksInfo{}

Expand All @@ -18,10 +24,13 @@ func TestDescribeStacks(t *testing.T) {

stacks, err := ExecuteDescribeStacks(atmosConfig, "", nil, nil, nil, false, false)
assert.Nil(t, err)
assert.NotNil(t, stacks)

dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
if shouldLogVerbose() {
dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
}
}

func TestDescribeStacksWithFilter1(t *testing.T) {
Expand All @@ -34,10 +43,13 @@ func TestDescribeStacksWithFilter1(t *testing.T) {

stacks, err := ExecuteDescribeStacks(atmosConfig, stack, nil, nil, nil, false, false)
assert.Nil(t, err)
assert.NotNil(t, stacks)

dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
if shouldLogVerbose() {
dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
}
}

func TestDescribeStacksWithFilter2(t *testing.T) {
Expand All @@ -51,10 +63,13 @@ func TestDescribeStacksWithFilter2(t *testing.T) {

stacks, err := ExecuteDescribeStacks(atmosConfig, stack, components, nil, nil, false, false)
assert.Nil(t, err)
assert.NotNil(t, stacks)

dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
if shouldLogVerbose() {
dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
}
}

func TestDescribeStacksWithFilter3(t *testing.T) {
Expand All @@ -68,10 +83,13 @@ func TestDescribeStacksWithFilter3(t *testing.T) {

stacks, err := ExecuteDescribeStacks(atmosConfig, stack, nil, nil, sections, false, false)
assert.Nil(t, err)
assert.NotNil(t, stacks)

dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
if shouldLogVerbose() {
dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
}
}

func TestDescribeStacksWithFilter4(t *testing.T) {
Expand All @@ -85,10 +103,13 @@ func TestDescribeStacksWithFilter4(t *testing.T) {

stacks, err := ExecuteDescribeStacks(atmosConfig, "", nil, componentTypes, sections, false, false)
assert.Nil(t, err)
assert.NotNil(t, stacks)

dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
if shouldLogVerbose() {
dependentsYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(dependentsYaml)
}
}

func TestDescribeStacksWithFilter5(t *testing.T) {
Expand Down Expand Up @@ -117,9 +138,11 @@ func TestDescribeStacksWithFilter5(t *testing.T) {
assert.Equal(t, "ue2", tenant1Ue2DevStackComponentsTerraformComponentVarsEnvironment)
assert.Equal(t, "dev", tenant1Ue2DevStackComponentsTerraformComponentVarsStage)

stacksYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(stacksYaml)
if shouldLogVerbose() {
stacksYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(stacksYaml)
}
}

func TestDescribeStacksWithFilter6(t *testing.T) {
Expand All @@ -144,9 +167,11 @@ func TestDescribeStacksWithFilter6(t *testing.T) {
tenant1Ue2DevStackComponentsTerraformWorkspace := tenant1Ue2DevStackComponentsTerraformComponent["workspace"].(string)
assert.Equal(t, "tenant1-ue2-dev", tenant1Ue2DevStackComponentsTerraformWorkspace)

stacksYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(stacksYaml)
if shouldLogVerbose() {
stacksYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(stacksYaml)
}
}

func TestDescribeStacksWithFilter7(t *testing.T) {
Expand All @@ -171,9 +196,11 @@ func TestDescribeStacksWithFilter7(t *testing.T) {
tenant1Ue2DevStackComponentsTerraformWorkspace := tenant1Ue2DevStackComponentsTerraformComponent["workspace"].(string)
assert.Equal(t, "test-component-override-3-workspace", tenant1Ue2DevStackComponentsTerraformWorkspace)

stacksYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(stacksYaml)
if shouldLogVerbose() {
stacksYaml, err := u.ConvertToYAML(stacks)
assert.Nil(t, err)
t.Log(stacksYaml)
}
}

func TestDescribeStacksWithEmptyStacks(t *testing.T) {
Expand Down
103 changes: 103 additions & 0 deletions pkg/utils/test_utils.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
package utils

import (
"fmt"
"os"
"strings"
"testing"

"github.com/stretchr/testify/assert"
)

// TestVerbosity controls the verbosity level of test output
type TestVerbosity int

const (
// TestVerbosityQuiet minimal output, only failures
TestVerbosityQuiet TestVerbosity = iota
// TestVerbosityNormal standard test output
TestVerbosityNormal
// TestVerbosityVerbose detailed test output
TestVerbosityVerbose
)

var (
// DefaultTestVerbosity is the default verbosity level for tests
DefaultTestVerbosity = TestVerbosityNormal
// Store descriptions to avoid repetition
printedDescriptions = make(map[string]bool)
)

// GetTestVerbosity returns the current test verbosity level
func GetTestVerbosity() TestVerbosity {
verbStr := os.Getenv("ATMOS_TEST_VERBOSITY")
switch verbStr {
case "quiet":
return TestVerbosityQuiet
case "verbose":
return TestVerbosityVerbose
default:
return DefaultTestVerbosity
}
}

// TestLogf logs a message during test execution based on verbosity level
func TestLogf(t *testing.T, minVerbosity TestVerbosity, format string, args ...interface{}) {
if GetTestVerbosity() >= minVerbosity {
t.Logf(format, args...)
}
}

// TestLog logs a message during test execution based on verbosity level
func TestLog(t *testing.T, minVerbosity TestVerbosity, args ...interface{}) {
if GetTestVerbosity() >= minVerbosity {
t.Log(args...)
}
}

// LogTestDescription logs a test description only once to avoid repetition
func LogTestDescription(t *testing.T, description string) {
if description == "" {
return
}

// Create a unique key for the description to avoid repetition within the same test
key := fmt.Sprintf("%s-%s", t.Name(), description)
if !printedDescriptions[key] {
if GetTestVerbosity() == TestVerbosityQuiet {
t.Logf("Test Description: %s", description)
} else {
t.Logf("\nTest Description: %s", description)
}
printedDescriptions[key] = true
}
}

// LogTestFailure logs detailed failure information based on verbosity
func LogTestFailure(t *testing.T, description string, expected, actual interface{}, extraInfo ...string) {
LogTestDescription(t, description)

verbosity := GetTestVerbosity()
if verbosity == TestVerbosityQuiet {
t.Errorf("Expected: %v\nGot: %v", expected, actual)
} else {
t.Errorf("\nExpected: %v\nGot: %v", expected, actual)
if len(extraInfo) > 0 && verbosity >= TestVerbosityNormal {
t.Logf("Additional Info:\n%s", strings.Join(extraInfo, "\n"))
}
}
}

// AssertTestResult is a wrapper for testify/assert that respects verbosity
func AssertTestResult(t *testing.T, assertion func() bool, description string, msgAndArgs ...interface{}) bool {
result := assertion()
if !result {
LogTestDescription(t, description)
return assert.True(t, result, msgAndArgs...)
}
if GetTestVerbosity() >= TestVerbosityVerbose {
LogTestDescription(t, description)
return assert.True(t, result, msgAndArgs...)
}
return result
}
Loading