Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Vendoring Issues with Globs and Symlinks #984

Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

Conversation

Listener430
Copy link
Collaborator

@Listener430 Listener430 commented Jan 29, 2025

what

Done:

  1. append //. at the end to github repo url (in order to clone entire repo)
  2. Removed any symlinks inside go-getter
  3. Fixed support for double star globs in excluded_paths and included_paths:
    included_paths:
      - "**/{demo-library,demo-stacks}/**/*.{tf,md}"
    excluded_paths:
      - "**/demo-library/**/*.{tfvars,tf}"
    
  4. Added test to vendoring scenario in fixtures
  5. Added depth=1 for all github downloads through custom detector
  6. Breaking change: * now correctly matches a single segment. Anyone using a single star to match multiple segments should change it to **. This should never have matched multiple segments so long as double star was supposed to work.

why

  • double star globs were not correctly matching multiple segments in all cases
  • vendoring without a shallow depth is 2x slower
  • the //. is an esoteric expression to copy all files from the root. Rather than expect users to know this, we default it where it makes sense.

references

Summary by CodeRabbit

Summary by CodeRabbit

  • New Features

    • Enhanced GitHub URL handling with custom getter mechanisms.
    • Added functionality to remove symlinks during git clone operations.
    • Introduced comprehensive file and directory copying with pattern matching capabilities.
    • Added new components to manage specific source files with inclusion and exclusion patterns.
    • Introduced a new documentation section on using glob patterns for vendoring.
  • Improvements

    • Refined git cloning process with custom getter configuration.
    • Improved URL parsing for GitHub repository detection.
    • Enhanced test coverage for the atmos CLI with new file existence checks.

@Listener430 Listener430 added the bugfix Change that restores intended behavior label Jan 29, 2025
@Listener430 Listener430 self-assigned this Jan 29, 2025
@Listener430 Listener430 requested a review from a team as a code owner January 29, 2025 18:30
@Listener430 Listener430 requested a review from osterman January 29, 2025 18:31
Copy link
Contributor

coderabbitai bot commented Jan 29, 2025

📝 Walkthrough

Walkthrough

The changes in this pull request enhance the GitHub repository fetching mechanism within the Atmos vendoring process. A new field, source, is added to the CustomGitHubDetector struct, which aids in determining the repository's structure. The RegisterCustomDetectors function is modified to accept this new parameter. Additionally, a CustomGitGetter type is introduced, which implements a Get method that performs a standard git clone operation and removes symlinks from the destination directory using a new helper function, removeSymlinks. A new file for copying files with inclusion/exclusion patterns is also introduced.

Changes

File Change Summary
internal/exec/go_getter_utils.go - Added source field to CustomGitHubDetector
- Updated RegisterCustomDetectors to accept source
- Introduced CustomGitGetter type with Get method
- Added removeSymlinks function for symlink removal
internal/exec/copy_glob.go - Introduced file for copying with inclusion/exclusion patterns; added functions for copying files and directories recursively.
internal/exec/vendor_model.go - Updated downloadAndInstall to use copyToTargetWithPatterns instead of copyToTarget.
tests/fixtures/scenarios/vendor/vendor.yaml - Added new component entry for "test globs" with specified source and path patterns.
tests/test-cases/demo-stacks.yaml - Expanded atmos_vendor_pull test cases with additional file existence checks.

Assessment against linked issues

Objective Addressed Explanation
Vendor entire repo [DEV-2964]
Support exclude_paths and include_paths with globs [DEV-2964]
Fix symlink handling during vendoring [DEV-2964]
Support for shallow clones [#1025]
Handle symlinks correctly [#698]

Possibly related PRs

  • fix(vendor): Remove excessive URL validation #1024: The changes in the main PR are related to the modifications in the internal/exec/go_getter_utils.go file, specifically in the handling of GitHub URLs and the Detect method, which may interact with the URL validation logic altered in the retrieved PR.
  • package manager for vendor pull #768: The changes in the main PR, particularly the modifications to the vendor_model.go file, are related to the enhancements made in the retrieved PR regarding the vendor pull functionality, as both involve updates to the handling of vendor components and the introduction of new command options.
  • fix bug atmos vendor pull URI cannot contain path traversal sequences and git schema #899: The changes in the main PR, specifically the modifications to the internal/exec/go_getter_utils.go file, are related to the updates in the internal/exec/vendor_model.go file from the retrieved PR, as both involve enhancements to the handling of GitHub URLs and the copying process in the context of vendoring.

Suggested labels

minor

Suggested reviewers

  • aknysh
  • osterman

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary or @auto-summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai or @auto-title anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
internal/exec/go_getter_utils.go (2)

160-168: Eliminate potential duplication with existing logic.

This RegisterCustomGetters seems to mirror the RegisterCustomDetectors pattern. Consider centralizing shared logic or converting both into a unified registration function to reduce redundancy.


207-220: Consider configurable symlink removal.

Automatically stripping symlinks is helpful, but give the user control to skip or confirm. This might be vital for some use cases or compliance requirements.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 255a05a and f9b3348.

📒 Files selected for processing (1)
  • internal/exec/go_getter_utils.go (3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Analyze (go)
  • GitHub Check: Summary
🔇 Additional comments (3)
internal/exec/go_getter_utils.go (3)

102-109: Check for potential unintended path expansion.

Appending //. whenever the path ends with .git or has only three parts might impact cases where the user wants to download a subdirectory. Confirm that this logic is aligned with user expectations.


188-199: Good approach for overriding the default getters.

Overriding 'git' with a custom implementation is a practical solution. Be sure to keep an eye on future expansions (like s3, gcs) if needed.


222-236: Validate error handling on symlink removal.

If removing a symlink fails due to permission errors or other reasons, you might want more detailed logging or fallback behavior. Evaluate whether to continue, warn, or halt.

coderabbitai[bot]
coderabbitai bot previously approved these changes Jan 29, 2025
@osterman
Copy link
Member

Please add a test for this type of vendoring. It can be in the vendoring scenario we already have.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
internal/exec/go_getter_utils.go (1)

202-231: Enhance symlink handling with additional safeguards.

While the implementation is functional, consider these improvements:

  1. Pass AtmosConfig to removeSymlinks instead of using an empty config
  2. Add protection against circular symlinks
  3. Log removed symlink details for audit purposes

Here's a suggested enhancement:

 type CustomGitGetter struct {
     getter.GitGetter
+    AtmosConfig schema.AtmosConfiguration
 }

 func (c *CustomGitGetter) Get(dst string, url *url.URL) error {
     if err := c.GitGetter.Get(dst, url); err != nil {
         return err
     }
-    return removeSymlinks(dst)
+    return removeSymlinks(dst, c.AtmosConfig)
 }

-func removeSymlinks(root string) error {
+func removeSymlinks(root string, config schema.AtmosConfiguration) error {
+    visited := make(map[string]bool)
     return filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
         if err != nil {
             return err
         }
         if info.Mode()&os.ModeSymlink != 0 {
+            // Check for circular symlinks
+            realPath, err := filepath.EvalSymlinks(path)
+            if err != nil {
+                return fmt.Errorf("failed to evaluate symlink %s: %w", path, err)
+            }
+            if visited[realPath] {
+                return fmt.Errorf("circular symlink detected: %s", path)
+            }
+            visited[realPath] = true
+
+            // Log symlink details before removal
+            target, _ := os.Readlink(path)
-            u.LogWarning(schema.AtmosConfiguration{}, fmt.Sprintf("Removing symlink: %s", path))
+            u.LogWarning(config, fmt.Sprintf("Removing symlink: %s -> %s", path, target))
             return os.Remove(path)
         }
         return nil
     })
 }
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f9b3348 and ed1af3d.

📒 Files selected for processing (1)
  • internal/exec/go_getter_utils.go (4 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (4)
internal/exec/go_getter_utils.go (4)

73-73: LGTM! Field addition enhances URL handling capabilities.

The addition of the source field to store the original URL is a clean enhancement.


151-157: LGTM! Clean function signature update.

The modification properly propagates the source URL to the detector while maintaining the existing detector registration logic.


171-194: LGTM! Well-documented changes with secure defaults.

The changes are well-documented and the getters map provides a comprehensive set of protocols. Good security practice to have S3 and GCS getters disabled by default.


103-109: LGTM! Clean implementation of top-level repository detection.

The logic correctly handles bare repository URLs by appending "//." when appropriate. The debug logging provides good visibility into the decision-making process.

Let's verify the URL transformation behavior:

✅ Verification successful

URL handling logic is correctly implemented and follows go-getter conventions! 💪

The code properly handles repository URLs by:

  • Detecting bare repository URLs (without subdirectories)
  • Adding "//." suffix only when needed (for .git URLs or standard org/repo formats)
  • Maintaining compatibility with go-getter's URL format requirements
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Test various GitHub URL patterns to ensure correct handling
echo "Testing URL transformations..."

# Create a test file
cat << 'EOF' > test.go
package main

import (
    "fmt"
    "os"
    u "github.com/cloudposse/atmos/pkg/utils"
)

func main() {
    detector := &CustomGitHubDetector{source: os.Args[1]}
    result, ok, err := detector.Detect(os.Args[1], "")
    fmt.Printf("Input: %s\nOutput: %s\nOK: %v\nErr: %v\n", os.Args[1], result, ok, err)
}
EOF

# Test cases
urls=(
    "github.com/org/repo.git"
    "github.com/org/repo"
    "github.com/org/repo/subdir"
)

for url in "${urls[@]}"; do
    echo "Testing URL: $url"
    go run test.go "$url"
    echo "---"
done

Length of output: 6203


Script:

#!/bin/bash
# Find the URL handling implementation
echo "Searching for URL handling implementation..."
ast-grep --pattern 'type $_ struct {
  source string
  $$$
}'

echo -e "\nSearching for Detect method implementation..."
ast-grep --pattern 'func ($_ $_) Detect($_, $_) ($_, $_, error) {
  $$$
}'

echo -e "\nSearching for URL parsing logic..."
rg -A 5 'strings\.Contains.*"//"' --type go

Length of output: 998

coderabbitai[bot]
coderabbitai bot previously approved these changes Jan 30, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
internal/exec/copy_glob.go (1)

168-255: Consider simplifying copyToTargetWithPatterns.

The function logic is clear, but the length and multiple branches make it complex. Splitting out smaller helper functions (e.g., for handling the included vs excluded paths) might improve maintainability and ease of unit testing.

🧰 Tools
🪛 GitHub Check: Build (macos-latest, macos)

[failure] 184-184:
too many arguments in call to u.LogTrace

🪛 GitHub Check: Build (ubuntu-latest, linux)

[failure] 184-184:
too many arguments in call to u.LogTrace

tests/fixtures/scenarios/vendor/vendor.yaml (1)

54-64: Add a test case to validate the newly introduced globs.

The expanded vendor configuration is helpful for controlling which files are included or excluded. Remember to add or update an integration test verifying that only .tf and .md files are included while .tfvars files are excluded for this source.

Would you like help creating a dedicated test that specifically checks the behavior of these globs?

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ed1af3d and 5a6789e.

📒 Files selected for processing (4)
  • internal/exec/copy_glob.go (1 hunks)
  • internal/exec/vendor_model.go (1 hunks)
  • tests/fixtures/scenarios/vendor/vendor.yaml (1 hunks)
  • tests/test-cases/demo-stacks.yaml (1 hunks)
🧰 Additional context used
🪛 GitHub Check: Build (macos-latest, macos)
internal/exec/copy_glob.go

[failure] 55-55:
too many arguments in call to u.LogTrace


[failure] 64-64:
too many arguments in call to u.LogTrace


[failure] 67-67:
too many arguments in call to u.LogTrace


[failure] 78-78:
too many arguments in call to u.LogTrace


[failure] 81-81:
too many arguments in call to u.LogTrace


[failure] 87-87:
too many arguments in call to u.LogTrace


[failure] 119-119:
too many arguments in call to u.LogTrace


[failure] 157-157:
too many arguments in call to u.LogTrace


[failure] 162-162:
too many arguments in call to u.LogTrace


[failure] 184-184:
too many arguments in call to u.LogTrace

🪛 GitHub Check: Build (ubuntu-latest, linux)
internal/exec/copy_glob.go

[failure] 55-55:
too many arguments in call to u.LogTrace


[failure] 64-64:
too many arguments in call to u.LogTrace


[failure] 67-67:
too many arguments in call to u.LogTrace


[failure] 78-78:
too many arguments in call to u.LogTrace


[failure] 81-81:
too many arguments in call to u.LogTrace


[failure] 87-87:
too many arguments in call to u.LogTrace


[failure] 119-119:
too many arguments in call to u.LogTrace


[failure] 157-157:
too many arguments in call to u.LogTrace


[failure] 162-162:
too many arguments in call to u.LogTrace


[failure] 184-184:
too many arguments in call to u.LogTrace

⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (2)
internal/exec/vendor_model.go (1)

310-310: Switching to copyToTargetWithPatterns looks good.

This change aligns with the new glob-handling logic. The final code is consistent with the PR’s objective to support inclusion and exclusion patterns.

tests/test-cases/demo-stacks.yaml (1)

196-204: Enhanced Vendoring File Checks Added

The new file existence entries in the "atmos_vendor_pull" test bolster the validation of the vendoring process. They ensure that the expected files from both the demo-library and demo-stacks components are present after cloning and post-symlink removal. This aligns well with the PR objectives addressing glob functionality and symlink handling.

internal/exec/copy_glob.go Outdated Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
internal/exec/go_getter_utils.go (2)

202-215: Consider enhancing error context.

While the implementation is solid, consider wrapping the errors with additional context to aid in debugging:

 func (c *CustomGitGetter) Get(dst string, url *url.URL) error {
 	// Normal clone
 	if err := c.GitGetter.Get(dst, url); err != nil {
-		return err
+		return fmt.Errorf("git clone failed for %s: %w", url, err)
 	}
 	// Remove symlinks
-	return removeSymlinks(dst)
+	if err := removeSymlinks(dst); err != nil {
+		return fmt.Errorf("failed to remove symlinks in %s: %w", dst, err)
+	}
+	return nil
 }

219-231: Consider adding path validation and detailed logging.

The symlink removal implementation could be enhanced with additional safety checks and logging:

 func removeSymlinks(root string) error {
+	// Validate root path
+	if _, err := os.Stat(root); err != nil {
+		return fmt.Errorf("invalid root path %s: %w", root, err)
+	}
+
 	return filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
 		if err != nil {
 			return err
 		}
 		if info.Mode()&os.ModeSymlink != 0 {
-			u.LogWarning(schema.AtmosConfiguration{}, fmt.Sprintf("Removing symlink: %s", path))
+			target, err := os.Readlink(path)
+			if err != nil {
+				return fmt.Errorf("failed to read symlink %s: %w", path, err)
+			}
+			u.LogWarning(schema.AtmosConfiguration{}, fmt.Sprintf("Removing symlink: %s -> %s", path, target))
 			// It's a symlink, remove it
-			return os.Remove(path)
+			if err := os.Remove(path); err != nil {
+				return fmt.Errorf("failed to remove symlink %s: %w", path, err)
+			}
 		}
 		return nil
 	})
 }
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4fdffa3 and b8d7a5a.

📒 Files selected for processing (2)
  • internal/exec/go_getter_utils.go (4 hunks)
  • internal/exec/vendor_model.go (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • internal/exec/vendor_model.go
⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (2)
internal/exec/go_getter_utils.go (2)

71-74: LGTM! Smart handling of top-level repository cloning.

The addition of the source field and the logic to append "//." for top-level repositories is a solid improvement. This ensures proper cloning behavior when no subdirectory is specified.

Also applies to: 103-109


151-155: LGTM! Well-documented parameter addition.

The addition of the source parameter and the detailed comments explaining its purpose make the code's intent clear and maintainable.

Also applies to: 171-175

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
internal/exec/copy_glob.go (1)

47-92: ⚠️ Potential issue

Fix LogTrace calls to match the updated function signature.

Based on the pipeline failures and past review comments, the LogTrace calls need to be updated.

Apply this fix to all LogTrace calls in the function:

-    u.LogTrace(atmosConfig, fmt.Sprintf("Error computing relative path for %q: %v", srcPath, err))
+    u.LogTrace(fmt.Sprintf("Error computing relative path for %q: %v", srcPath, err))
🧹 Nitpick comments (4)
internal/exec/copy_glob.go (3)

16-45: Consider handling errors from deferred Close() calls.

While the implementation is solid, it's good practice to handle errors from Close() operations, especially for the destination file where data corruption could occur.

Here's how you could improve the error handling:

 func copyFile(atmosConfig schema.AtmosConfiguration, src, dst string) error {
     sourceFile, err := os.Open(src)
     if err != nil {
         return fmt.Errorf("opening source file %q: %w", src, err)
     }
-    defer sourceFile.Close()
+    defer func() {
+        if cerr := sourceFile.Close(); cerr != nil && err == nil {
+            err = fmt.Errorf("closing source file %q: %w", src, cerr)
+        }
+    }()
 
     if err := os.MkdirAll(filepath.Dir(dst), os.ModePerm); err != nil {
         return fmt.Errorf("creating destination directory for %q: %w", dst, err)
     }
 
     destinationFile, err := os.Create(dst)
     if err != nil {
         return fmt.Errorf("creating destination file %q: %w", dst, err)
     }
-    defer destinationFile.Close()
+    defer func() {
+        if cerr := destinationFile.Close(); cerr != nil && err == nil {
+            err = fmt.Errorf("closing destination file %q: %w", dst, cerr)
+        }
+    }()

139-166: Consider extracting pattern suffixes as constants.

The pattern matching logic is well-implemented, but the magic strings could be made more maintainable.

Consider this improvement:

+const (
+    singleLevelGlob = "/*"
+    recursiveGlob   = "/**"
+)
+
 func getMatchesForPattern(atmosConfig schema.AtmosConfiguration, sourceDir, pattern string) ([]string, error) {
     fullPattern := filepath.Join(sourceDir, pattern)
     matches, err := u.GetGlobMatches(fullPattern)
     if err != nil {
         return nil, fmt.Errorf("error getting glob matches for %q: %w", fullPattern, err)
     }
     if len(matches) == 0 {
-        if strings.HasSuffix(pattern, "/*") {
-            recursivePattern := strings.TrimSuffix(pattern, "/*") + "/**"
+        if strings.HasSuffix(pattern, singleLevelGlob) {
+            recursivePattern := strings.TrimSuffix(pattern, singleLevelGlob) + recursiveGlob

168-255: Consider pre-allocating the map with an estimated size.

The implementation is solid, but the map allocation could be optimized when processing inclusion patterns.

Consider this optimization:

     if len(s.IncludedPaths) > 0 {
-        filesToCopy := make(map[string]struct{})
+        // Pre-allocate map with estimated size based on number of patterns
+        filesToCopy := make(map[string]struct{}, len(s.IncludedPaths)*10)

This pre-allocation helps avoid map resizing during population, potentially improving performance when dealing with many files.

internal/exec/go_getter_utils.go (1)

202-233: Consider adding progress logging for symlink removal.

The symlink removal implementation is solid, but for better observability, consider adding debug logging for the number of symlinks removed.

Here's a suggested enhancement:

 func removeSymlinks(root string) error {
+    removedCount := 0
     err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
         if err != nil {
             return err
         }
         if info.Mode()&os.ModeSymlink != 0 {
-            return os.Remove(path)
+            if err := os.Remove(path); err != nil {
+                return err
+            }
+            removedCount++
             return nil
         }
         return nil
     })
+    if removedCount > 0 {
+        u.LogDebug(fmt.Sprintf("Removed %d symlinks from %s", removedCount, root))
+    }
     return err
 }
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b8d7a5a and 06f5f3b.

📒 Files selected for processing (2)
  • internal/exec/copy_glob.go (1 hunks)
  • internal/exec/go_getter_utils.go (4 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (5)
internal/exec/copy_glob.go (2)

3-13: LGTM! Well-organized imports with clear purpose.

The imports are appropriately chosen for file operations and include an optimized copy library with a clear explanatory comment.


94-137: LGTM! Excellent symlink handling implementation.

The function properly implements the PR objective of removing symlinks by explicitly skipping them during the copy process. The error handling is comprehensive and the recursive logic is sound.

internal/exec/go_getter_utils.go (3)

71-74: LGTM! Clean struct extension.

The addition of the source field to CustomGitHubDetector is a well-structured change that maintains backward compatibility.


171-194: LGTM! Well-documented client configuration.

The client configuration is clear and the comments explaining the source parameter's importance are helpful.


219-233: Consider adding test coverage for symlink removal.

The symlink removal functionality would benefit from unit tests to verify its behavior.

Would you like me to generate unit tests for the removeSymlinks function to ensure it handles various scenarios correctly?

internal/exec/go_getter_utils.go Show resolved Hide resolved
internal/exec/copy_glob.go Outdated Show resolved Hide resolved
internal/exec/go_getter_utils.go Outdated Show resolved Hide resolved
tests/fixtures/scenarios/vendor/vendor.yaml Show resolved Hide resolved
internal/exec/copy_glob.go Outdated Show resolved Hide resolved
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <[email protected]>
coderabbitai[bot]
coderabbitai bot previously approved these changes Feb 6, 2025
coderabbitai[bot]
coderabbitai bot previously approved these changes Feb 6, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
internal/exec/copy_glob.go (2)

1-14: Add package documentation.

Consider adding a package-level documentation comment to describe the purpose and functionality of this package.

Add this documentation above the package declaration:

+// Package exec provides functionality for copying files and directories with pattern matching support.
+// It handles file permissions, symlinks, and supports inclusion/exclusion patterns for selective copying.
 package exec

17-46: Optimize error handling in copyFile.

The function is well-implemented, but we can improve the error handling by checking the result of Close() operations.

Consider this optimization:

 	sourceFile, err := os.Open(src)
 	if err != nil {
 		return fmt.Errorf("opening source file %q: %w", src, err)
 	}
-	defer sourceFile.Close()
+	defer func() {
+		if cerr := sourceFile.Close(); cerr != nil && err == nil {
+			err = fmt.Errorf("closing source file %q: %w", src, cerr)
+		}
+	}()
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a54dafb and 17db3ed.

📒 Files selected for processing (1)
  • internal/exec/copy_glob.go (1 hunks)
🧰 Additional context used
📓 Learnings (1)
internal/exec/copy_glob.go (1)
Learnt from: Listener430
PR: cloudposse/atmos#984
File: internal/exec/copy_glob.go:0-0
Timestamp: 2025-02-06T13:38:07.216Z
Learning: The `u.LogTrace` function in the `cloudposse/atmos` repository accepts `atmosConfig` as its first parameter, followed by the message string.
⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (2)
internal/exec/copy_glob.go (2)

56-56: Well-implemented logging pattern!

Excellent use of structured logging with charmbracelet/log. The debug messages are informative and will be helpful for troubleshooting.

Also applies to: 68-69, 72-73, 96-98, 100-101


48-111: Consider revising error handling strategy in shouldSkipEntry.

The function silently continues on pattern matching errors, which might hide issues with malformed patterns.

Consider either:

  1. Propagating pattern matching errors to allow the caller to decide how to handle them
  2. Adding more detailed logging about why patterns were skipped

Let's verify if there are any malformed patterns in the codebase:

internal/exec/copy_glob.go Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
internal/exec/copy_glob.go (1)

365-365: Fix typo in debug message.

There's a typo in the debug message: "becasue" should be "because".

-    l.Debug("Directory is not copied becasue it is a shallow copy", "directory", relPath)
+    l.Debug("Directory is not copied because it is a shallow copy", "directory", relPath)
tests/fixtures/scenarios/vendor/vendor.yaml (1)

75-75: Remove trailing whitespace.

Remove trailing spaces to maintain consistent formatting.

-    
+
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 75-75: trailing spaces

(trailing-spaces)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 17db3ed and 1fea6ed.

📒 Files selected for processing (3)
  • internal/exec/copy_glob.go (1 hunks)
  • tests/fixtures/scenarios/vendor/vendor.yaml (1 hunks)
  • tests/test-cases/demo-stacks.yaml (1 hunks)
🧰 Additional context used
📓 Learnings (1)
internal/exec/copy_glob.go (1)
Learnt from: Listener430
PR: cloudposse/atmos#984
File: internal/exec/copy_glob.go:0-0
Timestamp: 2025-02-06T13:38:07.216Z
Learning: The `u.LogTrace` function in the `cloudposse/atmos` repository accepts `atmosConfig` as its first parameter, followed by the message string.
🪛 YAMLlint (1.35.1)
tests/fixtures/scenarios/vendor/vendor.yaml

[error] 75-75: trailing spaces

(trailing-spaces)

⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (4)
internal/exec/copy_glob.go (2)

291-294: LGTM! Smart optimization using the cp library.

The optimization to use the cp library when no patterns are defined is a great performance improvement.


24-26: 🏁 Script executed:

#!/bin/bash
# Search for file locking mechanisms
rg -A 5 'flock|Lock|Mutex|Synchronize'

Length of output: 28258

tests/fixtures/scenarios/vendor/vendor.yaml (1)

58-58: LGTM! Well-structured glob patterns.

The glob patterns are well-structured and provide good coverage:

  • **/{demo-library,demo-stacks}/**/*.{tf,md} for multiple directories and extensions
  • /weather/*.md for specific directory without double stars
  • **/demo-localstack/* and **/demo-library/** for shallow and deep copying

Also applies to: 69-69, 79-80

tests/test-cases/demo-stacks.yaml (1)

186-202: LGTM! Comprehensive test coverage for new glob patterns.

The added file existence checks provide good coverage for the new glob patterns:

  • Verifies README.md files in various directories
  • Includes both shallow and deep copied files
  • Tests exclusion patterns by not checking for .tf files in excluded paths

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
website/docs/core-concepts/vendor/vendor-manifest.mdx (1)

501-587: Excellent New Section on Vendoring with Globs

This new section is very thorough and offers practical guidance on using both greedy (**) and non-greedy (*) patterns, as well as examples using curly braces to match multiple extensions or directories. The examples are clear, and the explanation on exclusion rules is especially helpful.

A few minor stylistic nitpicks:

  • At lines 547, 549, 579, 584, and 587, consider using the typographical ellipsis character ("…") instead of three consecutive dots to enhance readability.

Overall, these additions will significantly help users understand and apply glob-based vendoring rules more effectively.

🧰 Tools
🪛 LanguageTool

[style] ~547-~547: Consider using the typographical ellipsis character here instead.
Context: ...ers, use **/demo-library/**/*. Using {...} for Multiple Extensions or Patterns ...

(ELLIPSIS)


[style] ~549-~549: Consider using the typographical ellipsis character here instead.
Context: ...le Extensions or Patterns Curly braces {...} allow for expanding multiple patterns...

(ELLIPSIS)


[style] ~579-~579: Consider using the typographical ellipsis character here instead.
Context: ... **/demo-library/**/archive/** Using {...} here prevents the need to write two s...

(ELLIPSIS)


[style] ~584-~584: Consider using the typographical ellipsis character here instead.
Context: ... include deeper subdirectories. 3. Use {...} to match multiple extensions or direc...

(ELLIPSIS)


[style] ~587-~587: Consider using the typographical ellipsis character here instead.
Context: ...included_paths, excluded_paths, and {...} expansion, you can precisely control ...

(ELLIPSIS)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1fea6ed and 618860d.

📒 Files selected for processing (1)
  • website/docs/core-concepts/vendor/vendor-manifest.mdx (2 hunks)
🧰 Additional context used
🪛 LanguageTool
website/docs/core-concepts/vendor/vendor-manifest.mdx

[style] ~547-~547: Consider using the typographical ellipsis character here instead.
Context: ...ers, use **/demo-library/**/*. Using {...} for Multiple Extensions or Patterns ...

(ELLIPSIS)


[style] ~549-~549: Consider using the typographical ellipsis character here instead.
Context: ...le Extensions or Patterns Curly braces {...} allow for expanding multiple patterns...

(ELLIPSIS)


[style] ~579-~579: Consider using the typographical ellipsis character here instead.
Context: ... **/demo-library/**/archive/** Using {...} here prevents the need to write two s...

(ELLIPSIS)


[style] ~584-~584: Consider using the typographical ellipsis character here instead.
Context: ... include deeper subdirectories. 3. Use {...} to match multiple extensions or direc...

(ELLIPSIS)


[style] ~587-~587: Consider using the typographical ellipsis character here instead.
Context: ...included_paths, excluded_paths, and {...} expansion, you can precisely control ...

(ELLIPSIS)

⏰ Context from checks skipped due to timeout of 90000ms (6)
  • GitHub Check: Build (macos-latest, macos)
  • GitHub Check: Analyze (javascript-typescript)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: website-deploy-preview
  • GitHub Check: Analyze (go)
  • GitHub Check: Summary
🔇 Additional comments (1)
website/docs/core-concepts/vendor/vendor-manifest.mdx (1)

220-220: Clarify Glob Support in the Schema Description

The updated line now explicitly mentions that included_paths and excluded_paths support POSIX‐style greedy globs (with double-star/globstar). This adds clarity to how patterns are interpreted and ties nicely to the new detailed glob section later on.

coderabbitai[bot]
coderabbitai bot previously approved these changes Feb 10, 2025
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <[email protected]>
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
website/docs/core-concepts/vendor/vendor-manifest.mdx (2)

509-511: Consider adding case sensitivity information.

The explanation of glob patterns is accurate. Consider adding a note about case sensitivity in glob pattern matching to make the documentation more comprehensive.

 - `*` (single asterisk) matches any sequence of characters **within a single path segment**.
 - `**` (double asterisk) matches across multiple path segments **recursively**.
+- Note: Glob patterns are case-sensitive by default.

514-532: Consider adding a visual directory tree example.

While the examples are clear, a visual directory tree representation would help users better understand the pattern matching behavior.

Add a directory tree example like this:

 This means:
 - All files within demo-library except those inside any `stargazers` subdirectory are vendored.
 - Any other files outside `stargazers` are unaffected by this exclusion.
+
+Example directory structure:
+```
+demo-library/
+├── README.md
+├── src/
+│   ├── main.tf
+│   └── stargazers/  # excluded
+│       └── list.tf
+└── test/
+    └── test.tf
+```
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 618860d and d2533a7.

📒 Files selected for processing (1)
  • website/docs/core-concepts/vendor/vendor-manifest.mdx (2 hunks)
🧰 Additional context used
🪛 LanguageTool
website/docs/core-concepts/vendor/vendor-manifest.mdx

[style] ~547-~547: Consider using the typographical ellipsis character here instead.
Context: ...ers, use **/demo-library/**/*. Using {...} for Multiple Extensions or Patterns ...

(ELLIPSIS)


[style] ~549-~549: Consider using the typographical ellipsis character here instead.
Context: ...le Extensions or Patterns Curly braces {...} allow for expanding multiple patterns...

(ELLIPSIS)


[style] ~579-~579: Consider using the typographical ellipsis character here instead.
Context: ... **/demo-library/**/archive/** Using {...} here prevents the need to write two s...

(ELLIPSIS)


[style] ~584-~584: Consider using the typographical ellipsis character here instead.
Context: ...t include deeper subdirectories. 3. Use {...} to match multiple extensions or direc...

(ELLIPSIS)


[style] ~587-~587: Consider using the typographical ellipsis character here instead.
Context: ...included_paths, excluded_paths, and {...} expansion, you can precisely control ...

(ELLIPSIS)

⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: website-deploy-preview
  • GitHub Check: Analyze (go)
  • GitHub Check: Summary
🔇 Additional comments (2)
website/docs/core-concepts/vendor/vendor-manifest.mdx (2)

501-504: Well-structured introduction to glob patterns!

The new section is logically placed and provides a clear introduction to the concept of glob patterns in the context of Atmos vendoring.


582-586: Excellent summary with key takeaways!

The key takeaways section effectively summarizes the important points about glob patterns, maintaining consistency with the document's style while providing actionable guidance.

🧰 Tools
🪛 LanguageTool

[style] ~584-~584: Consider using the typographical ellipsis character here instead.
Context: ...t include deeper subdirectories. 3. Use {...} to match multiple extensions or direc...

(ELLIPSIS)

@aknysh
Copy link
Member

aknysh commented Feb 11, 2025

@Listener430 since this PR is related to vendoring, please review this new issue and fix it in your PR

#1049

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugfix Change that restores intended behavior
Projects
None yet
3 participants