Develop #1542
Merged
Develop #1542
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…nd dependency updates - Added jq to the dependencies in the Docker base image installation script. - Updated the Python installation script to support handling of custom requirements via command-line arguments. - Implemented a new function to manage requirements installation, falling back to a default requirements file if none is provided. - Improved version check logic to prevent redundant installations of Python versions, enhancing efficiency in Docker environments.
- Updated various dependencies in go.mod and go.sum files, including cloud.google.com/go/compute/metadata to v0.5.2, google.golang.org/grpc to v1.69.2, and google.golang.org/protobuf to v1.36.1. - Refactored gRPC service definitions to use the latest protoc-gen-go and protoc versions, ensuring compatibility with the latest gRPC-Go features. - Introduced a new logger utility in core/utils/logger.go to improve logging capabilities across the application. - Added a README.md for gRPC setup and compilation guidance, enhancing developer experience. - Improved the Python installation script to handle version listing more effectively and ensure user-specific environment setup.
- Changed the default Python version in the Docker base image installation script from 3.12 to 3.12.8 to ensure the latest patch version is used. - This update enhances the reliability and security of the Python installation process in Docker environments.
- Replaced viper calls with utility functions in GetSystemInfo to improve code clarity and maintainability. - Added a new system.go file with utility functions for retrieving system version and edition information. - Enhanced PrintLogoWithWelcomeInfo to include detailed system information, improving user experience during server startup. - Updated output formatting for better readability and consistency in welcome messages.
- Added logging for error handling in the MasterService when setting a worker node offline, replacing the previous trace.PrintError with a more informative log message. - Enhanced WorkerService subscription method with debug logs to indicate subscription attempts and status, improving traceability during connection processes.
- Introduced a new "setup" command in the Python installation script to facilitate the setup of pyenv. - This enhancement improves the usability of the script by allowing users to easily configure their Python environment.
- Eliminated the default_version field from the DependencyConfig struct in dependency_config.go to streamline the configuration model. - This change simplifies the dependency management process and aligns with recent updates in the application structure.
- Modified the Python installation script to remove the default version assignment and enforce version specification during installation. - Updated usage instructions to reflect the change from a default version to requiring a user-defined version, improving clarity and preventing installation errors. - This change enhances the robustness of the installation process by ensuring users explicitly define the Python version they wish to install.
…g installation - Modified the Dockerfile to require Python version 3.12 explicitly during the installation process. - This change ensures that the installation script adheres to the specified version, improving consistency and reliability in Docker environments.
- Updated the dependencies in the Docker base image installation scripts by removing unnecessary packages and ensuring essential tools are included. - Simplified the Python installation script by refining the list of required libraries, enhancing clarity and reducing potential installation issues. - Modified the requirements.txt to include only necessary packages, improving the efficiency of the Python environment setup.
- Updated log messages in NodeServiceServer and TaskServiceServer to remove the "[NodeServiceServer]" and "[TaskServiceServer]" prefixes for cleaner output. - This change enhances log readability and maintains consistency across logging practices in the application.
- Added 'zip' and 'unzip' utilities to the dependency installation script for improved file handling capabilities. - Updated usage instructions in the Python installation script to include a new 'setup' command for configuring pyenv, enhancing user experience and script functionality.
- Added support for new dependency file types: 'go.mod' and 'pom.xml' in dependency.go. - Refactored command configuration in runner.go to improve logging and error handling. - Introduced a new method to configure Node.js paths, enhancing environment setup for tasks. - Enhanced IPC message handling with detailed logging for better traceability. - Updated service logging to remove unnecessary prefixes for cleaner output. - Improved command execution handling in process.go for better compatibility across platforms.
- Changed the default task log path from '/var/log/crawlab/tasks' to '/app/logs/tasks' to align with the application's directory structure and improve portability in different deployment environments.
… log path - Improved error messages in the FileLogDriver's cleanup method to include error details for better debugging. - Updated the default task log path from '/app/logs/tasks' to '/var/log/crawlab/tasks' to ensure consistency across environments.
- Enhanced the IPC message handling in runner_test.go by adding detailed logging for better traceability. - Refactored the test setup to use channels for synchronization and improved error handling during message processing. - Updated the runner.go file to rename variables for clarity and streamline the IPC reader implementation. - Improved the cleanup process in tests to ensure proper resource management and context cancellation.
…ependency management - Introduced a new DependencyInstallerService interface to define methods for managing dependency installation commands. - Implemented registry service for managing the DependencyInstallerService instance. - Enhanced the task runner to install dependencies if available, including command execution and logging for stdout and stderr. - Improved error handling and logging throughout the task runner's dependency installation process. - Updated the runner's methods to utilize the new dependency management features, ensuring better integration and functionality.
- Changed route parameter from ':id' to ':key' in settings-related routes for better clarity and consistency. - Updated GetSetting, PostSetting, and PutSetting functions to use the new ':key' parameter. - Introduced IsAutoInstallEnabled method in DependencyInstallerService to check auto-installation status. - Enhanced the task runner to check if auto installation is enabled before proceeding with dependency installation. - Improved initialization of settings data in the system service, ensuring proper insertion of initial settings.
…d execution - Added a task status update to 'processing' at the start of the Run method in runner.go, improving task tracking. - Removed redundant task status update from the end of the Run method to streamline the execution flow. - Updated command execution in process.go to use 'bash' instead of 'sh' for better compatibility across environments.
- Removed redundant home directory retrieval and nvm checks in the configureNodePath method. - Introduced a new utility function GetNodeModulesPath to centralize the logic for determining the global node_modules path. - Updated environment variable setup to use the new utility function, improving clarity and maintainability of the code.
- Updated IPC reader initialization in runner_test.go to use a channel for signaling readiness, improving synchronization. - Added error logging when writing to the pipe to enhance traceability during tests. - These changes improve the reliability and clarity of the test setup for the task runner.
- Changed package manager command from 'apt-get' to 'apt' for improved usability. - Added additional Python dependencies: 'libbz2-dev', 'libreadline-dev', 'libsqlite3-dev', 'xz-utils', and 'liblzma-dev' to enhance the environment setup. - Updated 'crawlab-sdk' version in requirements.txt from '0.7.0rc1' to '0.7.0rc5' for compatibility and feature improvements.
- Introduced a new method configureGoPath in runner.go to set the GOPATH environment variable based on the retrieved Go path. - Updated configureEnv to call configureGoPath, ensuring the Go path is configured alongside Node.js paths. - Added a new utility function GetGoPath in config.go to retrieve the Go path from configuration, with a default fallback. - These changes enhance the task runner's environment setup by supporting Go development alongside existing Node.js configurations.
…r tests - Added configuration initialization in db.go to ensure proper setup of application settings. - Refactored runner_test.go to streamline IPC message handling by introducing a setupPipe function and an initRunner function for better readability and maintainability. - Improved synchronization in tests by using channels for signaling readiness and processing, enhancing the reliability of IPC message handling. - Updated test cases to validate IPC message processing and error handling, ensuring robustness in the task runner's functionality.
…te handling - Replaced direct calls to getBaseFileFsSvc with a new method fs.GetBaseFileFsSvc in base_file.go for improved clarity and maintainability. - Introduced SpiderTemplateService interface and implemented registry service for managing spider templates, enhancing template handling in the spider controller. - Added template-related fields to the Spider model to support template functionality. - Created utility functions for string case conversions in utils/string.go to facilitate consistent formatting across the codebase. - Updated environment configuration to retrieve the Python path dynamically, improving flexibility in the task runner's setup.
- Introduced a new function ToKebabCase in utils/string.go to convert strings to kebab-case format. - The function trims whitespace, converts to lowercase, and replaces spaces and underscores with hyphens, enhancing string manipulation capabilities in the codebase.
- Updated the Spider model to introduce a new SpiderTemplateParams struct for improved template handling. - Refactored string utility functions in utils/string.go to include a new replaceChars function, streamlining character replacement across multiple functions. - Enhanced ToSnakeCase and ToKebabCase functions to utilize the new replaceChars function for better maintainability and readability. - Added splitStringWithQuotes function to facilitate string manipulation with quotes, improving overall utility in string processing.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| RUN npm install -g pnpm | ||
|
|
||
| # Copy application code | ||
| ADD . . |
Check notice
Code scanning / SonarCloud
Prefer COPY over ADD for copying local resources Low
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.