Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make logging.yml read by default #3831

Merged
merged 21 commits into from
May 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/source/logging/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,17 @@
If the `KEDRO_LOGGING_CONFIG` environment variable is not set, Kedro will use the [default logging configuration](https://github.com/kedro-org/kedro/blob/main/kedro/framework/project/default_logging.yml).
```

## Custom `CONF_SOURCE` with logging
SajidAlamQB marked this conversation as resolved.
Show resolved Hide resolved

When you customise the [`CONF_SOURCE`](../configuration/configuration_basics.md#how-to-change-the-configuration-source-folder-at-runtime) setting in your Kedro project, it determines where Kedro looks for configuration files, including the logging configuration file. However, changing `CONF_SOURCE` does not automatically update the path to `logging.yml`. To use a custom location or filename for the logging configuration, you must explicitly set the `KEDRO_LOGGING_CONFIG` environment variable.

Check warning on line 47 in docs/source/logging/index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/logging/index.md#L47

[Kedro.toowordy] 'However' is too wordy
Raw output
{"message": "[Kedro.toowordy] 'However' is too wordy", "location": {"path": "docs/source/logging/index.md", "range": {"start": {"line": 47, "column": 269}}}, "severity": "WARNING"}

By default, Kedro looks for a file named `logging.yml` in the `conf` directory. If you move or rename your logging configuration file after changing `CONF_SOURCE`, specify the new path using the `KEDRO_LOGGING_CONFIG` environment variable:
```bash
export KEDRO_LOGGING_CONFIG=<project_root>/custom_config_folder/custom_logging_name.yml
```

Please note that adjusting `CONF_SOURCE` or renaming `logging.yml` without updating the logging configuration accordingly can lead to Kedro not locating the file, which will result in the default logging settings being used instead.

Check notice on line 54 in docs/source/logging/index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/logging/index.md#L54

[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.
Raw output
{"message": "[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.", "location": {"path": "docs/source/logging/index.md", "range": {"start": {"line": 54, "column": 1}}}, "severity": "INFO"}

### How to show DEBUG level messages
To see `DEBUG` level messages, change the level of logging in your project-specific logging configuration file (`logging.yml`). We provide a `logging.yml` template:

Expand Down
29 changes: 25 additions & 4 deletions kedro/framework/project/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,13 +216,34 @@ def configure(self, pipelines_module: str | None = None) -> None:
class _ProjectLogging(UserDict):
def __init__(self) -> None:
"""Initialise project logging. The path to logging configuration is given in
environment variable KEDRO_LOGGING_CONFIG (defaults to default_logging.yml)."""
path = os.environ.get(
"KEDRO_LOGGING_CONFIG", Path(__file__).parent / "default_logging.yml"
)
environment variable KEDRO_LOGGING_CONFIG (defaults to conf/logging.yml)."""

# Check if a user path is set in the environment variable
user_logging_path = os.environ.get("KEDRO_LOGGING_CONFIG")

# Check if the default logging configuration exists
default_logging_path = Path("conf/logging.yml")
SajidAlamQB marked this conversation as resolved.
Show resolved Hide resolved
if not default_logging_path.exists():
default_logging_path = Path(__file__).parent / "default_logging.yml"

# Use the user path if available, otherwise, use the default path
if user_logging_path and Path(user_logging_path).exists():
path = Path(user_logging_path)
else:
path = default_logging_path

# Load and apply the logging configuration
logging_config = Path(path).read_text(encoding="utf-8")
self.configure(yaml.safe_load(logging_config))

# Log info about the logging configuration
if not user_logging_path and default_logging_path == Path("conf/logging.yml"):
logger = logging.getLogger(__name__)
logger.info(
f"Using `{path}` as logging configuration. "
f"You can change this by setting the KEDRO_LOGGING_CONFIG environment variable accordingly."
)

def configure(self, logging_config: dict[str, Any]) -> None:
"""Configure project logging using ``logging_config`` (e.g. from project
logging.yml). We store this in the UserDict data so that it can be reconfigured
Expand Down
30 changes: 30 additions & 0 deletions tests/framework/project/test_logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,3 +144,33 @@ def test_rich_traceback_disabled_on_databricks(

rich_traceback_install.assert_not_called()
rich_pretty_install.assert_called()


def test_default_logging_info_emission(monkeypatch, capsys):
# Expected path and logging configuration
expected_path = Path("conf/logging.yml")
dummy_logging_config = yaml.dump(
{"version": 1, "loggers": {"kedro": {"level": "INFO"}}}
)

# Setup environment and path mocks
monkeypatch.delenv("KEDRO_LOGGING_CONFIG", raising=False)
monkeypatch.setattr(Path, "exists", lambda x: x == expected_path)
monkeypatch.setattr(
Path,
"read_text",
lambda x, encoding="utf-8": dummy_logging_config
if x == expected_path
else FileNotFoundError("File not found"),
)

from kedro.framework.project import _ProjectLogging

_ProjectLogging()

captured = capsys.readouterr()

expected_message = f"Using `{expected_path}`"
assert (
expected_message in captured.out
), f"Expected message not found in logs: {captured.out}"