Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify logging module #43

Merged
merged 6 commits into from
May 13, 2024
Merged

Modify logging module #43

merged 6 commits into from
May 13, 2024

Conversation

LuQQiu
Copy link
Collaborator

@LuQQiu LuQQiu commented May 10, 2024

No description provided.

@LuQQiu LuQQiu requested a review from lucyge2022 May 10, 2024 22:37
@@ -120,6 +116,17 @@ def __init__(
if preload_path is not None:
self.alluxio.load(preload_path)

if "log_level" in test_options:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

think u should include in the options

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alluxiofs is an inner Python module
For each Python process, the end launch program should define to log level, and all the inner Python module inherit the logging configuration.
E.g. in our benchmark entry point, we should define the log level and log file there

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no I meant to say why not use the dict "options" ? its also user defined at time of instantiating alluxiofs instance right?

elif log_level == "WARN" or log_level == "WARNING":
logger.setLevel(logging.WARN)
else:
logger.warning(f"Unsupported log level: {log_level}")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and lets set a default warning level to be ERROR (logging has ERROR level correct? )

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logging usually don't have error, if it's error, the program should directly error out and failed
but they do have error log level

start_time = time.time()
res = alluxio_impl(self, path, *args, **kwargs)
self.logger.debug(
f"Successfully called {alluxio_impl.__name__} against alluxio server with args ({args}), kwargs ({kwargs}) for {time.time() - start_time} ms"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this time.time() is in sec

@LuQQiu
Copy link
Collaborator Author

LuQQiu commented May 13, 2024

@ChunxuTang @lucyge2022 PTAL thanks!
Use bench.py test out the logging
By default the logging should not set any config so that python parent modules' logging config can be passed in.
Add a test_option {"log_level": "debug"} to explicitly override alluxio related logging to debug

Copy link
Collaborator

@lucyge2022 lucyge2022 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LuQQiu LuQQiu merged commit 93105ae into fsspec:main May 13, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants