Addresses Issue #245 for better logging #332

patilvishal0597 · 2025-01-16T23:09:18Z

Added logger_util to enable package and class wide logging in langchain-aws. Added logging for invoke and ainvoke.

Added a logger util for configuration of logger. Tested logging of invoke, ainvoke, and converse request/responses

Behaviour:

LANGCHAIN_AWS_DEBUG = LANGCHAIN_AWS_DEBUG_ROOT = True --> Logs debug messages across boto3 and application

LANGCHAIN_AWS_DEBUG = True; LANGCHAIN_AWS_DEBUG_ROOT = True --> Logs only applcation debug messages

LANGCHAIN_AWS_DEBUG = False; LANGCHAIN_AWS_DEBUG_ROOT = True --> Logs only application info messages

LANGCHAIN_AWS_DEBUG = LANGCHAIN_AWS_DEBUG_ROOT = False --> Logs only application info messages

Invoke call:

llm = ChatBedrock(
        model_id="us.anthropic.claude-3-5-sonnet-20241022-v2:0",
        region_name="us-east-1",
        model_kwargs={
            "max_tokens": 100,
            "top_p": 0.9,
            "temperature": 0.1,
        },
    )

# input to llm
messages = [
        (
            "system",
            "You are a helpful assistant that translates English to French. Translate the user sentence.",
        ),
        ("human", "I love going out for a walk when the weather is bright and sunny."),
    ]

# Invoke the llm
response = llm.invoke(messages)

Logging output:

Debug logs with LANGCHAIN_AWS_DEBUG and LANGCHAIN_AWS_DEBUG_ROOT env vars as True

...
...
....
2025-01-15 14:30:31 DEBUG | [connectionpool.py:1051] | urllib3.connectionpool - Starting new HTTPS connection (1): bedrock-runtime.us-east-1.amazonaws.com:443
2025-01-15 14:30:33 DEBUG | [connectionpool.py:546] | urllib3.connectionpool - https://bedrock-runtime.us-east-1.amazonaws.com:443 "POST /model/us.anthropic.claude-3-5-sonnet-20241022-v2%3A0/invoke HTTP/11" 200 299
2025-01-15 14:30:33 DEBUG | [parsers.py:241] | botocore.parsers - Response headers: {'Date': 'Wed, 15 Jan 2025 22:30:33 GMT', 'Content-Type': 'application/json', 'Content-Length': '299', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'xxxxxx', 'X-Amzn-Bedrock-Invocation-Latency': '1077', 'X-Amzn-Bedrock-Output-Token-Count': '22', 'X-Amzn-Bedrock-Input-Token-Count': '40'}
2025-01-15 14:30:33 DEBUG | [parsers.py:242] | botocore.parsers - Response body:
<botocore.response.StreamingBody object at 0x108e595d0>
2025-01-15 14:30:33 DEBUG | [hooks.py:238] | botocore.hooks - Event needs-retry.bedrock-runtime.InvokeModel: calling handler <botocore.retryhandler.RetryHandler object at 0x108e58c80>
2025-01-15 14:30:33 DEBUG | [retryhandler.py:211] | botocore.retryhandler - No retry needed.
2025-01-15 14:30:33 INFO | [bedrock.py:603] | root - The output message sent by user: content="J'aime me promener quand il fait beau et ensoleillé." additional_kwargs={'usage': {'prompt_tokens': 40, 'completion_tokens': 22, 'total_tokens': 62}, 'stop_reason': 'end_turn', 'model_id': 'us.anthropic.claude-3-5-sonnet-20241022-v2:0'} response_metadata={} usage_metadata={'input_tokens': 40, 'output_tokens': 22, 'total_tokens': 62}
J'aime me promener quand il fait beau et ensoleillé.
(base) vishankp@7cf34de71c79 aws %

Logging output when LANGCHAIN_AWS_DEBUG and LANGCHAIN_AWS_DEBUG_ROOT flags are False and logger is initialized with a module_name:

2025-01-15 14:39:40 INFO | langchain_aws.chat_models.bedrock - The input message sent by user: [SystemMessage(content='You are a helpful assistant that translates English to French. Translate the user sentence.', additional_kwargs={}, response_metadata={}), HumanMessage(content='I love going out for a walk when the weather is bright and sunny.', additional_kwargs={}, response_metadata={})]
2025-01-15 14:39:40 ERROR | langchain_aws.chat_models.bedrock - Testing error log
2025-01-15 14:39:41 INFO | langchain_aws.chat_models.bedrock - The output message sent by user: content="J'aime me promener quand il fait beau et ensoleillé." additional_kwargs={'usage': {'prompt_tokens': 40, 'completion_tokens': 22, 'total_tokens': 62}, 'stop_reason': 'end_turn', 'model_id': 'us.anthropic.claude-3-5-sonnet-20241022-v2:0'} response_metadata={} usage_metadata={'input_tokens': 40, 'output_tokens': 22, 'total_tokens': 62}
J'aime me promener quand il fait beau et ensoleillé.

michaelnchin

Thanks @patilvishal0597, looks good overall. Added a few minor comments.

michaelnchin · 2025-01-21T10:56:02Z

libs/aws/langchain_aws/logger_util.py

+import os
+
+# Environment variable to set the application logger(s) in debug mode during runtime
+__DEBUG = True if os.environ.get("LANGCHAIN_AWS_DEBUG") else False


os.environ.get() returns a string - this line will set __DEBUG to True in any case where LANGCHAIN_AWS_DEBUG has been defined. So we might see unexpected behavior if the user sets something like LANGCHAIN_AWS_DEBUG=false.

An explicit string comparison like this would work better:

__DEBUG = os.getenv("LANGCHAIN_AWS_DEBUG", "").lower() in ["true", "1"]

michaelnchin · 2025-01-21T10:56:29Z

libs/aws/langchain_aws/logger_util.py

+# Flag for root debug logger to set the root debug logger in debug mode during runtime
+# Root debug logger will print boto3 as well as application debug logs if set to true
+# This flag will be set to true if LANGCHAIN_AWS_DEBUG = LANGCHAIN_AWS_DEBUG_ROOT = true
+__ROOT_DEBUG = __DEBUG if os.environ.get("LANGCHAIN_AWS_DEBUG_ROOT") else False


Same as for __DEBUG on Line 5.

michaelnchin · 2025-01-21T10:57:06Z

libs/aws/langchain_aws/logger_util.py

+# else ERROR
+if __DEBUG:
+    DEFAULT_LOG_LEVEL: int = logging.DEBUG
+else:
+    DEFAULT_LOG_LEVEL: int = logging.INFO


The comment says ERROR, but conditional uses INFO level.

Changing the default log level to "ERROR"

michaelnchin · 2025-01-21T11:16:10Z

libs/aws/langchain_aws/logger_util.py

+DEFAULT_LOG_FILE = os.environ.get("LANGCHAIN_AWS_LOG_OUTPUT", "-")
+if DEFAULT_LOG_FILE == "-":


Is there a reason for overriding with - here? Can we use the os.environ.get() default value so that the conditional doesn't need to check for an explicit value?

I've added - as a default value in case the DEFAULT_LOG_FILE value isn't set. Do we want to have a default value as None to make this cleaner?

It's unnecessary to set None explicitly - os.environ.get already returns None by default if the variable doesn't exist.

fixed in the new commit

michaelnchin · 2025-01-24T05:08:33Z

libs/aws/langchain_aws/logger_util.py

+
+# Environment variable to set the application logger(s) in debug mode during runtime
+LANGCHAIN_AWS_DEBUG: str = os.environ.get("LANGCHAIN_AWS_DEBUG", "false")
+__DEBUG: bool = True if LANGCHAIN_AWS_DEBUG.lower() in ["true", "1"] else False


Nit: simplify to

__DEBUG: bool = LANGCHAIN_AWS_DEBUG.lower() in ["true", "1"]

michaelnchin · 2025-01-24T05:30:17Z

libs/aws/langchain_aws/logger_util.py

+except ImportError:
+    colorama = None
+    coloredlogs = None
+
+    DEFAULT_LOG_FORMATTER: logging.Formatter = logging.Formatter(DEFAULT_LOG_FORMAT)


IMO we shouldn't fail this silently - add an logger statement in case users are interested in enabling coloring

michaelnchin

LGTM - thanks @patilvishal0597 !

Added `logger_util` to enable package and class wide logging in `langchain-aws`. Added logging for `invoke` and `ainvoke`. langchain-ai#245

michaelnchin suggested changes Jan 21, 2025

View reviewed changes

patilvishal0597 force-pushed the issue-245-better-logging branch from 55aeab9 to 9eb0ea3 Compare January 22, 2025 19:06

patilvishal0597 requested a review from michaelnchin January 22, 2025 19:07

patilvishal0597 force-pushed the issue-245-better-logging branch from 9eb0ea3 to c3768d5 Compare January 23, 2025 21:31

michaelnchin suggested changes Jan 24, 2025

View reviewed changes

patilvishal0597 force-pushed the issue-245-better-logging branch from c3768d5 to 694f93f Compare January 24, 2025 22:43

michaelnchin approved these changes Jan 24, 2025

View reviewed changes

Addresses logging Issue

9c2432e

Added `logger_util` to enable package and class wide logging in `langchain-aws`. Added logging for `invoke` and `ainvoke`. langchain-ai#245

patilvishal0597 force-pushed the issue-245-better-logging branch from 4c98e33 to 9c2432e Compare January 24, 2025 23:21

Merge branch 'main' into issue-245-better-logging

b3dd693

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Addresses Issue #245 for better logging #332

Addresses Issue #245 for better logging #332

patilvishal0597 commented Jan 16, 2025

michaelnchin left a comment

michaelnchin Jan 21, 2025

michaelnchin Jan 21, 2025

michaelnchin Jan 21, 2025

patilvishal0597 Jan 22, 2025

michaelnchin Jan 21, 2025

patilvishal0597 Jan 22, 2025

michaelnchin Jan 23, 2025

patilvishal0597 Jan 23, 2025

michaelnchin Jan 24, 2025

michaelnchin Jan 24, 2025

michaelnchin left a comment

		DEFAULT_LOG_FILE = os.environ.get("LANGCHAIN_AWS_LOG_OUTPUT", "-")
		if DEFAULT_LOG_FILE == "-":

Addresses Issue #245 for better logging #332

Are you sure you want to change the base?

Addresses Issue #245 for better logging #332

Conversation

patilvishal0597 commented Jan 16, 2025

Behaviour:

Invoke call:

Logging output:

Debug logs with LANGCHAIN_AWS_DEBUG and LANGCHAIN_AWS_DEBUG_ROOT env vars as True

Logging output when LANGCHAIN_AWS_DEBUG and LANGCHAIN_AWS_DEBUG_ROOT flags are False and logger is initialized with a module_name:

michaelnchin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelnchin left a comment

Choose a reason for hiding this comment