This folder provides a step-by-step guide on Langchain multi-route implementation, leveraging the Anthropic Claude V2 model alongside Langchain to deliver various capabilities:
- Querying Databases: Demonstrates how to query an Athena database using Langchain's SQLDatabaseChain, enabling the AI to fetch and present data directly from structured databases in response to user queries.
- Semantic Searches: Shows how to perform advanced semantic searches within an OpenSearch Vector index by integrating Langchain's ConversationalRetrievalChain and OpenSearch, enhancing the AI's ability to understand and retrieve relevant information based on the context of the conversation.
- Triggering Lambda Functions: Illustrates how to execute custom Lambda functions for specific tasks through Langchain's Custom Chain, allowing for a wide range of actions, from data manipulation to initiating workflows, based on user requests.
- Specialized Chatbot Interactions: Explains how to use Langchain's LLMChain for engaging in specialized conversations and performing tasks by leveraging large language models, ensuring that the AI can handle a variety of user intents with high precision.
- Natural Conversational Engagement: Focuses on using the same LLMChain to maintain a natural and fluid conversation flow, making interactions feel more human-like and intuitive.
A key aspect covered is the preservation of conversation context and chat history, which is crucial for the AI to understand the user's ongoing intent and provide responses that are coherent and contextually enriched. Please review the Memory implementation section for an overview.
- User Input Reception: The user presents a question/query to the Conversational AI system.
- Initial LLM Evaluation: An LLM evaluates each question along with the chat history from the same session to determine its nature and which subject area it falls under (e.g., SQL, action, search, SME).
- Router Chain Activation:
- If the question is identified with a subject, the Router Chain directs it to the corresponding Destination Chain.
- The Default Chain handles queries that don't match specific subjects, providing insights through the Bedrock Model.
- Subject-Specific Routing:
- SQL-related queries are sent to the SQL Destination Chain for database interactions.
- Action-oriented questions trigger the Custom Lambda Destination Chain for executing operations.
- Search-focused inquiries proceed to the RAG Destination Chain for information retrieval.
- SME-related questions go to the SME/Expert Destination Chain for specialized insights.
- Destination Chain Processing: Each Destination Chain takes the input and executes the necessary models or functions:
- SQL Chain uses Amazon Athena for executing queries.
- RAG Chain utilizes Amazon OpenSearch for semantic searches.
- Custom Lambda Chain executes AWS Lambda functions for actions.
- SME/Expert Chain provides insights via the Bedrock Model.
- Response Generation and Delivery: Responses from each Destination Chain are formulated into coherent insights by the LLM. These insights are then delivered to the user, completing the query cycle.
We use Langchain’s Memory system to add context for the LLM based on previous interactions. For example, if a user asks questions about a device, the LLM can determine the correct device ID as long as it was mentioned in a previous message. We use RunnableWithMessageHistory to add memory to specific chains. There are multiple memory implementations to store and retrieve history. We use AWS DynamoDB with langchain's DynamoDBChatMessageHistory to externalize memory storage which allows for a loosely coupled design. It also provides persistent memory so that the user can recall and continue a previous chat session. This is shown in the following image.
This folder helps to set up the multi-route chain app using CDK. Specifically, it contains the following stacks:
- MultiRouteChainBaseInfraStack: it creates an S3 bucket and uploads all the data to be used by the Multi-route Chain APP to that bucket. It also creates the DynamoDB table to store chat history.
- MultiRouteChainSqlChainStack: it creates a Glue Crawler to crawl the data in the data bucket and sets up Athena data catelog.
- MultiRouteChainRagStack: it creates an OpenSearch serverless collection for vector search. And add the data into the index
docs. - MultiRouteChainActionLambdaStack: it creates an action lambda to send SES email. Please consider to implement your own logic to perform actions on the device
- MultiRouteChainFrontendStack: it creates a Streamlit app running on ECS Fargate to interact with the LLM.
- The solution is only available in
us-east-1andus-west-2. Please choose either of them to proceed. - Enable models in Amazon Bedrock: for this use case, you need to enable Anthropic Claude V2 and Titan Embeddings models
- SES setup (verify email)
ses_action_lambdaLambda function implements the logic of sending email notificaiton. It is required to setup SES and verify and sender and recipient's emails beforehand. - Get started with CDK.
- Install Docker. Because we are bundling Lambda functions when running CDK so we need to install Docker. Please see the blog post about Building, bundling and deploying applications with the AWS CDK
- Clone the repository.
- Export
AWS_DEFAULT_REGIONto either region following the CDK cli guide.
export AWS_DEFAULT_REGION={us-east-1, us-west-2}
-
Change directory to
langchain-multi-route-implementation -
To manually create a virtualenv on MacOS and Linux:
python3 -m venv .venv -
After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.
source .venv/bin/activateIf you are a Windows platform, you would activate the virtualenv like this:
.venv\Scripts\activate.bat -
Once the virtualenv is activated, you can install the required dependencies.
pip install -r ../requirements.txt -
CDK bootstrap
cdk bootstrap -
At this point you can now synthesize the CloudFormation template for this code.
cdk synth -
Deploy the application
cdk deploy --all \ --require-approval never \ --parameters MultiRouteChainActionLambdaStack:sender=<The SES verified sender's email> \ --parameters MultiRouteChainActionLambdaStack:recipient=<The SES verified recipient's email>
Once the CDK deployment has completed, there will be a Streamlit URL printed out alongside with deployment time in CDK Outputs.
Run the following commands to destroy all Stacks.
cdk destroy --all
Enter y upon the prompt to destroy each Stack.
This section outlines additional security and cost related considerations for the solution
Secure CDK deployments with IAM permission boundaries
To initialize the CDK stack, we run cdk bootstrap which by default grants full AdministratorAccess to the CloudFormation deployment. For production deployments, we recommend following the CDK security and safety development guide to properly configure least-privilege IAM permissions.
Troubleshooting
While interacting with the application you may encounter the following error in case an exception is thrown during request processing: Error occurred when calling MultiRouteChain. Please review application logs for more information.. In that case, please review the associated CloudWatch Logs log group for the exception name and detailed error message. To locate the correct log group, please navigate to the CloudWatch dashboard from the AWS Management Console and select Log groups from the left panel. Enter Streamlit in the log groups search box and select the log group title which begins with MultiRouteChainFrontendStack. Here you can select the Log stream associated with the error timeframe to troubleshoot further.
CDK Nag Suppressions
cdk-nag is a tool used with the AWS Cloud Development Kit (CDK) to perform static analysis on your CDK application. It can help ensure the application complies with best practices and certain security baseline rules. Certain cdk-nag findings are suppressed for this solution which can be reviewed in the app.py file.
CloudFront TLS Certificate Security Policy
This soluton uses the Default CloudFront Certificate (*.cloudfront.net) which automatically sets the security policy to TLSv1. You may enforce a higher TLS version by associating your own TLS certificate to the CloudFront Distribution. Please reference Supported protocols and ciphers between viewers and CloudFront and Using HTTPS with CloudFront
KMS Encryption for Athena query results
The SQL Database chain in this solution uses Amazon Athena to run SQL queries on an S3 data source registered with the AWS Glue Data Catalog. You can optionally enable encryption at rest for Athena query results. Please follow the documentation link here for more information Encrypting Athena query results stored in Amazon S3.
Load balancer access logs and CloudWatch Container Insights
This CDK project defines an Application Load Balancer as part of the ApplicationLoadBalancedFargateService construct. This contruct provides a Fargate service running on an ECS cluster fronted by an application load balancer. It is defined in the frontend Stack definition file frontend_stack.py. The following features may be relevant for monitoring a troubleshooting the application.
-
Elastic Load Balancing provides access logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses. You can use these access logs to analyze traffic patterns and troubleshoot issues.
-
You can use CloudWatch Container Insights to collect, aggregate, and summarize metrics and logs from your containerized applications and microservices. CloudWatch automatically collects metrics for many resources, such as CPU, memory, disk, and network. Container Insights also provides diagnostic information, such as container restart failures, to help you isolate issues and resolve them quickly. To enable Container Insights