Skip to content

Commit

Permalink
Readme 4 (#13)
Browse files Browse the repository at this point in the history
* Update README.md to enhance documentation on services, dependencies, and new chatbot integration

* Update README.md to improve image presentation

* Update README.md to enhance image presentation

---------

Co-authored-by: Jamie Liu <[email protected]>
  • Loading branch information
99-j and Jamie Liu authored Dec 13, 2024
1 parent 2831d21 commit 872340e
Show file tree
Hide file tree
Showing 3 changed files with 69 additions and 20 deletions.
89 changes: 69 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ These services work together to form a stable and efficient microservices applic

### Diagram

![AirSense-design](/imgs/airsense_diagram.png)
<img src="/imgs/airsense-2.png" alt="airsense-2" style="width:70%;">

### Services

Expand All @@ -54,9 +54,30 @@ Designed for **Publics**.

`AirGateway` is powered by Flask and aims to handle users' and administrators' restful API requests and forward them to proper mircro-services (`AirConnector`, `AirStore`). It has the ability to handle at least 10,000 requests per seconds.

- Query test example:
1. Users can utilize the advanced search function to query any keywords throughout the entire database, with results returned in milliseconds.

`[host_uri]/api/dynamic_query?table_name=busiest_airports_by_delays&limit=6&where=Avg_Departure_Delay:87,Total_Arrivals:4`
<div style="display: flex; justify-content: space-around;">
<img src="/imgs/home-1.png" alt="home-1" style="width:45%;">
<img src="/imgs/home-2.png" alt="home-2" style="width:45%;">
</div>

2. Users can interact with an interactive database exploration tool, enabling them to view data by order, ascending, descending, limit, and more.

<div style="display: flex; justify-content: space-around;">
<img src="/imgs/explore-1.png" alt="explore-1" style="width:45%;">
<img src="/imgs/explore-2.png" alt="explore-2" style="width:45%;">
</div>

3. Gateway also provides powerful query API portal, allowing user and other services query data via Restful API easily.

<div style="display: flex; justify-content: space-around;">
<img src="/imgs/api-1.png" alt="api-1" style="width:45%;">
<img src="/imgs/api-2.png" alt="api-2" style="width:45%;">
</div>

Query test example:

`[host_uri]/api/dynamic_query?table_name=busiest_airports_by_delays&limit=6&where=Avg_Departure_Delay:87,Total_Arrivals:4`

#### AirConnector

Expand All @@ -65,47 +86,71 @@ Designed for **Administrator**.
- _Access using the following link:
[https://pywepkfr9p.us-east-2.awsapprunner.com/](https://pywepkfr9p.us-east-2.awsapprunner.com/)_

`AirConnector` connectes `AirBroker` and `AirStore`. It transfers data tables from databricks to `AirStore` and ready for query by `AirGateway`.
`AirConnector` connectes `AirBroker` and `AirStore`. It transfers data tables from databricks to `AirStore` and ready for query by `AirGateway`. Administrator can check the status of two database services, manually transfer data, and see history of operations.

- Transfering test example:

`curl -X POST "[host_uri]:8000/transfer_table?table_name=delays_by_day`

### Dependencies
<img src="/imgs/admin.png" alt="AdminDash" style="width:600px;">

## Leveraging LLM for Chatbot

We have integrated an advanced chatbot powered by OpenAI's API. This allows users to interact with the application through natural language conversations. Users can ask questions and get insights about flights, data, and any concern while using this application

### Key Features:

- **Natural Language Processing**: Understands and processes user queries in plain English.
- **Real-time Responses**: Provides instant answers to user questions.
- **Contextual Understanding**: Maintains context to handle follow-up questions effectively.

1. Languages and Frameworks:
<img src="/imgs/chat.png" alt="Chatbot" style="width:600px;">

- Python 3.12.8 with Flask and FastAPI for API handling.
## Dependencies

- SQL for querying and GraphSQL database for data storage.
### Languages and Frameworks:

2. Tools and Services:
- Python 3.12.8

- Databricks: For data processing.
- Flask: For powering `AirGateway` user interface and Restful API requests.

- Docker: Containerization of services for consistency and portability.
- FastAPI: For powering `AirConnector` and administrator interface.

- AWS App Runner: For auto-scaling microservices.
- Spark: For preprocessing raw data and outputting clean tables in Databricks.

- AWS RDS Aurora: As the relational database solution.
- SQL: For querying and GraphSQL database for data storage.

- Locust: For load testing Flask applications.
### Tools and Services:

- CloudFormation: Infrastructure as Code for streamlined AWS resource provisioning.
- Databricks: For data preprocessing.

3. Development Tools:
- Docker: Containerization of services for consistency and portability.

- Pyenv: Python version management.
- AWS Elastic Container Registry (ECR): Store, manage, and deploy container images securely.

- Virtual Environment: For dependency isolation during development.
- AWS App Runner: For auto-scaling microservices.

- GitHub Actions: CI/CD pipelines for automated testing and deployment.
- AWS RDS Aurora: As the relational database solution.

- Locust: For load testing Flask applications.

- CloudFormation: Infrastructure as Code for streamlined AWS resource provisioning.

- CloudWatch: For monitoring and observability of AWS resources and applications.

### Development Tools:

- Pyenv: Python version management.

- Virtual Environment: For dependency isolation during development.

- GitHub Actions: CI/CD pipelines for automated testing and deployment.

## Development

For details regarding repo structure and how to set up development environment, see [development.md](/development.md).

### Logging
## Logging

We use AWS CloudWatch for logging. AWS CloudWatch is a
monitoring and observability service provided by AWS.
Expand Down Expand Up @@ -193,3 +238,7 @@ Resources:

All reflections including team member self reflection files and
team reflection after meeting are included in the `Team_Reflections` folder in this repository.

## About

![about](/imgs/about.png)
Binary file added imgs/api-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added imgs/api-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 872340e

Please sign in to comment.