Fix tests; Add READMEs; Rename persistence->repos

README.md

@@ -2,45 +2,41 @@
 
 ## Overview
 
-Project Nalgonda is an innovative platform for managing and executing AI-driven swarm agencies.
-Built upon the [OpenAI Assistants API](https://platform.openai.com/docs/assistants/overview),
-it extends functionality through specialized tools and a sophisticated management system for AI agencies.
-It combines robust FastAPI architecture, Firebase Firestore, and OpenAI's GPT models for dynamic agency
-and agent management.
+Project Nalgonda is an advanced platform designed for the orchestration and operational management of AI-driven swarm agencies. Leveraging [OpenAI Assistants API](https://platform.openai.com/docs/assistants/overview), it enriches the capabilities of the AI agencies with specialized tools and a robust management system. Built on a solid FastAPI framework and employing Firebase Firestore along with OpenAI's GPT models, it enables dynamic agency and agent management at scale.
 
 ## Key Components
 
-- **Configuration Managers**: Manage configurations for AI agencies, agents and tools.
-- **WebSocket Connection Manager**: Handles WebSocket connections for interactive agency-client communication.
-- **Custom Tools**: Includes tools like `SearchWeb`, `GenerateProposal`, `BuildDirectoryTree`, and more.
-- **FastAPI Web Server**: For API routing, CORS middleware, Firebase initialization, and WebSocket communication.
-- **Data Models**: Pydantic models for agencies, agents, and tool configurations and request validation.
-- **Data Persistence**: Firestore for storing tool, agent, and agency configurations.
-- **Caching**: Redis for efficient caching of agency states.
+- **Configuration Managers**: Centralized management of configurations for agencies, agents, and individual tools.
+- **WebSocket Connection Manager**: Ensures real-time interactive communication between agencies and clients through WebSocket connections.
+- **Custom Tools**: A suite of specialized tools including `SearchWeb`, `GenerateProposal`, `BuildDirectoryTree`, among others, designed to augment the functionalities of the AI agents.
+- **FastAPI Web Server**: Manages API routing, initializes CORS middleware, sets up Firebase, and facilitates WebSocket communication.
+- **Data Models**: Utilizes Pydantic models for defining and validating configurations of agencies, agents, tools, as well as the structure of request data.
+- **Repositories**: Utilizes Firestore for robust storage and querying capabilities for tool, agent, and agency configurations -- ensuring a seamless persistence layer.
+- **Caching**: Employs Redis for efficient and scalable caching of agency states (sessions) for optimized performance.
 
 ## Features
 
-- **Tool Configuration**: Configure tools with code and parameters.
-- **Agent Configuration**: Configure agents with knowledge and tools.
-- **Agency Configuration**: Set up agencies with agents.
-- **Tool Execution**: Execute tools for various tasks.
-- **User Management**: Manage user access to different agencies [TODO].
-- **API and WebSocket Routers**: Define API endpoints and WebSocket routes.
-- **Security**: Basic implementations of JWT authentication and authorization [TBD].
+- **Tool Configuration**: Offers extensive flexibility in customizing tools with specific codes and parameters for varied tasks.
+- **Agent Configuration**: Allows detailed setup of agents with specialized knowledge areas and toolsets.
+- **Agency Configuration**: Facilitates the creation and management of agencies composed of configured agents.
+- **Tool Execution**: Executes tools within an established framework for accomplishing a wide range of tasks.
+- **User Management**: User access management features for interaction with different agencies.
+- **API and WebSocket Routers**: Lays down a comprehensive set of API endpoints and WebSocket routes for external interactions and real-time communications.
+- **Security**: Basic implementation of JWT for user authentication and authorization with plans for further enhancements.
 
 ## Installation
 
-1. Ensure Python 3.11+ and Node.js 20.11+ are installed.
-2. Install Python dependencies (from `requirements.txt` or using Poetry).
-3. Set up environment variables in ".env", reference in ".env.testing".
-   - Use `cat ~/ai-in-hand-firebase-adminsdk-....json | jq -c .` for Google Credentials.
-4. In `frontend` directory, run:
-   - for local development: `npm install && npm run start`
-   - for production (builds to nalgonda/ui/ directory and is served by FastAPI):
-   `npm install -g gatsby-cli && npm install --global yarn && yarn install && yarn build`
+Follow these steps for setting up the environment and running the Nalgonda project locally:
+
+1. Ensure Python 3.11+ and Node.js 20.10+ are installed.
+2. Install Python dependencies either from `requirements.txt` or using Poetry.
+3. Configure environment variables in ".env", taking ".env.testing" as a reference point (used only for local development).
+4. To set up the frontend:
+   - For local development: `npm install && npm run start`
+   - For production: `npm install -g gatsby-cli && npm install --global yarn && yarn install && yarn build` (builds to nalgonda/ui/ directory to be served by FastAPI)
 
 ### Running the Application
-Start the FastAPI server: `uvicorn nalgonda.main:app --reload`
+Start the FastAPI server with: `uvicorn nalgonda.main:app --reload`
 
 ## Deployment to Heroku
 
@@ -54,11 +50,10 @@ Start the FastAPI server: `uvicorn nalgonda.main:app --reload`
 ## Usage
 
 ### API Endpoints
-To interact with the platform, use the Postman collection, which includes all necessary routes and variables for testing.
+The provided Postman collection encompasses all the necessary routes and variables, facilitating extensive testing and interaction with the platform.
 
 ### WebSocket Communication
-Connect to WebSocket endpoints (`/v1/ws/{agency_id}`, `/v1/ws/{agency_id}/{session_id}`)
-for real-time communication with AI agencies.
+Outlines the process for establishing WebSocket connections (`/v1/ws/{agency_id}/{session_id}`) for real-time interactions.
 
 ## Areas for Improvement
-- Enhanced exception handling, security, documentation, testing, caching logic, and database interactions.
+Enhance exception handling, security measures, documentation quality (e.g. docstrings), testing robustness.

nalgonda/README.md

@@ -0,0 +1,30 @@
+# Nalgonda Project Overview
+
+The Nalgonda project is a comprehensive backend application designed to manage agencies, agents, and their interactions.
+It employs a variety of custom tools, data models, persistent storage mechanisms, backend services,
+external dependencies, and routed endpoints to deliver a fully functional system for agency management.
+
+## Directory Overview
+
+- **Custom Tools**: Contains tools for directory structure printing, proposal generation, file content printing,
+Airtable integration, web searching, code summarization, and program writing based on specific parameters.
+
+- **Models**: Defines data structures and logic for agency configurations, agent details, authentication,
+request handling, and tool configurations, ensuring data integrity and operation logic.
+
+- **Repositories**: Manages Firestore interactions for data storage and retrieval,
+implementing functionalities for agency and agent configurations, tool data, and user information management.
+
+- **Services**: Provides backend functionalities including agency and agent management, caching,
+real-time WebSocket communication, and threading.
+
+- **Dependencies**: Manages external service integrations and real-time communication necessities,
+including OAuth2 authentication, dependency injections, and WebSocket connections.
+
+- **Routers**: Orchestrates the application's request routing logic, handling API endpoint definitions
+across various versions and functionalities like agency operations, agent management, authentication,
+and session handling.
+
+This document serves to provide a broad overview of the project's structure and should be supplemented
+by reading the detailed README files within each directory for a comprehensive understanding
+of each component's functionality.

nalgonda/custom_tools/README.md

@@ -0,0 +1,26 @@
+# Custom Tools Directory
+
+This directory contains a collection of custom tools developed to extend the functionalities of the Nalgonda project.
+Each tool performs a specific task and can be called upon when needed.
+
+## Tools and Descriptions
+
+- **Build Directory Tree**: Prints the structure of directories and files within a specified directory
+while preventing directory traversal.
+
+- **Generate Proposal**: Generates a proposal for a project based on a project brief.
+Uses a GPT model to generate text based on input.
+
+- **Print File Contents**: Prints the contents of a specific file, preventing directory traversal.
+
+- **Save Lead to Airtable**: Saves new lead information to Airtable, including name, email, and lead details.
+
+- **Search Web**: Performs a web search and returns the results. It uses the `duckduckgo_search` library.
+
+- **Summarize Code**: Summarizes the code of a specified file using GPT-3.
+It relies on the `PrintFileContents` tool to access the code text.
+
+- **Utils**: Contains utility functions including directory traversal checks.
+
+- **Write and Save Program**: Writes and saves files that represent a program/application
+to the disk based on specified parameters.

nalgonda/custom_tools/__init__.py

@@ -3,7 +3,7 @@
 
 from nalgonda.custom_tools.build_directory_tree import BuildDirectoryTree
 from nalgonda.custom_tools.generate_proposal import GenerateProposal
-from nalgonda.custom_tools.print_file import PrintFileContents
+from nalgonda.custom_tools.print_file_contents import PrintFileContents
 from nalgonda.custom_tools.save_lead_to_airtable import SaveLeadToAirtable
 from nalgonda.custom_tools.search_web import SearchWeb
 from nalgonda.custom_tools.summarize_code import SummarizeCode

nalgonda/custom_tools/print_file.py → nalgonda/custom_tools/print_file_contents.py

@@ -9,9 +9,7 @@
 class PrintFileContents(BaseTool):
     """Print the contents of a specific file."""
 
-    file_name: Path = Field(
-        ..., description="The name of the file to be printed. It can be a relative or absolute path."
-    )
+    file_name: Path = Field(..., description="The name of the file to be printed. It can be a relative path.")
 
     _validate_file_name = field_validator("file_name", mode="after")(check_directory_traversal)
 

nalgonda/dependencies/README.md

@@ -0,0 +1,9 @@
+# Dependencies Directory
+
+This directory is crucial for managing the external dependencies and service interfaces used throughout the project.
+It contains:
+
+- `auth.py`: Handles user authentication and authorization with OAuth2, including JWT token management.
+- `dependencies.py`: Centralizes dependency injections for Redis, Agency, Agent, and Thread managers among others,
+ensuring modularity and ease of use.
+- `websocket_connection_manager.py`: Manages WebSocket connections enabling real-time communication in the application.

nalgonda/dependencies/auth.py

@@ -6,7 +6,7 @@
 from starlette.status import HTTP_400_BAD_REQUEST, HTTP_403_FORBIDDEN
 
 from nalgonda.models.auth import TokenData, UserInDB
-from nalgonda.persistence.user_repository import UserRepository
+from nalgonda.repositories.user_repository import UserRepository
 from nalgonda.settings import settings
 
 oauth2_scheme = OAuth2PasswordBearer(tokenUrl="v1/api/token")

nalgonda/dependencies/dependencies.py

@@ -1,8 +1,8 @@
 from fastapi import Depends
 from redis import asyncio as aioredis
 
-from nalgonda.persistence.agency_config_firestore_storage import AgencyConfigFirestoreStorage
-from nalgonda.persistence.agent_config_firestore_storage import AgentConfigFirestoreStorage
+from nalgonda.repositories.agency_config_firestore_storage import AgencyConfigFirestoreStorage
+from nalgonda.repositories.agent_config_firestore_storage import AgentConfigFirestoreStorage
 from nalgonda.services.agency_manager import AgencyManager
 from nalgonda.services.agent_manager import AgentManager
 from nalgonda.services.caching.redis_cache_manager import RedisCacheManager

nalgonda/models/README.md

@@ -0,0 +1,17 @@
+# Models Directory
+
+This directory contains the data models used across the Nalgonda project.
+These models define the data structures, validation logic, and relationships between data entities.
+They are essential for the proper functioning of the application, ensuring data integrity and facilitating data operations.
+
+## Models and Descriptions
+
+- **AgencyConfig**: Represents the configuration of an agency, including identifiers, names, instructions, agents, and the agency chart.
+
+- **AgentConfig**: Defines the configuration for an individual agent, detailing identifiers, descriptions, instructions, and tools.
+
+- **ToolConfig**: Configures tools within the application, describing tool identifiers, names, versions, and approval status.
+
+- **Auth Models**: Includes models for token generation, user details, and authentication processes.
+
+- **Request Models**: Covers models for handling requests, specifically for threading and messaging within an agency.

nalgonda/repositories/README.md

@@ -0,0 +1,20 @@
+# Repositories Directory
+
+This directory is responsible for all persistent storage interactions within the Nalgonda project,
+particularly with Firestore. It contains implementations for storing, retrieving, and managing configurations
+and data for agencies, agents, tools, and users.
+
+## Files and Descriptions
+
+- **AgencyConfigFirestoreStorage.py**: Manages the persistence of agency configurations in Firestore,
+providing methods to load and save these configurations based on owner or agency IDs.
+
+- **AgentConfigFirestoreStorage.py**: Similar to its agency counterpart, this file handles the persistence
+of agent configurations, offering functionalities to load and save agent data.
+
+- **ToolConfigFirestoreStorage.py**: Facilitates storing and retrieving tool configurations in Firestore.
+This includes handling tool data validation and manipulation.
+
+- **UserRepository.py**: Manages user data within Firestore, including functions to retrieve and update user records.
+This setup reinforces the project's data integrity and provides a seamless integration with Firebase's Firestore
+for its storage needs.

nalgonda/routers/README.md

@@ -0,0 +1,11 @@
+# Routers Directory
+
+This directory defines the API routing logic for the project, organizing routes into specific functionalities
+and versions. Key components include:
+
+- `__init__.py` & `v1/__init__.py`: Establish the application's routing logic, categorizing the APIs and handling errors.
+- `agency.py`, `agent.py`, `auth.py`, `session.py`, `tool.py`: Define the endpoints for managing agencies, agents,
+authentication, sessions, and tools, respectively.
+- `websocket.py`: Sets up WebSocket endpoints for real-time messaging.
+
+The Swagger documentation is available at `/v1/docs`.

nalgonda/routers/v1/api/agency.py

@@ -9,7 +9,7 @@
 from nalgonda.dependencies.dependencies import get_agency_manager, get_agent_manager
 from nalgonda.models.agency_config import AgencyConfig
 from nalgonda.models.auth import UserInDB
-from nalgonda.persistence.agency_config_firestore_storage import AgencyConfigFirestoreStorage
+from nalgonda.repositories.agency_config_firestore_storage import AgencyConfigFirestoreStorage
 from nalgonda.services.agency_manager import AgencyManager
 from nalgonda.services.agent_manager import AgentManager
 

nalgonda/routers/v1/api/agent.py

@@ -9,7 +9,7 @@
 from nalgonda.dependencies.dependencies import get_agent_manager
 from nalgonda.models.agent_config import AgentConfig
 from nalgonda.models.auth import UserInDB
-from nalgonda.persistence.agent_config_firestore_storage import AgentConfigFirestoreStorage
+from nalgonda.repositories.agent_config_firestore_storage import AgentConfigFirestoreStorage
 from nalgonda.services.agent_manager import AgentManager
 
 logger = logging.getLogger(__name__)

nalgonda/routers/v1/api/session.py

@@ -9,7 +9,7 @@
 from nalgonda.dependencies.dependencies import get_agency_manager, get_thread_manager
 from nalgonda.models.auth import UserInDB
 from nalgonda.models.request_models import AgencyMessagePostRequest, ThreadPostRequest
-from nalgonda.persistence.agency_config_firestore_storage import AgencyConfigFirestoreStorage
+from nalgonda.repositories.agency_config_firestore_storage import AgencyConfigFirestoreStorage
 from nalgonda.services.agency_manager import AgencyManager
 from nalgonda.services.thread_manager import ThreadManager
 
@@ -80,7 +80,9 @@ async def post_agency_message(
         raise HTTPException(status_code=HTTP_404_NOT_FOUND, detail="Agency not found")
 
     try:
-        response = await asyncio.to_thread(agency.get_completion, message=user_message, yield_messages=False)
+        response = await asyncio.to_thread(
+            agency.get_completion, message=user_message, yield_messages=False, message_files=None
+        )
         return {"response": response}
     except Exception as e:
         logger.exception(e)

nalgonda/routers/v1/api/tool.py

@@ -6,7 +6,7 @@
 from nalgonda.dependencies.auth import get_current_active_user, get_current_superuser
 from nalgonda.models.auth import UserInDB
 from nalgonda.models.tool_config import ToolConfig
-from nalgonda.persistence.tool_config_firestore_storage import ToolConfigFirestoreStorage
+from nalgonda.repositories.tool_config_firestore_storage import ToolConfigFirestoreStorage
 from nalgonda.services.tool_service import generate_tool_description
 
 tool_router = APIRouter(tags=["tool"])

nalgonda/routers/v1/websocket.py

@@ -7,8 +7,8 @@
 from websockets.exceptions import ConnectionClosedOK
 
 from nalgonda.dependencies.dependencies import get_agency_manager
-from nalgonda.dependencies.websocket_connection_manager import WebSocketConnectionManager
 from nalgonda.services.agency_manager import AgencyManager
+from nalgonda.services.websocket_connection_manager import WebSocketConnectionManager
 
 logger = logging.getLogger(__name__)
 connection_manager = WebSocketConnectionManager()

nalgonda/services/README.md

@@ -0,0 +1,21 @@
+# Services Directory
+
+This directory contains critical components that constitute the backend services of the Nalgonda project.
+These services enable various functionalities from managing entities like agencies and agents to handling caching,
+WebSocket connections, and threading.
+
+## Services and Descriptions
+
+- **Agency Manager**: Manages agencies and their configurations, including retrieval, update, and caching of agency information.
+
+- **Agent Manager**: Manages agents including creating, updating, and retrieving agent information.
+
+- **Cache Manager**: Defines an abstract base for cache management operations.
+
+- **Redis Cache Manager**: Implements cache management using Redis as the backend.
+
+- **Thread Manager**: Manages communication threads between agents within an agency, enabling threaded interactions.
+
+- **Tool Service**: Provides tool description generation using GPT-3.
+
+- **WebSocket Manager**: Manages WebSocket connections between the frontend and backend.

nalgonda/services/agency_manager.py

@@ -5,7 +5,7 @@
 from agency_swarm import Agency, Agent, get_openai_client
 
 from nalgonda.models.agency_config import AgencyConfig
-from nalgonda.persistence.agency_config_firestore_storage import AgencyConfigFirestoreStorage
+from nalgonda.repositories.agency_config_firestore_storage import AgencyConfigFirestoreStorage
 from nalgonda.services.agent_manager import AgentManager
 from nalgonda.services.caching.redis_cache_manager import RedisCacheManager
 

nalgonda/services/agent_manager.py

@@ -5,7 +5,7 @@
 
 from nalgonda.custom_tools import TOOL_MAPPING
 from nalgonda.models.agent_config import AgentConfig
-from nalgonda.persistence.agent_config_firestore_storage import AgentConfigFirestoreStorage
+from nalgonda.repositories.agent_config_firestore_storage import AgentConfigFirestoreStorage
 from nalgonda.settings import settings
 
 logger = logging.getLogger(__name__)