Plugin system

[luandro]

Create a plugin system built around the FastAPI web framework, which will expose endpoints for various plugins. This system will route incoming requests to specific plugins based on the defined routes, validate requests, execute plugin logic, handle responses, and update relevant database tables. Additionally, the system will manage error handling, response queuing, and monitoring.

Requirements

1. API Endpoints:

   • Define REST API routes that allow interaction with different plugins.
     • POST /api/{plugin}/{intent} - Route for plugin-specific actions based on the intent provided.
     • GET /api/status - Endpoint to retrieve system status.

2. Plugin Router:

   • Request Validation: Ensure requests are correctly structured and secure.
   • Plugin Logic Execution: Route to the appropriate plugin, executing logic based on the intent.
   • Response Handling: Process the response, either updating the database or sending it to the error handler if unsuccessful.

3. Database Update Logic:

   • Begin a transaction, update necessary tables, and commit the transaction.
   • Rollback in case of failures to maintain database integrity.

4. Queue Response Management:

   • Calculate the appropriate send time for the response.
   • Format and store the response in the tosend-messages table for later dispatch.

5. Error Handling:

   • Log error details and save failed requests to a dead-letter queue (DLQ) for troubleshooting.
   • Implement a retry mechanism to reprocess failures as needed.

6. Monitoring:

   • Track and monitor metrics such as:
     • API performance
     • Plugin success rates
     • Database transaction metrics
     • Queue processing metrics

Diagram

flowchart TB
    subgraph "Plugin System"
        API["REST API:
            Routes:
            - POST /api/{plugin}/{intent}
            - GET /api/status"]
            
        Router["Plugin Router:
            1. Validate request
            2. Execute plugin logic
            3. Handle response"]
            
        DBUpdate[" Updates:
            - Begin transaction
            - Update tables
            - Commit/rollback"]
            
        Queue["Queue Response:
            1. Calculate send time
            2. Format response
            3. Store in tosend-messages"]
            
        Error["Error Handler:
            - Log details
            - Save to DLQ
            - Retry mechanism"]

        API --> Router
        Router -->|Success| DBUpdate
        Router -->|Failure| Error
        DBUpdate --> Queue
        Error -->|Retry| Router
    end
    
    subgraph "Database Tables"
        Users[(users)]
        Territories[(territories)]
        Ethnic[(ethnic_groups)]
        Projects[(projects)]
        
        DBUpdate --> Users
        DBUpdate --> Territories
        DBUpdate --> Ethnic
        DBUpdate --> Projects
    end
    
    ToSend[(tosend-messages)]
    Queue --> ToSend
    
    Monitor["Monitoring:
        - API performance
        - Plugin success rates
        - Database metrics
        - Queue metrics"]
        
    Monitor -.->|Track| Router

Loading

Acceptance Criteria

• FastAPI routes are defined and can direct requests to the appropriate plugin.
• Plugin router validates requests, processes plugin logic, and updates database tables successfully.
• Queue responses are correctly timed, formatted, and stored in tosend-messages.
• Error handling mechanism logs issues, sends them to the DLQ, and retries failed requests.
• Monitoring metrics are tracked and logged for system performance insights.