Beta2: PDF File Summarisation. briefops-status and briefops-ingest co…

…mmands

README.md

@@ -1,6 +1,6 @@
 # BriefOps
 
-**BriefOps** is a Slack app that helps teams stay on top of important conversations by summarising Slack channels over specific periods. Built on **Google Cloud**, it leverages **Vertex AI** for summarisation, offering a fully secure, private deployment in your own Google Cloud environment.
+**BriefOps** is a Slack app that helps teams stay on top of important conversations by summarising Slack Uploaded Documents, and Slack channels over specific periods,. Built on **Google Cloud**, it leverages **Vertex AI** for summarisation, offering a fully secure, private deployment in your own Google Cloud environment.
 
 ## Published Blog
 
@@ -12,15 +12,18 @@ The actualy hands-on Experience writing up this App/Integration is published on
 
 It's A Slack Channel Summarisation App, hosted on Google Cloud, uses Gemini generative AI/LLM to summarise long Channel Conversaions (and uploaded Documents - due shortly)
 
-`This is Beta1 Release: Slack Channel Summarisation capability only`
+`November 2024: This is Beta2 Release: Slack Channel, and Uploaded PDF Document Summarisation capabilities.`
 
-![About BriefOps](images/about-briefops-slack-app.png "About BriefOps")
+![About BriefOps](images/About_BriefOps_App.png "About BriefOps")
 
 ## Features
 
 - **Channel Summarisation**: Use `/briefops` to summarise conversations over a set period, helping users catch up on important updates. _(default over last 7 days)_
 - **Secure and Private**: Fully deployed within your own Google Cloud project, ensuring enterprise-level privacy and security.
 - **Customisable Deployments**: Choose the Google Cloud region and adjust configurations to meet your operational needs.
+- **Expanding Functionality:** Moving beyond Slack conversations, BriefOps can now handle internal documents with ``/briefops-ingest <slack_file>`, making it a more versatile tool for information management.
+- **Improving User Experience:** By providing status updates with `/briefops-status` and usage information, you can better manage your interactions with the service, leading to increased satisfaction.
+- **Driving Efficiency:** Summarizing external content and providing usage insights helps teams work more efficiently, focusing on critical information and optimizing resource use.
 
 ## Getting Started
 
@@ -36,15 +39,77 @@ Before starting, ensure you have the following:
 
 ![BriefOps Solution HLD](images/architecture-HLD.png "BriefOps Solution HLD C4")
 
-
 This user-centric view better shows how Slack commands directly lead to Google Cloud components interacting and how the Slack User remains the main stakeholder throughout the summarisation process. This also emphasises privacy and clarity, indicating where processing occurs and how responses come back to the initiating user or thread.
 
 
+## How this Works - Usage
+
+
+### `/briefops`
+
+
+- **Channel Summarisation**: The `/briefops` command is a powerful tool for teams, enabling the summarisation of entire Slack channel conversations over a configurable period. This is ideal for large teams and busy channels where members may miss critical updates or need a quick recap.
+  - **Default Time Frame**: By default, `/briefops` summarises the last 7 days, making it easy to catch up on a week's worth of discussions.
+  - **Custom Time Frame**: Users can specify a custom time frame by entering the desired number of days (e.g., `/briefops 14` for the last 14 days), allowing for targeted summarisation.
+  - **Summarisation with Contextual Insights**: The command leverages Vertex AI's latest Gemini models, which provide contextually rich and concise summaries, capturing essential points and highlighting key takeaways.
+  - **Automated Thread Management**: Summaries are automatically posted back into the channel, creating a single-threaded history of key points that team members can reference. This helps reduce the time spent manually scanning or summarising long conversations and increases productivity.
+  - **Enhanced Meeting and Project Workflow**: Teams can use `/briefops` to summarise critical meetings, project updates, or collaborative discussions, creating a concise record for all members to review and act on.
+
+The Example of Summarisation of the Channel, along with Document summaries:
+
+![BriefOps Channel Summary](images/briefops-summarisation-of-ingests.png "BriefOps summary of channel with other summaries")
+
+### `/briefops-ingest`
+
+The `/briefops-ingest` command allows users to ingest external documents or data sources into the BriefOps system for summarization and analysis. This feature extends BriefOps beyond summarizing Slack conversations to include external content, making it a powerful tool for aggregating and summarizing information from multiple sources.
+
+![BriefOps Ingest](images/briefops-ingest-command.png "BriefOps Ingest Document to summarise")
+
+**Usage:**
+
+```/briefops-ingest <Slack file URL>```
+
+**What It Does:**
+
+- Accepts a URL or file as input.
+- Fetches and processes the content securely.
+- Summarizes the content using the same Vertex AI models used for Slack conversations.
+- Provides a concise summary directly within Slack.
+
 
+### `/briefops-status`
+
+The `/briefops-status` command provides users with information about their usage and the status of the BriefOps service. This includes details such as the number of summaries used, remaining daily limits, and any relevant notifications regarding service availability.
+
+![BriefOps Status](images/briefops-status-command.png "BriefOps Status with ingested file stats")
+
+
+**Usage:**
+
+```/briefops-status```
+
+
+**What It Does:**
+
+- Displays your current usage statistics, including:
+  - Number of summaries generated today.
+  - Remaining summaries available under the free tier limit.
+- Provides information about any service updates or maintenance notices.
+- Helps you manage your usage and stay within your allocated limits.
+
+
+## Deploy /BriefOps Application on Slack with Google Cloud
+
+### Step 0: Create your Slack App
+
+Follow the Slack's very own onboarding guide here https://api.slack.com/docs/apps
+
+You'd need all your Slack Bot and App keys and Secrets kept safely to deploy this app.
 
 ### Step 1: Enable Required Google Cloud APIs
 
-export GOOGLE_CLOUD_PROJECT="Your-ProjectID"
+`export GOOGLE_CLOUD_PROJECT="Your-ProjectID"`
+
 Run the following command to enable all necessary Google Cloud services:
 
 ```bash

config.js

@@ -23,7 +23,7 @@ module.exports = {
 
   SUMMARIZATION_MAX_OUTPUT_TOKENS: process.env.SUMMARIZATION_MAX_OUTPUT_TOKENS
     ? parseInt(process.env.SUMMARIZATION_MAX_OUTPUT_TOKENS)
-    : 256, // Default max output tokens
+    : 1000, // Default max output tokens
 
   SUMMARIZATION_TOP_P: process.env.SUMMARIZATION_TOP_P
     ? parseFloat(process.env.SUMMARIZATION_TOP_P)

fs-summary.js

@@ -0,0 +1,134 @@
+const { downloadFile, uploadToGCS } = require('./utils'); // Utilities for file operations
+const { summarizeDocumentFromGCS } = require('./vertexAIClient'); // Summarization function
+const db = require('./firebaseClient'); // Firestore client
+const { VERTEX_AI_MODEL_PDF, GCS_BUCKET_NAME } = require('./config'); // Configurations
+
+module.exports = function (app) {
+  // Command to ingest a document
+  app.command('/briefops-ingest', async ({ command, ack, respond, say }) => {
+    try {
+      await ack();
+
+      const slackFileUrl = command.text.trim(); // Get Slack file URL from the command
+      const fileId = slackFileUrl.split('/').find((part) => part.startsWith('F')); // Extract file ID
+
+      if (!fileId) {
+        throw new Error('Invalid Slack file URL. Could not extract the file ID.');
+      }
+
+      await respond({
+        text: `Ingestion started: Processing the document from the URL: ${slackFileUrl}`,
+      });
+
+      console.log(`[INFO] Received Slack file URL for ingestion: ${slackFileUrl}`);
+      console.log(`[INFO] Extracted file ID: ${fileId}`);
+
+      // Fetch file information from Slack
+      const result = await app.client.files.info({
+        token: process.env.SLACK_BOT_TOKEN,
+        file: fileId,
+      });
+
+      const file = result.file;
+      console.log(`[INFO] File to be ingested: ${file.name}, type: ${file.mimetype}`);
+
+      await respond({
+        text: `Fetching file information from Slack for file ID: ${fileId}...`,
+      });
+
+      // Download the file from Slack
+      const fileContent = await downloadFile(file.url_private_download, file.mimetype);
+      console.log(`[INFO] Downloading the document: ${file.name}`);
+
+      await respond({
+        text: `Downloading the document: ${file.name}...`,
+      });
+
+      // Upload the file to Google Cloud Storage
+      const gcsFileUri = await uploadToGCS(fileContent, file.name, GCS_BUCKET_NAME);
+      console.log(`[INFO] File uploaded to GCS at: ${gcsFileUri}`);
+
+      // Store the file metadata in Firestore for later access
+      await db.collection('ingestedFiles').add({
+        fileId,
+        fileName: file.name,
+        gcsUri: gcsFileUri,
+        createdAt: new Date(),
+      });
+
+      await respond({
+        text: `File uploaded to GCS. Starting Vertex AI summarization for ${file.name}...`,
+      });
+
+      // Summarize the document using Vertex AI
+      const summary = await summarizeDocumentFromGCS(gcsFileUri);
+      if (!summary) {
+        throw new Error('Failed to summarize the PDF.');
+      }
+
+      // Store the summary result in Firestore
+      await db.collection('documentSummaries').add({
+        fileId,
+        fileName: file.name,
+        summary,
+        createdAt: new Date(),
+      });
+
+      // Notify the whole channel about the success of ingestion
+      await say({
+        text: `🎉 The document *${file.name}* has been successfully ingested and summarized. Here’s a brief summary: \n${summary}\nFeel free to query @briefops for further details!`,
+        channel: command.channel_id,
+      });
+
+      console.log(`[INFO] Public notification sent for successful ingestion and summarization of ${file.name}.`);
+
+    } catch (error) {
+      console.error(`[ERROR] Ingestion failed: ${error.message}`);
+
+      // Notify the user about the failure
+      await respond({
+        text: `[ERROR] An error occurred during ingestion: ${error.message}. Please check the logs for more details.`,
+      });
+    }
+  });
+
+  // Handle @briefops mention events
+  app.event('app_mention', async ({ event, say }) => {
+    try {
+      const text = event.text.trim();
+      console.log(`[INFO] @briefops was mentioned with text: ${text}`);
+
+      // Check for document querying requests or conversation summaries
+      if (text.includes('document')) {
+        // Fetch the document details from Firestore
+        const docsSnapshot = await db.collection('ingestedFiles').get();
+        const documents = docsSnapshot.docs.map(doc => doc.data());
+
+        let responseText = 'Here are the documents available for querying:\n';
+        documents.forEach((doc) => {
+          responseText += `• *${doc.fileName}* - available at GCS URI: ${doc.gcsUri}\n`;
+        });
+
+        // Reply with the list of available documents
+        await say({
+          text: responseText,
+          thread_ts: event.thread_ts || event.ts,
+        });
+
+      } else {
+        // Handle other types of @briefops mentions
+        await say({
+          text: 'I am ready to assist with summarization and document queries! 🚀',
+          thread_ts: event.thread_ts || event.ts,
+        });
+      }
+
+    } catch (error) {
+      console.error(`[ERROR] Error handling @briefops mention: ${error.message}`);
+      await say({
+        text: `[ERROR] An error occurred while processing your request: ${error.message}.`,
+        thread_ts: event.thread_ts || event.ts,
+      });
+    }
+  });
+};

main.js

@@ -7,6 +7,8 @@ const https = require('https');
 const axios = require('axios');
 const validations = require('./validations');
 const summarization = require('./summarization');
+const fsSummary = require('./fs-summary'); // Import the fs-summary.js module
+
 
 // Set a global axios default timeout
 axios.defaults.timeout = 15000; // 15 seconds
@@ -38,6 +40,8 @@ validations.validateSecrets().then(() => {
     console.log('Slack App initialized successfully.');
   } catch (error) {
     console.error('Error initializing Slack App:', error);
+    console.error('Please ensure that all necessary environment variables are set correctly and valid.');
+    console.error('Check SLACK_BOT_TOKEN, SLACK_SIGNING_SECRET, and SLACK_APP_TOKEN in your environment.');
     process.exit(1); // Exit if the initialization fails
   }
 
@@ -62,6 +66,7 @@ validations.validateSecrets().then(() => {
 
   socketModeClient.on('error', (error) => {
     console.error('Socket Mode connection error:', error);
+    console.error('Check your network connectivity and ensure that the SLACK_APP_TOKEN is correct.');
   });
 
   validations.retrySocketConnection(socketModeClient); // Add retry logic for socket connection
@@ -70,37 +75,65 @@ validations.validateSecrets().then(() => {
   console.log('Initializing modules...');
   try {
     summarization(app);
+    fsSummary(app);  // Initialize the fs-summary module
+
     console.log('Modules loaded successfully.');
   } catch (error) {
     console.error('Error loading modules:', error);
+    console.error('Please check if all required modules are properly configured and available.');
     process.exit(1); // Exit if modules fail to load
   }
 
-  // Add slash command listener for /briefops
-  app.command('/briefops', async ({ command, ack, respond }) => {
-    console.log(`Received /briefops command from user: ${command.user_name}, text: ${command.text}`);
-    try {
-      console.log('Acknowledging /briefops command...');
-      await ack(); // Acknowledge the command promptly
-      console.log('Acknowledged /briefops command successfully.');
-
-      // Add a small delay to simulate processing time and ensure acknowledgement is separated
-      setTimeout(async () => {
-        try {
-          console.log('Responding to /briefops command...');
-          await respond({ text: `Processing your request: ${command.text}` });
-          console.log('/briefops command processed and responded successfully.');
-        } catch (responseError) {
-          console.error('Error responding to /briefops command:', responseError);
-          await respond({ text: 'An error occurred while processing your request.' });
-        }
-      }, 10); // Delay of 10 milliseconds to avoid blocking ack()
+  // Adjusted fetchChannelMessages to handle 'not_in_channel' error
+async function fetchChannelMessages(channelId) {
+  try {
+    const result = await app.client.conversations.history({
+      token: process.env.SLACK_BOT_TOKEN,
+      channel: channelId,
+    });
 
-    } catch (error) {
-      console.error('Error handling /briefops command:', error);
-      await respond({ text: 'An error occurred while processing your request.' });
+    return result.messages;
+  } catch (error) {
+    console.error('Error fetching channel messages:', error);
+
+    // Check for the specific 'not_in_channel' error
+    if (error.data?.error === 'not_in_channel') {
+      throw new Error(
+        `It looks like the bot is not in the channel. Please invite the bot to the channel by typing: \`/invite @briefops\`.`
+      );
     }
-  });
+
+    // Handle other errors as a general fallback
+    throw new Error('Failed to fetch messages due to an unexpected error.');
+  }
+}
+
+// Slash command listener with enhanced error message
+app.command('/briefops', async ({ command, ack, respond }) => {
+  console.log(`Received /briefops command from user: ${command.user_name}, text: ${command.text}`);
+  try {
+    await ack(); // Acknowledge the command
+
+    console.log('Fetching channel messages...');
+    const messages = await fetchChannelMessages(command.channel_id); // Assuming you're using the channel_id from the command
+
+    console.log('Messages fetched successfully.');
+    await respond({ text: `Messages fetched successfully. Processing your request: ${command.text}` });
+
+    // Proceed with summarization or other logic
+    // ...
+
+  } catch (error) {
+    console.error('Error handling /briefops command:', error.message);
+
+    // Respond to the user with the improved error message
+    await respond({ text: error.message });
+  }
+});
+
+app.command('/briefops-status', async ({ command, ack, respond }) => {
+  await validations.handleStatusCommand({ command, ack, respond });
+});
 
   // Start the app
   (async () => {
@@ -117,7 +150,7 @@ validations.validateSecrets().then(() => {
 
       server.on('error', (err) => {
         if (err.code === 'EADDRINUSE') {
-          console.error(`Port ${port} is already in use.`);
+          console.error(`Port ${port} is already in use. Please choose a different port or stop the conflicting service.`);
           process.exit(1);
         } else {
           console.error('Server error:', err);
@@ -134,6 +167,7 @@ validations.validateSecrets().then(() => {
       console.log(`⚡️ BriefOps app is running on port ${port}!`);
     } catch (error) {
       console.error(`Error starting the app on port ${port}:`, error);
+      console.error('Please check if the port is available and that all necessary environment variables are set.');
       process.exit(1);
     }
   })();
@@ -143,6 +177,7 @@ validations.validateSecrets().then(() => {
   validations.testSlackApi();
 }).catch((error) => {
   console.error('Failed to validate secrets or initialize the app:', error);
+  console.error('Please ensure all secrets are correctly configured in Google Secret Manager or environment variables.');
   process.exit(1);
 });