Create a vector search from youtube audio transcripts

README.md

@@ -1,156 +1,37 @@
-# AI Toolchain
-AI Toolchain is a collection of tools for quickly building and deploying machine learning models for various use cases. Currently, the toolchain includes a text translation model, and more models may be added in the future.
-
-## Prerequisites
-
-- [Docker](https://docs.docker.com/)
-- [NginX](https://www.nginx.com/resources/wiki/start/)
-
-## Setup
-
-AI-Tools Models at present can be packaged and deployed in two ways:
-- As a Single Image
-- Multiple Images
-
-### Single Image
-In single image setup, we have a single quart app hosting all the models. It is bundled as a single Docker Image. All models can be accessed following this path convention 
-`http://domain_name/<use_case>/<provider>/<mode>`
-
-Services are usually provided in either **remote** mode or **local** mode.  
-
-For example, the text_translation service provided by google can be accessed on your local system at this URL:
-`http://localhost:8000/text_translation/google/remote`
-
-### Deploy Single Image Setup
-```
-docker build -t <image-name> .
-docker run -p <host_port>:<container_port> <image-name>
-```
-
-### Multiple Image
-In Multi Image Setup, each model has its own quart application and docker image. All model APIs are served via NGINX by utilizing proxies to route incoming requests to the appropriate destinations.  
-
-There won't be any difference in accessing the models. The text_translation service provided by google can still be accessed on your local system at this endpoint:
-`http://domain_name/text_translation/google/remote`
-
-### Deploy Multi Image Setup
-For mutliple image setup, the docker-compose file and nginx configuration files are generated dynamically based on the config.json file. The following commands might require sudo privileges.   
-
-Run this script to generate the configuration files. 
-```
-./generate.sh
-```
-Setup NginX configuration
-```
-sudo cp <DOMAIN_NAME>.conf /etc/nginx/conf.d/
-sudo nginx -t //Checks for configuration errors
-sudo nginx -s reload
-```
-- For **Windows** copy to `C:\nginx\conf\` and reload the server.
-- For **MacOS** copy to `/usr/local/etc/nginx/` and reload the server.
-
-The .conf file will be present in your project root path after running the `generate.sh` script.  
-
-Build Image
-```
-sudo docker-compose -f ./docker-compose-generated.yaml build
-sudo docker-compose -f docker-compose-generated.yaml up
-```
-
-## Adding a new model locally
-To add a new model you need to create sub-directory in `src`. The subdirectory will follow `use_case/provider/mode` format.
-
-To deploy your application, the dockerfile must:
-- copy program files
-- install requirements
-- build and run the application
-
-After you have developed and dockerized the application, to add a new model, you need to add a key pair inside models object in the config file with the following properties:
-
-- serviceName : Name of the service. This will also be the docker image's name. 
-- modelBasePath: Dockerfile location of your model.
-- apiBasePath: Endpoint from where you wish to access the model. Should follow this convention: `<use_case>/<provider>/<mode>`.
-- containerPort: Port at which the application will be hosted inside the container.
-- environment: Another JSON object with key-value pairs. Should contain any relevant secrets or API_KEYS required to run the application.
-
-For example, if you want to add a new Text Translation AI Model from OPENAI in remote mode, you need to do as follows:
-```
-{
-      "serviceName": "text_translation_google",
-      "modelBasePath": "src/text_translation/google/remote/.",
-      "apiBasePath": "text_translation/google/remote",
-      "containerPort": 8000,
-      "environment": {
-        "OPENAI_API_KEY": "${OPENAI_API_KEY}",
-        "GOOGLE_CLOUD_SECRET": "${GCP_SECRET}"
-      }
-}
-```
-## Run Ansible Script
-
-### Requirements
-- [HashiCorp Vault](https://www.hashicorp.com/products/vault)
-- [Docker Swarm](https://docs.docker.com/engine/swarm/)
-- Github Personal Access Token
-
-### Steps
-
-1. Create a `inventory.ini` file with target machine configuration. Target Machine is where the ansible script will run.  
-Here is a sample `inventory.ini` file:  
-```
-[swarm_manager]
-ee01:ac8:1561::2 ansible_connection=ssh ansible_user=github ansible_ssh_private_key_file=/home/github/.ssh/id_rsa
-```  
-For More Information Refer https://www.digitalocean.com/community/tutorials/how-to-set-up-ansible-inventories and https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html
-You may modify it according to your needs.  
-2. Create a Hashicorp Vault to store your secrets. You need to store the following types of secrets:(In separate Secret Paths mentioned below. This is not optional),
-- Github Credentials. (Path = `secret/github`)
-  - `USERNAME` (Github Username)
-  - `PAT` (Github Personal Access Token. It should have read and write access to the contents of the repository)
-- Target Machine Credentials (Path = `secret/common`)
- - `DDIR` (Destination Directory, place where the files will be downloaded and generated)
- - `REPONAME` (Github Repository Name. This could be Samagra-Development/ai-tools or the name of your fork.)  
- Please note: You don't have to add the entire Github URL, only the Repo Name. for ex: If you fork the ai-tools repo to your account. The repo URL would be something like `<your_username>/ai-tools`.  
- This setup will only work **if** you have a restructure branch in your fork/repo with `generate.sh` and `config.json` in your root folder. Please be aware that modifying the file structure will break the code.
-- Environement Secrets for the Docker Images (Path  = `secret/config`)  
-  - `DOCKER_REGISTRY_URL` (ghcr.io if you are pulling from Github Pacakges. Can be Docker Hub as well)
-  - `GITHUB_REPOSITORY` (ai-tools unless told otherwise)
-  - `OPENAI_API_KEY`, and other secrets pertaining to the docker images. This list might exapand as the project grows and supports multiple models. Please take a look at the `sample.env` for an exhaustive list of other env keys that are required by the models.
-
-  Refer to the faq to see how to add secrets to hashicorp
-3. Set the Vault Credentials in the environment of the target machine so that it can access the remotely hosted Hashicorp Vault.
-  - `VAULT_ADDR`: (Vault Address)
-  - `ANSIBLE_HASHI_VAULT_TOKEN`: (Vault Root Login Token)  
-You can add environment variables as follows:
-```
-export VAULT_ADDR=http://x.x.x.x:8200
-export ANSIBLE_HASHI_VAULT_TOKEN=abc12345
-```
-For the Vault Credentials to persist add it to `.bashrc`
-```
-echo 'export VAULT_ADDR=http://x.x.x.x:8200' >> ~/.bashrc
-echo 'export 'ANSIBLE_HASHI_VAULT_TOKEN=35r3zxcc' >> ~/.bashrc
-source ~/.bashrc
-```
-
-Alternatively, you can pass the variables during run time as Command Line Arguments using `--extra-vars` field. 
-Here is an example:   
-```
-ansible-playbook -i inventory.ini swarm.yml --extra-vars "VAULT_ADDR='http://127.0.0.1:8200' VAULT_TOKEN=abc.123" --ask-become-pass
-```
-
-### Additional Steps
-1. Change Hosts in Ansible Playbook to the Node which is acting as the swarm manager. (Make sure a Docker Swarm is running in that machine)
-
-### To run the Ansible Playbook
-```
-ansible-playbook swarm.yml -i inventory.ini
-```
-## Contributing
-Contributions to AI Toolchain are welcome! To contribute, please follow these guidelines:
-
-1. Fork the repository and create a new branch for your feature or bug fix.
-2. Write tests for your changes.
-3. Submit a pull request describing your changes and why they are needed.
-
-Thank you for considering contributing to AI Toolchain!
+# Youtube_transcripts
+
+# Description
+Possess the ability to parse every video on a YouTube channel or playlist, extract audio transcripts, and embed the videos in a vector database so that search and retrieval are possible.
+
+# Implementation Details
+It will consist of the following:
+
+* Get the user's playlist or channel link.
+* Take audio clips out of every video in the playlist or link.
+* Take out the timestamps and transcripts from each video.
+* Make chunks from the transcript (you may use any fancy chunking algorithm or just use basic chunks like four minutes of audio).
+* Use an LLM call to summarise each video, then save each chunk separately.
+* Use COLBERT to embed this in a vector database (Ragatoullie - LangChain). Make use of this as a guide.
+* Turn on COLBERT retrieval and search for the embedded information.
+* A search for a question returns obtain the necessary content's timestamps, a YouTube link, and similar content.
+
+# Product Name
+AI Tools
+
+# Organization Name
+SamagraX
+
+# Domain
+NA
+
+# Tech Skills Needed
+Pytorch/ Python, ML
+
+# Category
+Feature
+
+# Mentor(s)
+@GautamR-Samagra
+
+# Complexity
+Medium

final_03.py

@@ -0,0 +1,159 @@
+import os
+from pydub import AudioSegment
+from pytube import Playlist
+from openai import OpenAI
+import sqlite3
+from datetime import datetime
+
+# Function to download audio from the playlist URL
+def download_audio_from_playlist(playlist_url, download_directory):
+    # Create a Playlist object
+    playlist = Playlist(playlist_url)
+
+    # Create the directory if it doesn't exist
+    if not os.path.exists(download_directory):
+        os.makedirs(download_directory)
+
+    # Iterate through each video in the playlist
+    for video in playlist.videos:
+        try:
+            # Get the highest resolution audio stream
+            audio_stream = video.streams.filter(only_audio=True, file_extension='wav').first()
+
+            # Download the audio stream
+            audio_stream.download(output_path=download_directory)
+
+            print(f"Audio downloaded for video: {video.title}")
+
+        except Exception as e:
+            print(f"Error downloading audio for video {video.title}: {str(e)}")
+
+    print("Audio download complete.")
+
+# Function to slice audio into 4-minute chunks
+def slice_audio(input_dir, file_name, output_dir, target_length):
+    song = AudioSegment.from_wav("%s/%s.wav" % (input_dir, file_name))
+    timestamps = []
+    chunks = []
+
+    i = 0
+    while i < len(song)/(float(target_length)*1000):
+        seconds = float(target_length) * 1000 * i
+        seconds2 = float(target_length) * 1000 * (i+1)
+        cut = song[seconds:seconds2]
+
+        # Store timestamp and chunk name
+        timestamp = datetime.now()
+        chunk_name = f"{file_name}-{float(target_length)}sec-{i}.wav"
+        cut.export(f"{output_dir}/{chunk_name}", format="wav")
+
+        timestamps.append(timestamp)
+        chunks.append(chunk_name)
+        i += 1
+
+    print("Audio slicing complete.")
+    return timestamps, chunks
+
+# Function to perform speech-to-text using Whisper model
+def transcribe_audio_with_whisper(audio_directory):
+    client = OpenAI(api_key='your_openai_api_key')  # Replace with your OpenAI API key
+
+    transcripts = []
+
+    # Iterate through each audio file in the directory
+    for audio_file_name in os.listdir(audio_directory):
+        if audio_file_name.endswith('.wav'):
+            audio_file_path = os.path.join(audio_directory, audio_file_name)
+
+            # Open the audio file
+            with open(audio_file_path, "rb") as audio_file:
+                # Perform transcription using Whisper ASR model
+                transcript = client.audio.transcriptions.create(
+                    model="whisper-1",
+                    file=audio_file,
+                    response_format="text"
+                )
+
+                print(f"Transcription for {audio_file_name}:")
+                print(transcript)
+
+                transcripts.append(transcript)
+
+    return transcripts
+
+# Function to perform text summarization using Whisper model
+def text_summarization(transcripts):
+    client = OpenAI(api_key='your_openai_api_key')  # Replace with your OpenAI API key
+
+    summarizations = []
+
+    # Iterate through each transcription
+    for transcript in transcripts:
+        # Extract text from the transcription
+        text = transcript['data'][0]['text']
+
+        # Perform text summarization using Whisper model
+        summary = client.completions.create(
+            model="gpt-3.5-turbo-instruct",
+            prompt=f"Summarize the text: {text}",
+            temperature=0.7,
+            max_tokens=300,
+            top_p=1,
+            frequency_penalty=0,
+            presence_penalty=0
+        )
+
+        if 'choices' in summary and len(summary['choices']) > 0:
+            summarization = summary['choices'][0]['text']
+            print(f"Summarization for the transcription:")
+            print(summarization)
+
+            summarizations.append(summarization)
+
+    return summarizations
+
+def save_to_database(link, chunks, timestamps, summarizations):
+    connection = sqlite3.connect('TextSummarisation.db')  
+    cursor = connection.cursor()
+
+    for i, (chunk, timestamp, summarization) in enumerate(zip(chunks, timestamps, summarizations), start=1):
+        # Assuming you have a table named 'audio_data' with columns 'link', 'chunk_number', 'timestamp', 'summarization'
+        query = "INSERT INTO audio_data (link, chunk_number, timestamp, summarization) VALUES (?, ?, ?, ?)"
+        cursor.execute(query, (link, i, timestamp, summarization))
+
+    connection.commit()
+    connection.close()
+
+def main():
+    global input_dir, target_length, file_name, output_dir, playlist_url, chunks, timestamps
+
+    # Get user input for playlist URL
+    playlist_url = input("Enter the playlist video link: ")
+
+    # Set parameters (you can modify these as needed)
+    input_dir = 'audio_downloads'
+    file_name = 'output'
+    output_dir = 'audio_chunks'
+    target_length = 240  # in seconds
+    chunks = []
+    timestamps = []
+
+    # Download audio from the user-provided playlist URL
+    download_audio_from_playlist(playlist_url, input_dir)
+
+    # Slice the downloaded audio files into 4-minute chunks
+    timestamps, chunks = slice_audio(input_dir, file_name, output_dir, target_length)
+
+    # Perform speech-to-text using Whisper model
+    transcripts = transcribe_audio_with_whisper(output_dir)
+
+    # Perform text summarization using Whisper model
+    summarizations = text_summarization(transcripts)
+
+    save_to_database(playlist_url, chunks, timestamps, summarizations)
+
+    print("Text Summarizations:")
+    print(summarizations)
+
+if __name__ == "__main__":
+    main()