#Research Project on Implementation of Speech Module on Ohmni Telepresence Robot
This documents serves as a complete description to setup and start a speech module on ohmni robot
Maintainer: Sharath Nataraj Email id: [email protected]
- cogrob_chatbot/
- efficient_wakeword/
- flask_ros_demo/
- porcupine_wake_word/
- src/
- CMakeLists.txt
- [To Readme.txt](./src/porcupine_wake_word/To Readme.txt)
- package.xml
- porcupine_wakeword_model/
- rasa_ros/
- tts/
- vosk_ros/
- CMakeLists.txt
- Ohmni_kai.zip
- Our_data.db
- mailer.py
- requirements.txt
- webbhook.py
-
pull this repo into robot by using linux shell follow instructions from : https://docs.ohmnilabs.com/
-
cd speech_module -
pip install -r src/requirements.txt
A google mail id has been created to manage multiple services for the robot
email id: [email protected]
password: Ohmni@123@iaas
This module allows you to choose between two wake word recognition models 1 and 2
- This repo contains pre-trained model in the folder porcupine_wakeword_model/ Extract the contents and save it outside the repository since the model is quite big.
- This model can be directly used to for inference from kai_wake_word.py
- The path to extracted model can be passed while astarting the wake word, Remember the porcupine wake word must be started before launching the other nodes.
- picovoice console: https://console.picovoice.ai/signup
- Login through the given google account
- training and creating access key: https://picovoice.ai/docs/porcupine/
- working with pre-trained models:
rosrun porcupine_wake_word kai_wake_word.py --access_key 7segjTjkamk6oOQb/SLE8OBex4vHMdGxW+Z2cxX1EthTIJ13wNPnqg== --keyword_path path_to_extracted file/Hello-Kai_en_linux_v2_1_0.ppn - Custom trained model:
rosrun porcupine_wake_word kai_wake_word.py --access_key access_key_generated_from the previous steps --keyword_path path_to_extracted file/Hello-Kai_en_linux_v2_1_0.ppn
- some extra requirements,
pip install librosa && pip install ffmpeg - This repo also contains a pre-trained efficent-net model sample_refs/,
- remember to change the directorty path of the pre-trained model in line 12 of effnet_word.py
- it is also possible to train a new wake word using the instructions https://github.com/Ant-Brain/EfficientWord-Net#generating-custom-wakewords
- simply replace json modle inside the sample_refs directory and change the directory path of newly trained model at line 12 of effnet_word.py
- remember to run the wake word node before starting other nodes.
- node can be started by
rosrun efficient_wakeword effnet_word.py rostopic pub -1 /eff_wake_word_status std_msgs/String "data: 'on'"
Next is the speech recognition models
by default this branch contains only vosk as STT model, switch to devel branch to find more models
- Vosk STT is pre-compiled with models and can be started easily with a simle command
rosrun vosk_ros vosk_node.py
- This repository provides two differet NLU models which can be choosen by choosing different launch files in launch/
- login to google cloud platform from the given google credentials
- create a servcice key using the email id given and sownload the json file: https://cloud.google.com/iam/docs/creating-managing-service-account-keys#iam-service-account-keys-create-console
- add this to bot environment shell by
export GOOGLE_APPLICATION_CREDENTIALS='/path_to_doownloaded_file/service_key.json' roslaunch vosk_ros start_dialogflow_service.launch
- navigate to cogrob_chatbot/ and start the rasa server
rasa run -m models --endpoints endpoints.yml --port 5002 --credentials credentials.yml --enable-apiroslaunch vosk_ros start_rasa_service.lauch- this node is currently under development this can be used to get replies on the frontend but some features like apointment scheduling system and controlling IOT systems may not work, refer rasa docs to change intents and rules inside cogrob_chatbot/
- contact [email protected] for dialogflow service key.
- clone this repository onto the robo
- add the service key to evironment using
export GOOGLE_APPLICATION_CREDENTIALS='/path_to_doownloaded_file/service_key.json' catkin_makerosrun rosrun porcupine_wake_word kai_wake_word.py --access_key 7segjTjkamk6oOQb/SLE8OBex4vHMdGxW+Z2cxX1EthTIJ13wNPnqg== --keyword_path path_to_porcupine_model/Hello-Kai_en_linux_v2_1_0.ppn &roslaunch vosk_ros start_dialogflow_service.launch- this starts all the required nodes for the operation
- this configuration can be made from a host computer and not on ohmni robot
- login to google dialogflow with the given credentials at the start of the readme file
- install ngrok: https://ngrok.com/download
- start ngrok service
ngrok http 8000this connects dialogflow with webhook.py application - copy the url generated and paste it in fullfilment section of dialogflow console with a route
/webhook, the url looks like : www.example_url.com/webhook - save the changes
- the console can be used to add changes to intents
- Use the SD card provided with the deliverables, and insert it to Raspberry pi 4 with atleast 2GB of RAM, the SD card is configured to automatically connect to Service lab's WiFi. and boot Home Assistant.
- Check the IP adddress of Raspberry pi and make a note of it
- add the IP address of Raspberry pi in line 20 of webbhook.py and line 17 of mailer.py, mailer.py provides an example schema of how the employee database can be structured, more customization and development can be made to suite the needs.
- start both the applications
python3 webook.pyandpython3 mailer.py - setup of mailer.py is rather simple nacvigate to app passwords in google account setup and create a password with other application. copy and paste the password at line 15 of mailer.py
- This ensures mails can be sent through this python script.