Feature: Continued conversation with text on screen for ESP32-S3-Box-3

[jaymunro]

This is in draft PR #173 pending more testing.

Here is a demo video of this beta of Voice assistant with Continued conversation and text.

Demo of Continued conversation BETA

I'm looking for testers who can give feedback on issues, further suggestions for improvement, etc. I've tested it with the HA streaming wake word but it works much better with the on-device wake word. Once it is verified, we can hopefully get it accepted in the ESPHome repository for everybody. Please leave issues and suggestions here.

To install, in your device's yaml change this:

packages:
  esphome.voice-assistant: github://esphome/firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml
  ...

to this:

packages:
  # esphome.voice-assistant: github://esphome/firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml
  esphome.voice-assistant: github://jaymunro/esphome_firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml@continued-conversation
  ...

If you change the wake word model, add another substitution for the wake word text as follows:

substitutions:
  ...
  micro_wake_word_model: hey_jarvis
  wake_word_text: "Hey Jarvis"

Note this is a beta and the repo above is temporary until after the PR is accepted.

Before doing an Install, be sure to do a Clean build files from the 3-dot hamburger menu in ESPHome.

[jaymunro]

Comments in https://community.home-assistant.io/t/esp32-s3-box3/638287/280?u=hjm so far confirm operation and request the following:

The display text is working pretty good - except localization (it’s in EN) - is there any chance to get it working in system default language (mine is Slovenian for example)?

Wake word: from time to time the I get in the log:
09:31:48][D][voice_assistant:349]: Speaker buffer full, trying again next loop

the text doesn’t support some char, I’m french and turn off is “Éteins” It show a black rectangle instead of the É. would it be possible to have change the format of the date ? and time ?

...but in fact I need just: ščž ŠČŽ. As for date and time: "%d.%m.%y"

I would vote for a 24-hour clock (no AM/PM). And I would like the Danish letters æ ø å Æ Ø Å to be included

you can dynamically change the pipeline and so the wake word when not using the in device, that’s why the text can’t be good, or we could have a text_sensor to change it (I would love that in fact)

latin like french and spanish would need :
é É ç Ç è È ê Ê à À ô
there is more but not sure if we need them maybe we should find a way to have all of them, don’t know about the correct way to do that

[jaymunro]

I have now modified the glyphs to add Slovenian, Danish, French, Spanish, German

    glyphs:
      '&@!"''%()+=,-_.:°/?0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyzščžŠČŽæøåÆØÅéÉçÇèÈêÊàÀôäüöÄÜÖß'

[pepe59]

It could also be added for Czech.
Thank you

glyphs: ' !"%()+,-.:/0123456789AÁBCČDĎEÉĚFGHIÍJKLMNOÓPQRŘSŠTŤUÚVWXYZŽ_abcdefghijklmnopqrstuvwxyzěščřžýáíéťúů°'

[Bluhme1]

Would like to have an option to shift to 24-hour clock as we use in Continental Europe

Danish characters work beautifully

/Lars

[jaymunro]

I have found this link https://www.underware.nl/latin_plus/character_set/
Hopefully that will cover all needs without needing too many resources from the Box3.

I have in testing a version with 'date format' and 'time format' dropdown menus to replace the single dropdown presently in use and hope to put this up later today.

[jaymunro]

New version is up with many many more glyphs, fix for stuck condition at boot, and methods for selecting date and/or time formats.

Next is methods to allow for different languages in the text prompts.

[jaymunro]

Method has now been added for multilingual text. Reinstall the same package and see #173 (comment) on how to use.

[bobzer]

Thanks for the new format, it's great, I would love to get a few more format without year, like 5th march and tuesday 5th march.
My current config is :

substitutions:
  name: "esp32-s3-box-3"
  friendly_name: S3 Box 3
  starting_up: "Démarrage..."
  wake_word_prompt: "Dis \\\"Okay Nabu\\\""
  listening_prompt: "Que puis-je faire ?"
  error_prompt: "Oups, Désolé"

packages:
  # esphome.voice-assistant: github://esphome/firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml@main
  esphome.voice-assistant: github://jaymunro/esphome_firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml@continued-conversation
  esphome.voice-assistant-sensor: github://jaymunro/esphome_firmware/esp32-box-docks/box3-sensor-for-voice-assistant.yaml@sensor_dock

esphome:
  name: ${name}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}

api:
  encryption:
    key: *****************

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

but the text is still in english

[jaymunro]

When I do a Validate in ESPHome I see that the package being used is old and does not contain the most recent changes to the yaml in github. I suspect something is caching the package somewhere. I've tried restarting esphome, restarting core, but to no avail.

• I did a ful l reboot of HA OS with no effect. Old code is still used.
• Opening the github branch in a private browser shows the new code.
• Doing a git checkout continued-conversation on my local machine shows the new code.

Not sure what is going on here.

[bobzer]

I did a validate and then an install and it finally worked

[bobzer]

I got a strange black caractere when I ask him some sentences:
So I just ask him : "Comment tu vas ?" <= copy paste from assist query field
and strangely i got a space as a black rectangle (reproduce each time i tried)

looking at the photo, I just realize that the time is not shown.

[jaymunro]

@bobzer The black character would indicate something missing from the glyphs. Perhaps it is a special kind of space? The github editor however has changed to an ordinary space (ascii 0x0020). I copied your text into https://r12a.github.io/uniview/ and got this:
 ‎0043 LATIN CAPITAL LETTER C
 ‎006F LATIN SMALL LETTER O
 ‎006D LATIN SMALL LETTER M
 ‎006D LATIN SMALL LETTER M
 ‎0065 LATIN SMALL LETTER E
 ‎006E LATIN SMALL LETTER N
 ‎0074 LATIN SMALL LETTER T
 ‎0020 SPACE
 ‎0074 LATIN SMALL LETTER T
 ‎0075 LATIN SMALL LETTER U
 ‎0020 SPACE
 ‎0076 LATIN SMALL LETTER V
 ‎0061 LATIN SMALL LETTER A
 ‎0073 LATIN SMALL LETTER S
 ‎0020 SPACE
 ‎003F QUESTION MARK

The 2nd to last char is 20 hex, an ordinary space.

Can you try pasting your output directly into that website (or any you like that would show unicode, not just ascii) and see what you get?

For the missing time, try changing the format option to something else and back again to see if it returns. Sometimes if you do a reboot before it has saved it's cached variables to flash (about once every minute or two, it's in the logs), you get the old value back again, which may have been 'None', the default.

[jaymunro]

Not sure what is going on here.

Found this: https://community.home-assistant.io/t/esphome-addon-caches-yaml-depndency-from-github/657415/6?u=hjm

By default, single line package imports/adoptions are only re-downloaded once every 24 hours

The adopted single-line package can be replaced with a full schema package with custom refresh period
See Configuration Types — ESPHome 3

Build files and cached templates are stored in the addon’s folders in docker container.
Those files can be accessed if connected directly via ssh in /mnt/data/supervisor/addons/data/5c53de3b_esphome

So if my understanding is correct, to get new commits immediately and to not have to wait 24 hours...

In our device's yaml, we should replace this:

  esphome.voice-assistant: github://jaymunro/esphome_firmware/wake-word-voice-assistant/esp32-s3-box-3.yaml@continued-conversation

with this:

  remote_package:
    url: https://github.com/jaymunro/esphome_firmware
    ref: continued-conversation
    files: wake-word-voice-assistant/esp32-s3-box-3.yaml
    refresh: 0d

Note, this is only until the pr is accepted. After that, the url will change anyhow.

Edit: I can thank the new Claude for pointing me in the right direction. https://claude.ai/chat

[jaymunro]

#173 is ready and awaiting for review now.

[jaymunro]

I would love to get a few more format without year, like 5th march and tuesday 5th march

@bobzer, noted and once we have a stable version merged then maybe we can tweak it some more in a way that still keeps the interface simple.