Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Friend V2] Connect Friend v2's onboard memory to the Omi app ($2,000) #752

Closed
kodjima33 opened this issue Sep 5, 2024 · 8 comments
Closed

[Friend V2] Connect Friend v2's onboard memory to the Omi app ($2,000) #752

kodjima33 opened this issue Sep 5, 2024 · 8 comments
Assignees
@kevvz
Labels
firmware firmware work flutter flutter work help wanted Extra attention is needed Paid Bounty 💰 python

Comments

@kodjima33
Copy link
Collaborator

kodjima33 commented Sep 5, 2024 •
edited by josancamon19
Loading

Is your feature request related to a problem? Please describe.
Recently we've added onboard memory support for Friend dev kit 2 but it is not integrated with the Omi app

Describe the solution you'd like
You need to integrate onboard memory support with Omi app. You need to do that by creating a separate listener in the mobile app that will listen for when the device reconnects and checks and if there is any audio in onboard storage. If yes, the listener needs to:

  • receive the audio file
  • process it with a separate backend endpoint (you need to create the endpoint)
  • cut the audio file in multiple memories, if there are long waits
  • save those memories in the backend (processing each memory independently)
  • Ideally this endpoint streams to the backend, instead of waiting for the opus encoded data to arrive, as it can take a few hours, so this functionality will be mostly made by users while sleeping, letting their Friend sync.

Additional context
This is a paid task. Reward is $2000 in cash. Simply link your PR with this task and we will check it asap. Check how Contributions and bounties work here

  • If no one is assigned to this task, then no one is working on it.
  • We don't lock bounty tasks on you unless you've done PRs before. But you can solve that by doing this PR first
  • if you need the v2 device, build it yourself or come to our San Francisco office and we will give you one. DM @kodjima33 in telegram for details. If you can't do neither, buy device here and dm on the same telegram
@kodjima33 kodjima33 added firmware firmware work python flutter flutter work Paid Bounty 💰 help wanted Extra attention is needed labels Sep 5, 2024
@kodjima33 kodjima33 changed the title [Friend V2] Connect Friend v2 (custom PCB) to the Omi app ($2,000) [Friend V2] Connect Friend v2 to the Omi app ($2,000) Sep 5, 2024
@kodjima33 kodjima33 changed the title [Friend V2] Connect Friend v2 to the Omi app ($2,000) [Friend V2] Connect Friend v2's onboard memory to the Omi app ($2,000) Sep 5, 2024
@kevvz
Copy link
Collaborator

kevvz commented Sep 5, 2024

I will take this. I built the memory so I think I know how to integrate this best

@kodjima33
Copy link
Collaborator Author

@kevvz pls try but since you told me multiple times that you don't feel comfortable with flutter, if anyone else wants to try, they can still try to integrate it as well (We need to start shipping v2 asap)

@kevvz
Copy link
Collaborator

kevvz commented Sep 5, 2024

Yes, anyone else who wants to take on this task, feel free to do so!

@kevvz kevvz mentioned this issue Sep 8, 2024
Closed
@eltoob
Copy link

eltoob commented Sep 9, 2024

Question about the SD Card @kodjima33, don't you think that BLE bandwidth is going to be a problem. BLE has a max speed of 1Mbps.

@kevvz kevvz mentioned this issue Sep 10, 2024
Merged
josancamon19 added a commit that referenced this issue Sep 11, 2024
@josancamon19
Sdcard Support for App and Firmware Updates #752 (#810)
c64fd18 
#752
@kodjima33
Copy link
Collaborator Author

@eltoob no will not be a problem because we don't have another choice

@kevvz kevvz mentioned this issue Sep 15, 2024
Merged
josancamon19 added a commit that referenced this issue Sep 18, 2024
@josancamon19
added sd card processing pipeline #752 (#844)
56a3a1e 
#752 

This PR integrates the SDcard reading logic into the app, including
backend and flutter. If you properly configure the app and device, then
you can create memories from the SD card, even if you weren't connected
to the app while doing so.
@josancamon19
Copy link
Contributor

josancamon19 commented Sep 18, 2024 •
edited by kevvz
Loading

  • Once I open the app I want to get an alert in the top of the app, that is dismissable, but says, you have "X hours" of recording available." I also want this static alert below speech language in settings.

  • That should open a new page

    • This page should contain a single buton that says start importing memories
    • It should have a progress bar and an estimate left,
    • Somewhere here it should explain, that the rest of the app "will not work" realtime while they import this is happening.
    • Should explain they should have good internet connectivity while it happens.
    • The device storage bytes -> should be streamed to the backend, and the backend should stream the storage into Google cloud temporarilly (there's some existing code for this, reuse), there should be a timer on the backend, or a background tasks that if the "session" or "file id" stopped being streamed for longer than 10 seconds, it should be processed then
    • So if the connection cuts in the middle, it should still process whatever was transmitted, and the device shouldn't have those bytes anymore.
    • When processing on the backend
      • Use VAD endpoint to find cut's of audio in the conversation, separate as single files, every time the splits without voice are longer than 2 minutes
      • Process each file independetly as a single memory, and use fal_whisperx for the stt
      • Save a value in the memory object, that tell us that that memory was created from storage caching.

  • Once the file finishes streaming, it should let the user know that the file was received, and the processing will take a few minutes, check their notifications are on.

  • Once the processing of all memories are received, it should send a notification to the user letting them know that the memories where processed ~ The app should refresh this case the memories list, as if the app is not reopened the memories will not be seen.

@josancamon19
Copy link
Contributor

I might be missing a couple UX possible issues, or edge cases.

This was referenced Sep 24, 2024
Closed
Merged
josancamon19 added a commit that referenced this issue Sep 30, 2024
@josancamon19
#752 SD Card Streaming (#933)
2ab47c4 
#752 
Rebase of earlier PR

On app open, if there is a nonzero amount of bytes remaining on your
device, then you will Get a green popup saying that you have some amount
of seconds of audio remaining to download.
Upon pressing the green tab, you will be taken to the sd card page,
where there is a button that says 'click to download audio/memories'. It
has a green progress bar and a warning stating that you should have good
internet connection if yo want to download.
If you press the button, you will notice the green bar filling up and
the percentage remaining increasing. This means that the storage bytes
are being streamed successfully. If for any reason your internet
disconnects, then the bar will stop progressing and you will need to
repress the button to continue. The app will save how many bytes you
have already streamed for in case you want to continue downloading.
If the backend doesn't receive data after the first packet for at least
10 seconds, the resulting audio file will be parsed. It uses VAD, then
fat whisper to partition the audio file and processes them separately
into memories (it is also flagged as storage. Each separate file is also
saved in some backend.
Assuming a valid memory is created, the user will get a notification
upon the backend finishing the parsing. If the user presses it, then the
memory will automatically appear. It also works if you just go to the
memory screen.
@beastoin
Copy link
Collaborator

should we...?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kevvz kevvz

Labels
firmware firmware work flutter flutter work help wanted Extra attention is needed Paid Bounty 💰 python
Projects
omi TODO
Status: Done
Milestone
No milestone
Development

No branches or pull requests
5 participants
@eltoob @beastoin @josancamon19 @kodjima33 @kevvz