You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There exists a need to design some way of attaching arbitrary JSON objects to messages.
For Jupyter AI v3, we intend to provide @ commands that allow users to include files & variables from the notebook for context. It would be ideal to retrieve the data of these files/variables directly from the frontend and somehow attach them to the next input for the current user. Defining command handling in the frontend is the most flexible approach, as it allows access to every API provided by Jupyter, not just Python APIs. See this comment for details on the thought process here.
Attachments can also later be used to add support for multimodal models which accept both text & images, and may return text & images.
Proposed Solution
Modify the structure of Message to allow a list of attachments, whose contents are arbitrary and JSON-serializable.
Design some sort of Attachments API in the frontend.
Add a UI to let users attach files manually. This should call the Attachments API.
Show attachments as a list of toasts on top of the chat input, and allow users to remove attachments. I've included an example screenshot from GitHub Copilot to use as inspiration.
I'm not sure how the Attachments API should be accessed. I think it would be good to make it accessible from the YChat object in the frontend. This way, the YChat interface provides all of the APIs needed by chat commands.
Additional context
I'm currently quite busy with iterating on my existing designs & planning collaborations with new contributors. @brichet I'm assigning this to you as you have the most context on Jupyter Chat, and can help propose better ideas. Feel free to implement a proof-of-concept in a separate PR.
The text was updated successfully, but these errors were encountered:
Problem
There exists a need to design some way of attaching arbitrary JSON objects to messages.
For Jupyter AI v3, we intend to provide
@
commands that allow users to include files & variables from the notebook for context. It would be ideal to retrieve the data of these files/variables directly from the frontend and somehow attach them to the next input for the current user. Defining command handling in the frontend is the most flexible approach, as it allows access to every API provided by Jupyter, not just Python APIs. See this comment for details on the thought process here.Attachments can also later be used to add support for multimodal models which accept both text & images, and may return text & images.
Proposed Solution
Message
to allow a list of attachments, whose contents are arbitrary and JSON-serializable.I'm not sure how the Attachments API should be accessed. I think it would be good to make it accessible from the
YChat
object in the frontend. This way, theYChat
interface provides all of the APIs needed by chat commands.Additional context
I'm currently quite busy with iterating on my existing designs & planning collaborations with new contributors. @brichet I'm assigning this to you as you have the most context on Jupyter Chat, and can help propose better ideas. Feel free to implement a proof-of-concept in a separate PR.
The text was updated successfully, but these errors were encountered: