Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Design a way to attach JSON objects to messages #147

Open
dlqqq opened this issue Jan 8, 2025 · 0 comments
Open

Design a way to attach JSON objects to messages #147

dlqqq opened this issue Jan 8, 2025 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@dlqqq
Copy link
Member

dlqqq commented Jan 8, 2025

Problem

There exists a need to design some way of attaching arbitrary JSON objects to messages.

For Jupyter AI v3, we intend to provide @ commands that allow users to include files & variables from the notebook for context. It would be ideal to retrieve the data of these files/variables directly from the frontend and somehow attach them to the next input for the current user. Defining command handling in the frontend is the most flexible approach, as it allows access to every API provided by Jupyter, not just Python APIs. See this comment for details on the thought process here.

Attachments can also later be used to add support for multimodal models which accept both text & images, and may return text & images.

Proposed Solution

  1. Modify the structure of Message to allow a list of attachments, whose contents are arbitrary and JSON-serializable.
  2. Design some sort of Attachments API in the frontend.
  3. Add a UI to let users attach files manually. This should call the Attachments API.
  4. Show attachments as a list of toasts on top of the chat input, and allow users to remove attachments. I've included an example screenshot from GitHub Copilot to use as inspiration.
Screenshot 2025-01-07 at 5 22 06 PM

I'm not sure how the Attachments API should be accessed. I think it would be good to make it accessible from the YChat object in the frontend. This way, the YChat interface provides all of the APIs needed by chat commands.

Additional context

I'm currently quite busy with iterating on my existing designs & planning collaborations with new contributors. @brichet I'm assigning this to you as you have the most context on Jupyter Chat, and can help propose better ideas. Feel free to implement a proof-of-concept in a separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants