[Roadmap] Code Execution and User-Defined Functions

[gagb]

Code Execution

We want users to choose different code execution backends: command lines, iPython kernel, etc., and plug-in their own customized code executor. We also want to provide built-in containerized code executors.

Tasks

Preview Give feedback

1. Finish and merge Code executors #1405
2. Documentation update for code executor: notebooks.
3. Jupyter Code Executor
4. [Feature Request]: automatically quiet pip install in code executors #2075
   code-execution feature good first issue +3
   
5. [Feature Request]: partial error message from code executor #2060
   0.2 code-execution needs-triage +3

User-Defined Functions

We currently only supports function calling via OpenAI function call and tools. We plan to overcome this limitation by supporting user-defined functions in the new code executors. So these functions can be used in code blocks generated by code producer agents and executed by code executors.

Use Cases

• Will allow function calling with non-OpenAI models that do not support function calling
• Will allow agents to compose functions (by utilizing composability of language/free-form code)
• Will allow functions to manage dependencies and secrets

Tasks

Preview Give feedback

1. Check and close [Feature Request]: Using registered functions in code execution, useful for non-openai models. #1336
2. Make sure to look at Custom Client to ensure compatibility. Custom Model Client support #1345
3. Add initial implementation of skill_map #1409
   +0
4. Add a command line option to remove docker image onces completed: the agents should need to delete the docker they used #1399
5. Voyager-Style Skill Library for Agents #98
   help wanted research teach +3
6. Unify tool functions and user defined functions #2101
   code-execution tool-usage +2

[gagb]

Test comment for AutoAnny.

[AaronWard]

hey @gagb - let me know how i can help with this in any way. Been waiting for function calling on local models for a while so excited to see this is being implemented.

[gagb]

hey @gagb - let me know how i can help with this in any way. Been waiting for function calling on local models for a while so excited to see this is being implemented.

@AaronWard your help is much appreciated! Can you give @ekzhu feedback on #1405. It's a big change but we believe will make it much cleaner and easier to add this new style of function calling. Once #1405 is done we will port proposed changes in #1409 to use it.

We would like 1405 analyzed and merged soon. I added you as a reviewer.

[AaronWard]

@gagb Sounds good, i will take a look at it today after work.

[ekzhu]

Just add @BeibinLi 's comment in #1405

handle different types of system_message (not only str)
handle DeprecationWarning throughout AutoGen
adding more detailed tutorials for Capabilities
adding a higher level within AssistantAgent and UserProxyAgent to wrap complex (but common) executors.

[ekzhu]

Add @IANTHEREAL's comments for additional work on code executors:

1. Fix the usage of _ private methods in group chat managers
2. End-to-End tests with conversable agents and code executors for scenarios like plotting a stock chart.
3. Documentation and Notebooks
4. Design high-level binding methods to make sure compatibility between code producer and code executor agents.

[jackgerrits]

@ekzhu we should consider making it possible for a user to cancel an in progress execution (see here: #1626)

[rizickus]

In one of the linked issues, I addressed my concerns and confusion.

Are there any plans to unify skills and tools? In some sense, I prefer tools because you pass only very limited API information. With skills looks like in general you have to use more tokens to enable your agent to achieve the same thing (you provide the full code).

To sum up, the studio uses skills, but tools look "leaner" approach, so I am having paralysis on which way to move forward at this moment.

[rysweet]

consider migrating the roadmap issues to use milestones