Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not add system prompt part when dynamic system prompt function returns empty value #864

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

giacbrd
Copy link

@giacbrd giacbrd commented Feb 6, 2025

The system_prompt decorator allows to dynamically add parts to the messages. However, a part generated by a decorated function is always added, even if the function output is useless, e.g. an empty string. Sometimes the dynamic behaviour we need is to just increment the system prompt, according to the context.

This PR add this behaviour by:

  • avoiding adding a system prompt part to the messages if a system prompt function returns a value that evaluates to False
  • defining the return type of these functions as str | None, so that it is explicit that these function can optionally return a value

Example of a dynamic system prompt where we add some information only if the retrieve call is successful

@agent.system_prompt
async def get_more_information(ctx: RunContext[MyDeps]) -> str | None:
    response = await ctx.deps.http_client.get(
        f'https://example.com/{ctx.deps.record_id}',
        headers={'Authorization': f'Bearer {ctx.deps.api_key}'},
    )
    if response.status_code == 200 and response.text.strip():
        return f'You also have this information: {response.text}'
    return None

@github-actions github-actions bot temporarily deployed to deploy-preview February 6, 2025 17:07 Inactive
@github-actions github-actions bot temporarily deployed to deploy-preview February 6, 2025 19:52 Inactive
@github-actions github-actions bot temporarily deployed to deploy-preview February 6, 2025 20:36 Inactive
@github-actions github-actions bot temporarily deployed to deploy-preview February 6, 2025 20:52 Inactive
@github-actions github-actions bot temporarily deployed to deploy-preview February 6, 2025 21:10 Inactive
@giacbrd
Copy link
Author

giacbrd commented Feb 6, 2025

The use of Union[str, None] instead of str | None in most of the type hints is due to compatibility with python 3.9, which otherwise fails to run the new code

@sydney-runkle
Copy link
Member

@giacbrd,

Thanks for the contribution. I'll chat with the team about this idea tomorrow.

@@ -20,15 +20,15 @@ def __post_init__(self):
self._takes_ctx = len(inspect.signature(self.function).parameters) > 0
self._is_async = inspect.iscoroutinefunction(self.function)

async def run(self, run_context: RunContext[AgentDepsT]) -> str:
async def run(self, run_context: RunContext[AgentDepsT]) -> Union[str, None]: # noqa UP007
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be fine to use str | None thanks to from __future__ import annotations. There may be some places where you need to use Union, but those would only be places where we analyzed the type-hint at runtime.

Copy link
Author

@giacbrd giacbrd Feb 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to add from __future__ import annotations but I still got errors, I probably used it wrong (looking at your next comment), thanks

if self._takes_ctx:
args = (run_context,)
else:
args = ()

if self._is_async:
function = cast(Callable[[Any], Awaitable[str]], self.function)
function = cast(Callable[[Any], Awaitable[Union[str, None]]], self.function)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is a place where you need Union[str, None], though I think it would probably work to just use the string (i.e., 'str | None')

@github-actions github-actions bot temporarily deployed to deploy-preview February 12, 2025 01:13 Inactive
Comment on lines +175 to +178
if updated_part_content:
msg.parts[i] = _messages.SystemPromptPart(
updated_part_content, dynamic_ref=part.dynamic_ref
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems wrong to me — with this change, if the function returns None (or '') we just don't update the part, rather than removing it. I think there are a few ways to do this more correctly:

  • option 1: remove the part from the msg.parts list (which would need to be done carefully..)
  • option 2: allow None in msg.parts, I don't love this
  • option 3: set the msg.parts[i] to be a SystemPromptPart with empty string as the value, and modify the handling in the Model implementations to ignore SystemPromptParts with empty content.

@dmontagu
Copy link
Contributor

dmontagu commented Feb 12, 2025

I pushed a commit to undo some of the unnecessary typing changes.

I feel like the "right" way to fix the issue I pointed out might be to use '' as the value of dynamic system prompts when you want them excluded, and just have the Model implementations not emit an actual system prompt message in those cases.

Either way, we need to make sure that a dynamic system prompt that changes from returning something truthy to returning something non-truthy, or vice versa, doesn't lead to misbehavior.

@giacbrd
Copy link
Author

giacbrd commented Feb 12, 2025

@dmontagu The part updating behaviour is actually tricky. A developer would expect not to change anything if the function does not return a value, but not returning a value (i.e. returning None) should be equivalent to set an empty value with the approach I am proposing. I agree that, in order to avoid ambiguity, especially when updating, it should be better to keep dynamic prompt returning only strings. A developer should explicitly return an empty string to erase a part or return an identical value to keep the part unchanged. This means keeping everything as before on the "formal" aspects, but functionally optimizing the messages data.

The third option could the obvious one, however:

  • It is arbitrary for each model, it would be an optimization that is not offered by the framework, but it is "hidden on an higher layer", in model integrations (you mean setting a condition on a line like this one, right?)
  • The messages data structure would remain "dirty", e.g. when debugging we would always see and count those empty parts

Regarding the first option, do you consider that removing a part would be so dangerous? It is probably an eventuality that must be faced, sooner or later, in future developments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants