Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Erroring with Pydantic defeats the purpose of Marvin #936

Open
3 tasks done
landmann opened this issue Jun 11, 2024 · 2 comments
Open
3 tasks done

Erroring with Pydantic defeats the purpose of Marvin #936

landmann opened this issue Jun 11, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@landmann
Copy link

landmann commented Jun 11, 2024

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to look for a similar issue and didn't find it.
  • I searched the Marvin documentation for this feature.

Describe the current behavior

Seems like having Marvin error out when the response is outside the bounds of the responses defeats the purpose of marvin. If I'm giving marvin some bounds, it should use the numeric bounds to give me the response. At least, there should be a setting that lets me strictly adhere to said bounds. Otherwise there's no way for me to make sure the responses are valid, erroring my application.

Describe the proposed behavior

Said above.

Example Use

No response

Additional context

Traceback (most recent call last):
    process_text_data(url)
    ages = marvin.cast(
           ^^^^^^^^^^^^
    return run_sync(
           ^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/utilities/asyncio.py", line 106, in run_sync
    return context.run(asyncio.run, coroutine)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.7_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 321, in cast_async
    return await _generate_typed_llm_response_with_tool(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 187, in _generate_typed_llm_response_with_tool
    response = await generate_llm_response(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 126, in generate_llm_response
    tool_outputs = _get_tool_outputs(request, response)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/ai/text.py", line 134, in _get_tool_outputs
    tool_output = marvin.utilities.tools.call_function_tool(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/utilities/tools.py", line 176, in call_function_tool
    output = tool.function._python_fn(**arguments)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/marvin/utilities/tools.py", line 85, in tool_fn
    return TypeAdapter(model).validate_python(data).value
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/pydantic/type_adapter.py", line 260, in validate_python
    return self.validator.validate_python(object, strict=strict, from_attributes=from_attributes, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for FormatResponse
value.age
  Input should be less than or equal to 64 [type=less_than_equal, input_value=65, input_type=int]
    For further information visit https://errors.pydantic.dev/2.7/v/less_than_equal
@landmann landmann added the enhancement New feature or request label Jun 11, 2024
@zzstoatzz
Copy link
Collaborator

hi @landmann - thanks for the issue

can you provide the example you used to produce that stack trace?

@pietz
Copy link

pietz commented Sep 14, 2024

Yeah, I got the same error. I think it happens when you have a field defined like so:

rating: int = Field(..., ge=1, le=5, description="A rating from 1 to 5")

And the LLM returns a value outside this range. Are the <, >, <=, >= validators stored int he schema the model gets?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants