feat: image block for claude #9309

ntindle · 2025-01-21T16:03:31Z

I wanted claude to take image inputs

Changes 🏗️

Adds image inputs to claude (only latest message)

Checklist 📋

For code changes:

I have clearly listed my changes in the PR description
I have made a test plan
I have tested my changes according to the test plan:
- Built agent with it

qodo-merge-pro · 2025-01-21T16:03:54Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 Security concerns Input validation: The code processes base64 encoded image data from user input without proper validation of the content or size limits. This could potentially lead to memory issues or processing of malicious content. Consider adding validation for maximum image size and proper base64/MIME type validation.
⚡ Recommended focus areas for review Error Handling The error handling in the parse_response function returns partial results even when validation fails. Consider returning empty dict consistently on all error cases. try: parsed = json.loads(resp) if not isinstance(parsed, dict): return {}, f"Expected a dictionary, but got {type(parsed)}" if input_data.expected_format: miss_keys = set(input_data.expected_format.keys()) - set( parsed.keys() ) if miss_keys: return parsed, f"Missing keys: {miss_keys}" return parsed, None except JSONDecodeError as e: return {}, f"JSON decode error: {e}" Resource Management The Anthropic client is created for each call but not properly closed/cleaned up. Consider implementing proper resource management. client = anthropic.Anthropic(api_key=credentials.api_key.get_secret_value()) Input Validation The prompt_values handling assumes valid image data without thorough validation of the base64 content and MIME type. for key, value in values.items(): if isinstance(value, dict) and "content_type" in value and "data" in value: # This is an image content.append( { "type": "image", "source": { "type": "base64", "media_type": value["content_type"], "data": value["data"], }, } )

…' into zamilmajdy/multimediafilesupport

Bentlybro · 2025-01-24T14:23:46Z

i tried this and this is the full error

2025-01-24 14:21:34 Traceback (most recent call last):
2025-01-24 14:21:34   File "<frozen runpy>", line 198, in _run_module_as_main
2025-01-24 14:21:34   File "<frozen runpy>", line 88, in _run_code
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/rest.py", line 2, in <module>
2025-01-24 14:21:34     from backend.executor import DatabaseManager, ExecutionScheduler
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/executor/__init__.py", line 1, in <module>
2025-01-24 14:21:34     from .database import DatabaseManager
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/executor/database.py", line 4, in <module>
2025-01-24 14:21:34     from backend.data.credit import get_user_credit_model
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/data/credit.py", line 13, in <module>
2025-01-24 14:21:34     from backend.data.block_cost_config import BLOCK_COSTS
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/data/block_cost_config.py", line 3, in <module>
2025-01-24 14:21:34     from backend.blocks.ai_music_generator import AIMusicGeneratorBlock
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/blocks/__init__.py", line 24, in <module>
2025-01-24 14:21:34     importlib.import_module(f".{module}", package=__name__)
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
2025-01-24 14:21:34     return _bootstrap._gcd_import(name[level:], package, level)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/blocks/llm.py", line 1136, in <module>
2025-01-24 14:21:34     class ClaudeWithImageBlock(Block):
2025-01-24 14:21:34   File "/app/autogpt_platform/backend/backend/blocks/llm.py", line 1139, in ClaudeWithImageBlock
2025-01-24 14:21:34     class Input(BlockSchema):
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 226, in __new__
2025-01-24 14:21:34     complete_model_class(
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 658, in complete_model_class
2025-01-24 14:21:34     schema = cls.__get_pydantic_core_schema__(cls, handler)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/main.py", line 702, in __get_pydantic_core_schema__
2025-01-24 14:21:34     return handler(source)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
2025-01-24 14:21:34     schema = self._handler(source_type)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 881, in _generate_schema_inner
2025-01-24 14:21:34     return self._model_schema(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 693, in _model_schema
2025-01-24 14:21:34     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2025-01-24 14:21:34     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 693, in <dictcomp>
2025-01-24 14:21:34     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
2025-01-24 14:21:34         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1073, in _generate_md_field_schema
2025-01-24 14:21:34     common_field = self._common_field_schema(name, field_info, decorators)
2025-01-24 14:21:34                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1265, in _common_field_schema
2025-01-24 14:21:34     schema = self._apply_annotations(
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 2062, in _apply_annotations
2025-01-24 14:21:34     schema = get_inner_schema(source_type)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 84, in __call__
2025-01-24 14:21:34     schema = self._handler(source_type)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 2043, in inner_handler
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
2025-01-24 14:21:34     return self.match_type(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 988, in match_type
2025-01-24 14:21:34     return self._match_generic_type(obj, origin)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1026, in _match_generic_type
2025-01-24 14:21:34     return self._dict_schema(*self._get_first_two_args_or_any(obj))
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 372, in _dict_schema
2025-01-24 14:21:34     return core_schema.dict_schema(self.generate_schema(keys_type), self.generate_schema(values_type))
2025-01-24 14:21:34                                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
2025-01-24 14:21:34     return self.match_type(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 988, in match_type
2025-01-24 14:21:34     return self._match_generic_type(obj, origin)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1016, in _match_generic_type
2025-01-24 14:21:34     return self._union_schema(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1327, in _union_schema
2025-01-24 14:21:34     choices.append(self.generate_schema(arg))
2025-01-24 14:21:34                    ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 612, in generate_schema
2025-01-24 14:21:34     schema = self._generate_schema_inner(obj)
2025-01-24 14:21:34              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 886, in _generate_schema_inner
2025-01-24 14:21:34     return self.match_type(obj)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 958, in match_type
2025-01-24 14:21:34     return self._typed_dict_schema(obj, None)
2025-01-24 14:21:34            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-24 14:21:34   File "/usr/local/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1408, in _typed_dict_schema
2025-01-24 14:21:34     raise PydanticUserError(
2025-01-24 14:21:34 pydantic.errors.PydanticUserError: Please use `typing_extensions.TypedDict` instead of `typing.TypedDict` on Python < 3.12.
2025-01-24 14:21:34 
2025-01-24 14:21:34 For further information visit https://errors.pydantic.dev/2.10/u/typed-dict-version

…milmajdy/multimediafilesupport

…entials_store.py

…iders.py

…ons/credentials-input.tsx

…ons/credentials-provider.tsx

…pi/types.ts

majdyz · 2025-01-26T13:22:39Z

autogpt_platform/backend/backend/blocks/llm.py

+
+
+class Image(BaseModel):
+    content_type: str  # MIME type of the image


base64 URI is already within this format:

data:mime/type;base64=.......
https://developer.mozilla.org/en-US/docs/Web/URI/Schemes/data#syntax

#9320 already uses this format, and this can be shown to the frontend directly as embedded image/video/audio

would that type just be string?

ntindle requested a review from a team as a code owner January 21, 2025 16:03

ntindle requested review from Swiftyos and Pwuts and removed request for a team January 21, 2025 16:03

github-actions bot added size/l platform/backend AutoGPT Platform - Back end platform/blocks and removed size/l labels Jan 21, 2025

qodo-merge-pro bot added the Review effort [1-5]: 4 label Jan 21, 2025

github-actions bot added the size/l label Jan 21, 2025

majdyz added 10 commits January 23, 2025 17:39

feat(platform): Add multimedia file support & add basic Video blocks

33390ff

feat(platform): Add multimedia file support & add basic Video blocks

fed426f

feat(platform): Add multimedia file support & add basic Video blocks

da88da9

Merge branch 'dev' into zamilmajdy/multimediafilesupport

31e49fb

Remove test

2bfaf4d

Merge remote-tracking branch 'origin/zamilmajdy/multimediafilesupport…

81c403e

…' into zamilmajdy/multimediafilesupport

fix

50ad4a3

Revert

a1a52b9

Address changes

d311679

Merge branch 'dev' into zamilmajdy/multimediafilesupport

b052413

majdyz and others added 8 commits January 24, 2025 18:17

Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into za…

841500f

…milmajdy/multimediafilesupport

Merge branch 'dev' into zamilmajdy/multimediafilesupport

d72c93c

Merge branch 'dev' into zamilmajdy/multimediafilesupport

11ef048

Merge branch 'dev' into zamilmajdy/multimediafilesupport

769c75e

feat: screenshotone

23095f4

feat: image block for claude

20b4a0e

fix: lint

3aafa53

fix: use pydantic not typed dict

b8749f7

ntindle force-pushed the claude-image-blcok branch from 8ea1e5e to b8749f7 Compare January 26, 2025 13:09

github-actions bot added platform/frontend AutoGPT Platform - Front end size/xl and removed size/l labels Jan 26, 2025

ntindle changed the base branch from screenshotone to zamilmajdy/multimediafilesupport January 26, 2025 13:09

ntindle added 6 commits January 26, 2025 14:10

Discard changes to autogpt_platform/backend/backend/integrations/cred…

3e0bcbc

…entials_store.py

Discard changes to autogpt_platform/backend/backend/integrations/prov…

5aadbfe

…iders.py

Discard changes to autogpt_platform/backend/backend/util/settings.py

cba0536

Discard changes to autogpt_platform/frontend/src/components/integrati…

80ee8c6

…ons/credentials-input.tsx

Discard changes to autogpt_platform/frontend/src/components/integrati…

7682cbb

…ons/credentials-provider.tsx

Discard changes to autogpt_platform/frontend/src/lib/autogpt-server-a…

d3425ca

…pi/types.ts

github-actions bot removed the platform/frontend AutoGPT Platform - Front end label Jan 26, 2025

Delete autogpt_platform/backend/backend/blocks/screenshotone.py

d86a411

github-actions bot added size/l and removed size/xl labels Jan 26, 2025

fix: remove block/unblock as its been removed by twitter

96ffa64

majdyz reviewed Jan 26, 2025

View reviewed changes

ntindle and others added 4 commits January 26, 2025 15:05

Merge branch 'zamilmajdy/multimediafilesupport' into claude-image-blcok

5bb43c3

type alias string into MediaFile

a4fc0d6

Merge branch 'dev' into zamilmajdy/multimediafilesupport

643d1a9

Merge branch 'zamilmajdy/multimediafilesupport' into claude-image-blcok

c1e8451

Base automatically changed from zamilmajdy/multimediafilesupport to dev January 26, 2025 16:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: image block for claude #9309

feat: image block for claude #9309

ntindle commented Jan 21, 2025 •

edited

Loading

qodo-merge-pro bot commented Jan 21, 2025

Bentlybro commented Jan 24, 2025

majdyz Jan 26, 2025

majdyz Jan 26, 2025

ntindle Jan 26, 2025



		class Image(BaseModel):
		content_type: str # MIME type of the image

feat: image block for claude #9309

Are you sure you want to change the base?

feat: image block for claude #9309

Conversation

ntindle commented Jan 21, 2025 • edited Loading

Changes 🏗️

Checklist 📋

For code changes:

qodo-merge-pro bot commented Jan 21, 2025

PR Reviewer Guide 🔍

Bentlybro commented Jan 24, 2025

majdyz Jan 26, 2025

Choose a reason for hiding this comment

majdyz Jan 26, 2025

Choose a reason for hiding this comment

ntindle Jan 26, 2025

Choose a reason for hiding this comment

ntindle commented Jan 21, 2025 •

edited

Loading