Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(platform): Add multimedia file support & add basic Video blocks #9320

Merged
merged 17 commits into from
Jan 26, 2025

Conversation

majdyz
Copy link
Contributor

@majdyz majdyz commented Jan 23, 2025

Currently, there is no support for passing files in the platform, each generated file should be hosted somewhere.
This PR adds support of passing files temporarily during the execution to open up more block that does multimedia operations.

image image

Changes 🏗️

  • Add media support for passing files (local files, base64, URL) and FileStoreBlock (file version of StoreValueBlock)
  • Add initial multimedia blocks: LoopVideoBlock & AddAudioToVideoBlock.

Checklist 📋

For code changes:

  • I have clearly listed my changes in the PR description
  • I have made a test plan
  • I have tested my changes according to the test plan:
    • ...
Example test plan
  • Create from scratch and execute an agent with at least 3 blocks
  • Import an agent from file upload, and confirm it executes correctly
  • Upload agent to marketplace
  • Import an agent from marketplace and confirm it executes correctly
  • Edit an agent from monitor, and confirm it executes correctly

For configuration changes:

  • .env.example is updated or already compatible with my changes
  • docker-compose.yml is updated or already compatible with my changes
  • I have included a list of my configuration changes in the PR description (under Changes)
Examples of configuration changes
  • Changing ports
  • Adding new services that need to communicate with each other
  • Secrets or environment variable changes
  • New or infrastructure changes such as databases

@majdyz majdyz requested a review from a team as a code owner January 23, 2025 16:48
@majdyz majdyz requested review from ntindle and Swiftyos and removed request for a team January 23, 2025 16:48
@github-actions github-actions bot added platform/frontend AutoGPT Platform - Front end platform/backend AutoGPT Platform - Back end platform/blocks size/xl labels Jan 23, 2025
Copy link

netlify bot commented Jan 23, 2025

Deploy Preview for auto-gpt-docs-dev canceled.

Name Link
🔨 Latest commit da88da9
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs-dev/deploys/679272f1d02c5d0008b0efb7

Copy link

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 Security concerns

SSRF vulnerability:
The file download functionality in store_temp_file() accepts any HTTP/HTTPS URL without proper validation of the domain or file size limits. This could allow an attacker to make requests to internal network resources or cause denial of service through large file downloads. Consider implementing domain whitelisting and file size restrictions.

⚡ Recommended focus areas for review

Resource Management

Video and audio file handles are not properly closed after use. Missing cleanup of VideoFileClip and AudioFileClip objects which could lead to resource leaks.

clip = VideoFileClip(input_abspath)

# 3) Apply the loop effect
looped_clip = clip
if input_data.duration:
    # Loop until we reach the specified duration
    looped_clip = looped_clip.with_effects([Loop(duration=input_data.duration)])
elif input_data.n_loops:
    looped_clip = looped_clip.with_effects([Loop(n=input_data.n_loops)])
else:
    raise ValueError("Either 'duration' or 'n_loops' must be provided.")

assert isinstance(looped_clip, VideoFileClip)

# 4) Save the looped output
output_filename = f"{node_exec_id}_looped_{os.path.basename(local_video_path)}"
output_abspath = get_path(graph_exec_id, output_filename)

looped_clip = looped_clip.with_audio(clip.audio)
looped_clip.write_videofile(output_abspath, codec="libx264", audio_codec="aac")
Security Risk

The file download functionality accepts any HTTP/HTTPS URL without validating the domain or size limits, which could enable server-side request forgery (SSRF) or denial of service attacks.

elif file.startswith("http://") or file.startswith("https://"):
    # === URL ===
    parsed_url = urlparse(file)
    basename = os.path.basename(parsed_url.path) or str(uuid.uuid4())

    intended_path = os.path.join(temp_base, basename)
    absolute_path = _ensure_inside_base(intended_path)

    # Download
    resp = requests.get(file)
    resp.raise_for_status()
    with open(absolute_path, "wb") as f:
        f.write(resp.content)

Copy link

netlify bot commented Jan 23, 2025

Deploy Preview for auto-gpt-docs-dev canceled.

Name Link
🔨 Latest commit 643d1a9
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs-dev/deploys/67964abc4da13a0009ac49ba

Copy link

netlify bot commented Jan 23, 2025

Deploy Preview for auto-gpt-docs ready!

Name Link
🔨 Latest commit da88da9
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs/deploys/679272f1d02c5d0008b0efb2
😎 Deploy Preview https://deploy-preview-9320--auto-gpt-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Copy link

netlify bot commented Jan 23, 2025

Deploy Preview for auto-gpt-docs canceled.

Name Link
🔨 Latest commit 643d1a9
🔍 Latest deploy log https://app.netlify.com/sites/auto-gpt-docs/deploys/67964abc1dbf740008accf2f

@majdyz majdyz requested a review from Pwuts January 23, 2025 17:20
@majdyz majdyz mentioned this pull request Jan 24, 2025
4 tasks
@github-actions github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Jan 24, 2025
Copy link
Contributor

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

ntindle
ntindle previously approved these changes Jan 24, 2025
Copy link
Contributor

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

@github-actions github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Jan 24, 2025
ntindle
ntindle previously approved these changes Jan 24, 2025
ntindle
ntindle previously approved these changes Jan 26, 2025
@majdyz majdyz mentioned this pull request Jan 26, 2025
4 tasks
@majdyz majdyz added this pull request to the merge queue Jan 26, 2025
@ntindle ntindle removed this pull request from the merge queue due to a manual request Jan 26, 2025
@majdyz majdyz enabled auto-merge January 26, 2025 15:10
@majdyz majdyz added this pull request to the merge queue Jan 26, 2025
Merged via the queue into dev with commit bb3be44 Jan 26, 2025
24 checks passed
@majdyz majdyz deleted the zamilmajdy/multimediafilesupport branch January 26, 2025 16:22
waterstark pushed a commit to waterstark/AutoGPT that referenced this pull request Jan 30, 2025
…ignificant-Gravitas#9320)

Currently, there is no support for passing files in the platform, each
generated file should be hosted somewhere.
This PR adds support of passing files temporarily during the execution
to open up more block that does multimedia operations.

<img width="583" alt="image"
src="https://github.com/user-attachments/assets/c285de5a-c2a9-41a0-9be1-305a316879d6"
/>

<img width="1291" alt="image"
src="https://github.com/user-attachments/assets/d7bcaf38-80fa-4b51-91da-b4eed80a02c1"
/>


### Changes 🏗️

* Add media support for passing files (local files, base64, URL) and
`FileStoreBlock` (file version of `StoreValueBlock`)
* Add initial multimedia blocks: `LoopVideoBlock` &
`AddAudioToVideoBlock`.

### Checklist 📋

#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
  <!-- Put your test plan here: -->
  - [ ] ...

<details>
  <summary>Example test plan</summary>
  
  - [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
  - [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
  - [ ] Edit an agent from monitor, and confirm it executes correctly
</details>

#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)

<details>
  <summary>Examples of configuration changes</summary>

  - Changing ports
  - Adding new services that need to communicate with each other
  - Secrets or environment variable changes
  - New or infrastructure changes such as databases
</details>

---------

Co-authored-by: Nicholas Tindle <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
platform/backend AutoGPT Platform - Back end platform/blocks platform/frontend AutoGPT Platform - Front end size/xl
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants