Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OnDemandFeatureView: Allow python transformations for offline processing. #4490

Closed
job-almekinders opened this issue Sep 5, 2024 · 3 comments
Assignees
Labels
kind/feature New feature or request

Comments

@job-almekinders
Copy link
Contributor

job-almekinders commented Sep 5, 2024

Is your feature request related to a problem? Please describe.
When running a PythonTransformation for offline feature retrieval, an exception is raised here. I understand that python transformations on very large datasets do not scale well. However, in some cases, a user might still want to apply a python transformation to a smaller dataset in the offline store. Due to this error (and the missing implementation of PythonTransformation().transform_arrow()), this is currently not possible.

Describe the solution you'd like
Instead of raising an exception, we could still run the transformation, and also log a warning indicating that it may not scale well. On top of that we can implement the functionality in transform_arrow to make the transformation happen.

Describe alternatives you've considered

Additional context

@job-almekinders job-almekinders added the kind/feature New feature or request label Sep 5, 2024
@franciscojavierarceo franciscojavierarceo self-assigned this Oct 6, 2024
@franciscojavierarceo
Copy link
Member

Oh we should definitely discuss this because I agree. Also, for scaling Spark should really call the Transformation as a UDF and then it can scale very well.

@franciscojavierarceo
Copy link
Member

This has been implemented. 👍

@job-almekinders
Copy link
Contributor Author

Thanks a lot for this! @franciscojavierarceo

FYI @TomSteenbergen :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants