You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given that the Python GIL prevents us from running the PandasEngine (from #24) across multiple cores, we should have the ability to partition the input data into a number of separate processes and execute those in parallel. Probably some sort of transform_reduce would work. We'll need to make sure we handle separate interpreters correctly. My thought is that we keep each PandasEngine single threaded and simply spawn a bunch of them in the PythonTiler itself.
The text was updated successfully, but these errors were encountered:
Given that the Python GIL prevents us from running the PandasEngine (from #24) across multiple cores, we should have the ability to partition the input data into a number of separate processes and execute those in parallel. Probably some sort of
transform_reduce
would work. We'll need to make sure we handle separate interpreters correctly. My thought is that we keep each PandasEngine single threaded and simply spawn a bunch of them in thePythonTiler
itself.The text was updated successfully, but these errors were encountered: