You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, pmda assumes that all problems can be solved with a map-reduce scheme that maps over the frames. This scales well for a lot of problems. If an analysis type requires a lot of work per frame it might make sense to also parallelize the work on a single frame. This is not supported in the current.
I would, therefore, suggest adding a new base class FrameMapReduceBase that is a direct copy of ParallelAnalysisBase and replace the later. The structure would then be.
classParallelAnalysisBase:
""" define public API for *any* pmda class """defrun(self, get=None, scheduler=None, n_jobs=-1, *args, **kwargs)
ifgetisnotNone:
scheduler=getself._run(scheduler=scheduler, n_jobs=n_jobs, *args, **kwargs)
returnselfdef_run(self, scheduler=None, n_jobs=-1, *args, **kwargs):
raiseNotImplementedErrorclassFrameMapReduceBase(ParallelAnalysisBase):
""" Implement Map-Reduce and 'map' over frames """def_run(self, scheduler=None, n_jobs=-1, *args, **kwargs):
# current implementation
This would better document the parallelization scheme we use and be very explicit about what functions we expect for a user-facing API. Right now this would only be the run method and it should return a reference to the class. If an analysis uses multiple-cores but a different scheme to generate the DAG-graph like #47 it would only need to inherit from ParallelAnalysisBase and is free to implement it's own DAG.
The text was updated successfully, but these errors were encountered:
Currently, pmda assumes that all problems can be solved with a map-reduce scheme that maps over the frames. This scales well for a lot of problems. If an analysis type requires a lot of work per frame it might make sense to also parallelize the work on a single frame. This is not supported in the current.
I would, therefore, suggest adding a new base class
FrameMapReduceBase
that is a direct copy ofParallelAnalysisBase
and replace the later. The structure would then be.This would better document the parallelization scheme we use and be very explicit about what functions we expect for a user-facing API. Right now this would only be the
run
method and it should return a reference to the class. If an analysis uses multiple-cores but a different scheme to generate the DAG-graph like #47 it would only need to inherit fromParallelAnalysisBase
and is free to implement it's own DAG.The text was updated successfully, but these errors were encountered: