Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Probe-based OP Fusion & Reordering #464

Open
wants to merge 55 commits into
base: main
Choose a base branch
from
Open

Probe-based OP Fusion & Reordering #464

wants to merge 55 commits into from

Conversation

HYLcool
Copy link
Collaborator

@HYLcool HYLcool commented Oct 25, 2024

As the title says.

Also supports adaptive batch size.

@HYLcool
* init adapter
716e1f9
@HYLcool
+ add basic logic of workloads adaptation
584241b
@HYLcool
* update the adaptation logic
de67ccc
@HYLcool
+ add unittests for Monitor
4a368ed
@HYLcool
+ add unittests for Monitor
a9e1ec4
@HYLcool
* use multiprocessing to monitor resource utilization
8ade118
@HYLcool
+ add unittests for adapter
3c4320c
@HYLcool
* modification and fix for gece's comments
5a6c071
@HYLcool
* bug fixed: there is no attr _name in FusedFilter
fe2ce14
@HYLcool
* support OP fusion based on probed speed of each OP
b9d47ee
@HYLcool
* support OP fusion based on probed speed of each OP
f6e017f
@HYLcool
Merge branch 'feat/dj_adapter' into opt/probe_based_fusion
184fa08
@HYLcool
* fix bugs in fused OP speed calculation
7fff259 
+ add unit tests for probe-based OP fusion
@HYLcool
* bug fixed
3586a27
@HYLcool
Merge branch 'main' into opt/probe_based_fusion
98ae1ba 
# Conflicts:
#	data_juicer/core/adapter.py
@HYLcool
Merge branch 'main' into opt/probe_based_fusion
f0f23c8
@HYLcool
* bug fixed: enable batched when is_batched_op is True
676049d
@HYLcool
* bug fixed: enable batched when is_batched_op is True
af17e13
@HYLcool
* expand the test dataset batch according to the num_proc
8a89b24
@HYLcool
* support OP-wise adaptive batch size setting
af77fec
@HYLcool
Merge branch 'main' into opt/probe_based_fusion
2a6b617 
# Conflicts:
#	data_juicer/ops/base_op.py
@HYLcool
* support batched processing for 4 image OPs
784d21c
@HYLcool
* set mp method for each OP during probing as well
06b12c8
@HYLcool
+ add visualization graphs for monitor results
f65c02c
@HYLcool
Merge branch 'main' into opt/probe_based_fusion
71cc37c
@HYLcool
* share the same context space for a batch
f6e80ee
@HYLcool
* share the same context space for a batch
acdf007
@HYLcool
* share the same context space for a batch but with idx info
7334b10
@HYLcool
* support batched process for fused filter
a6565b3
@HYLcool
* bug fixed: add idx info to inter vars of word repetition filter, re…
8a637fa 
…duce the logic value for each op in fused filter
@HYLcool
* extract batched to an outer func
8bf87d0 
* modified for text_length_filter
@HYLcool
* use a branch to decide which funcs are used
4cd9e62
@HYLcool
* update context for each sample as well
ad5dbba
@HYLcool
* modify for mapper and whitespace_normalization_mapper
4d6ffe1
@HYLcool
* modify for two filters with context
1929ddd
@HYLcool
* allow optional args for batched funcs
b98f6d9
@HYLcool
* restore to batched version and rename to xxx_batched
c2188d5
@HYLcool
* restore to batched version and rename to xxx_batched
281c68d
@HYLcool
* restore to batched version and rename to xxx_batched
5bf3173
@HYLcool
* update docs for this modification
3b470f2
@HYLcool
Merge branch 'main' into opt/outer_batched
e60e64b
@HYLcool
* DO NOT allow to override the compute_stats or process methods in th…
23f1a4b 
…e subclass of Mapper and Filter
@HYLcool
* rename the methods for the newly-added OP image_face_count_filter
5ce818f
@HYLcool
Merge branch 'opt/outer_batched' into opt/probe_based_fusion
0c7405d 
# Conflicts:
#	data_juicer/ops/filter/image_aspect_ratio_filter.py
#	data_juicer/ops/filter/image_shape_filter.py
#	data_juicer/ops/filter/image_size_filter.py
#	data_juicer/ops/filter/image_text_similarity_filter.py
#	data_juicer/ops/op_fusion.py
@HYLcool
* rename FusedFilter with "_batched" suffix
634d245
@HYLcool
Merge branch 'main' into opt/probe_based_fusion
cd565ad 
# Conflicts:
#	data_juicer/ops/op_fusion.py
@HYLcool
- bug fixed: update context organization for FusedFilter
b849f2a
@HYLcool
Merge branch 'main' into opt/probe_based_fusion
9b5a7c6
@HYLcool
* merge main into this branch
5bef9f2
@HYLcool
* merge main into this branch
934456e
@HYLcool
* support probe-based op fusion for ray mode
dbb3956
@HYLcool
* restore to separate contexts of different samples
9e1c490
@HYLcool
* restore to separate contexts of different samples
309979b
@HYLcool
* avoid storing contexts from different samples into the same dict ob…
af0c66b 
…ject
@HYLcool
Merge branch 'main' into opt/probe_based_fusion
1e734bc
@HYLcool HYLcool self-assigned this Oct 25, 2024
@HYLcool HYLcool added the enhancement New feature or request label Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@HYLcool HYLcool

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@HYLcool