Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent execution of workflow steps #544

Merged
merged 159 commits into from
Dec 11, 2023
Merged
Changes from 1 commit
Commits
Show all changes
159 commits
Select commit Hold shift + click to select a range
fcb89e3
Move data collection to seperate method
calum-chamberlain Mar 31, 2023
16f23dd
start making queue consumers
calum-chamberlain Apr 2, 2023
acda3bb
Possible implementation - untested
calum-chamberlain Apr 3, 2023
56e6e6c
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 3, 2023
d146c78
Process closing
calum-chamberlain Apr 3, 2023
d1e8b8e
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 3, 2023
fae3e2f
Enforce same processing and run all in detect
calum-chamberlain Apr 3, 2023
8ddd44f
Debugging
calum-chamberlain Apr 4, 2023
431c2c7
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 4, 2023
7f4e72e
Threshold in right place and allow pickle reading
calum-chamberlain Apr 5, 2023
6e7a679
Do not allow more threads than are available
calum-chamberlain Apr 5, 2023
6c5d26b
Find peaks needs to not request too many threads to avoid segfaults
calum-chamberlain Apr 5, 2023
f47bfb5
Parallel spike test
calum-chamberlain Apr 6, 2023
8da5896
Multi-find peaks using multithreading rather than openmp - faster and…
calum-chamberlain Apr 6, 2023
0b20a4e
Working concurrency
calum-chamberlain Apr 6, 2023
2cfbd3f
Faster party addition
calum-chamberlain Apr 6, 2023
f25620e
Return errors properly
calum-chamberlain Apr 9, 2023
a148503
Test correct place for cached files
calum-chamberlain Apr 9, 2023
c79935b
Check for None in add
calum-chamberlain Apr 9, 2023
4279dbd
termination
calum-chamberlain Apr 9, 2023
c081bd1
Remove multi find peaks
calum-chamberlain Apr 10, 2023
10097c7
timeout
calum-chamberlain Apr 10, 2023
944c60e
Do not test removed funcs
calum-chamberlain Apr 10, 2023
f936750
Develop and test serial implementation
calum-chamberlain Apr 10, 2023
a3be6cc
Cope with old tribes not having attributes
calum-chamberlain Apr 10, 2023
bb61c8c
Prep for correaltion in seperate process - can be slowdown for large …
calum-chamberlain Apr 10, 2023
ef6fa1f
Faster throughput and timers
calum-chamberlain Apr 10, 2023
af0c46d
More correaltion prep out of mainProcess
calum-chamberlain Apr 11, 2023
86ccdab
Make sure kwargs aren't changed
calum-chamberlain Apr 11, 2023
81b4b30
typos
calum-chamberlain Apr 11, 2023
f4e953e
Reduced streams held in memory at any time
calum-chamberlain Apr 11, 2023
bd407a9
Patch serial
calum-chamberlain Apr 11, 2023
4ac4f39
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 11, 2023
27d610d
Better memory management
calum-chamberlain Apr 12, 2023
df5cd26
Better error message if detctions out of range
calum-chamberlain Apr 12, 2023
dcb7779
typo
calum-chamberlain Apr 12, 2023
e5870fe
Allow slightly out correlations due to fp precision accumulation
calum-chamberlain Apr 12, 2023
36218cb
Empty the sid queue in serial
calum-chamberlain Apr 12, 2023
4ffd2fc
Empty the sid queue in serial
calum-chamberlain Apr 12, 2023
f60d568
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 12, 2023
2ce2510
Catch updated error
calum-chamberlain Apr 12, 2023
82cc9ee
Windows compat file names
calum-chamberlain Apr 12, 2023
d3dbed4
Limit rate of stream accumulation
calum-chamberlain Apr 13, 2023
27c269e
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 13, 2023
5e19763
Only pass template names through queue
calum-chamberlain Apr 13, 2023
44858a8
Limit downlaod rate
calum-chamberlain Apr 13, 2023
6b88898
Run serial downloads in serial
calum-chamberlain Apr 13, 2023
d338f15
Run serial downloads in serial
calum-chamberlain Apr 13, 2023
9c86178
Run most relevent test concurrent and serial
calum-chamberlain Apr 14, 2023
228de78
Precision should be per channel
calum-chamberlain Apr 14, 2023
5d2fd30
Only group when needed
calum-chamberlain Apr 14, 2023
4639c18
Make serial processing serial
calum-chamberlain Apr 14, 2023
3a712b7
debug wait
calum-chamberlain Apr 14, 2023
d2882a3
Enforce unique template names
calum-chamberlain Apr 16, 2023
5a9731f
Reduce copying, use disk for cache and reduce processes
calum-chamberlain Apr 17, 2023
1749060
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 17, 2023
81050a2
Stickler
calum-chamberlain Apr 17, 2023
d6b5a18
Stickler
calum-chamberlain Apr 17, 2023
adc3a9d
Raise errors properly
calum-chamberlain Apr 18, 2023
07ca579
Windows compat file names
calum-chamberlain Apr 18, 2023
b5dd6d2
Split and merge for stream IO - make tutorials run
calum-chamberlain Apr 18, 2023
aff2a50
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 18, 2023
f2cd9b9
Run concurrent first
calum-chamberlain Apr 18, 2023
0b2c3d1
Pickle rather than mseed - mseed has size limits
calum-chamberlain Apr 18, 2023
d31d6d1
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Apr 18, 2023
543bd32
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Jul 13, 2023
a5ecf31
Do not edit lists within loops
calum-chamberlain Jul 13, 2023
81e51d5
debug output
calum-chamberlain Jul 13, 2023
1feda85
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Jul 13, 2023
c65c262
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Jul 17, 2023
4467e96
Catch template db deserialize errors; check for poison while waiting
calum-chamberlain Jul 17, 2023
bab2b90
Handle overlaps in client_detect properly
calum-chamberlain Jul 18, 2023
cad133a
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Jul 27, 2023
2bef502
Revert "Handle overlaps in client_detect properly"
calum-chamberlain Jul 27, 2023
030a929
More debug info for resampling - require double-precision
calum-chamberlain Sep 25, 2023
40cc0a0
Enforce catalog has events
calum-chamberlain Sep 25, 2023
efde3be
Debug differences in correlation due to one-sample out pre-resample
calum-chamberlain Sep 25, 2023
e27a80c
Cope with missing traces in remove
calum-chamberlain Sep 25, 2023
a493cca
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Sep 26, 2023
a777084
Allow passing of pre-computed groupings to tribe.detect
calum-chamberlain Oct 2, 2023
2f76294
Allow passing of pre-computed groupings to tribe.detect
calum-chamberlain Oct 2, 2023
f1e89d7
Merge branch 'develop' into parallel-steps
calum-chamberlain Oct 21, 2023
7bfc89d
Merge branch 'develop' into parallel-steps
calum-chamberlain Oct 21, 2023
8cf4528
Reduce debug output
calum-chamberlain Oct 21, 2023
c8f4a38
Do not remove .parties directory
calum-chamberlain Oct 21, 2023
05fe64e
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Oct 21, 2023
f25cada
Allow small variations due to processing differences
calum-chamberlain Oct 24, 2023
181d2ed
Retain double-precision in resample when given dp data
calum-chamberlain Oct 24, 2023
a1e22dc
Process unique directories for concurent work
calum-chamberlain Oct 24, 2023
77ea70a
Add loc kwarg for mpl update
calum-chamberlain Oct 24, 2023
f74e85f
Merge branch 'develop' into parallel-steps
calum-chamberlain Oct 27, 2023
e365f93
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Oct 27, 2023
62baecf
Merge after gap check and before length thresholding
calum-chamberlain Oct 27, 2023
7a46eb9
Reduce query size
calum-chamberlain Oct 27, 2023
15832aa
Be slightly forgiving in time
calum-chamberlain Oct 27, 2023
aefd908
Don't run slow tests twice
calum-chamberlain Oct 27, 2023
307ede9
Remove spare print
calum-chamberlain Oct 27, 2023
70c6be8
Match timing significance (1 sample)
calum-chamberlain Oct 27, 2023
6b8ef05
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Oct 29, 2023
c143d36
Move functions out for tribe for ease of maintaining
calum-chamberlain Nov 11, 2023
68de44d
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 14, 2023
3ff5029
Changelog
calum-chamberlain Nov 14, 2023
24988d7
Work on tutorial updates
calum-chamberlain Nov 16, 2023
b510a9f
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 16, 2023
92e776f
Use geonet AWS archive for data access
calum-chamberlain Nov 16, 2023
07c3cd7
Get dataset working
calum-chamberlain Nov 19, 2023
23a3257
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 19, 2023
acdff14
Filter picks
calum-chamberlain Nov 20, 2023
3d59d47
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 20, 2023
4d37db3
Complete tutorial
calum-chamberlain Nov 20, 2023
873dfd5
Cope with missing phase hints in picks
calum-chamberlain Nov 20, 2023
cb94fe1
Remove outdated tutorial tests
calum-chamberlain Nov 21, 2023
01c9361
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 21, 2023
52f9757
Debug output
calum-chamberlain Nov 26, 2023
14567b7
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 26, 2023
0c4dc2f
Do not use concurrent processing by default
calum-chamberlain Nov 28, 2023
c577ebd
Merge branch 'develop' into parallel-steps
calum-chamberlain Nov 28, 2023
229a00a
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 28, 2023
388d606
Do not run context manager in parallel when testing
calum-chamberlain Nov 28, 2023
3636578
Cope with files being cleaned early
calum-chamberlain Nov 28, 2023
cd929cb
Report on seed id overlap
calum-chamberlain Nov 28, 2023
24d23ec
Merge branch 'develop' into parallel-steps
calum-chamberlain Nov 28, 2023
bc22d4b
Flake8
calum-chamberlain Nov 28, 2023
974f987
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Nov 28, 2023
0476548
Flake8
calum-chamberlain Nov 28, 2023
4c0aecc
Standardise queue naming
calum-chamberlain Nov 30, 2023
3c00f57
Doc strings
calum-chamberlain Nov 30, 2023
cb557b6
Comment check
calum-chamberlain Dec 1, 2023
3ee0255
Do not build pdf by default
calum-chamberlain Dec 1, 2023
540fa89
Rerun notebook with only Warnings
calum-chamberlain Dec 1, 2023
c824ce0
Merge branch 'develop' into parallel-steps
calum-chamberlain Dec 1, 2023
51831c0
Enhance findpeaks coverage - remove unused _multi_decluster
calum-chamberlain Dec 1, 2023
1d703b9
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Dec 1, 2023
47ca25f
Add pre-grouped test
calum-chamberlain Dec 1, 2023
0200767
Remove unused process
calum-chamberlain Dec 1, 2023
cd12ba8
Remove unused import
calum-chamberlain Dec 1, 2023
7e4f526
Check for errors in party check
calum-chamberlain Dec 1, 2023
f9f9ec1
Start testing helper processes
calum-chamberlain Dec 7, 2023
4eb31f5
flake8
calum-chamberlain Dec 7, 2023
cd5e2de
Paths
calum-chamberlain Dec 7, 2023
517c856
Raise exception, not poison
calum-chamberlain Dec 7, 2023
e8c6704
Wait for poison
calum-chamberlain Dec 7, 2023
299b988
Wait for poison
calum-chamberlain Dec 7, 2023
8132747
Wait longer
calum-chamberlain Dec 7, 2023
d24c9b3
Check print
calum-chamberlain Dec 7, 2023
ecd67fc
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Dec 7, 2023
40e28b5
Restructure helper tests for re-use
calum-chamberlain Dec 8, 2023
0e2e5da
Basic tests for all processes
calum-chamberlain Dec 8, 2023
f5e81a5
Cope with Process coverage reporting
calum-chamberlain Dec 8, 2023
10e22e1
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Dec 10, 2023
8cd77d7
Allow longer wait if needed
calum-chamberlain Dec 10, 2023
e59c0d9
Merge branch 'parallel-steps' of https://github.com/eqcorrscan/EQcorr…
calum-chamberlain Dec 10, 2023
c9b734e
Do not wait for output to be empty for too long
calum-chamberlain Dec 10, 2023
f5267a6
Report threading processing
calum-chamberlain Dec 10, 2023
9f26cb0
Tribe doctests
calum-chamberlain Dec 11, 2023
3c54c1d
REstructure archive access to cover more
calum-chamberlain Dec 11, 2023
4059450
Coverage
calum-chamberlain Dec 11, 2023
02643fa
typo
calum-chamberlain Dec 11, 2023
9d863b6
Test event similarity checks
calum-chamberlain Dec 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Better memory management
calum-chamberlain committed Apr 12, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 27d610d0d70daaaabfdf4e7beb905d3564578566
Loading