Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[nf-tests] Assure reproducibility #55

Open
suzannejin opened this issue Apr 17, 2024 · 3 comments
Open

[nf-tests] Assure reproducibility #55

suzannejin opened this issue Apr 17, 2024 · 3 comments

Comments

@suzannejin
Copy link
Contributor

random sampling
There are many random sampling methods, including random.sample, and other low level within library sampling.
Setting random.seed(0) at the very beginning of a script won't work.

set operations
Sets are unordered, consequently everything handled with sets are not gonna follow a certain order, and this is not controllable.
However, set operations are very efficient.

Alternatives?

@mathysgrapotte
Copy link
Contributor

This will be checked on #73 a similar solution to shuffling #70 can be done (testing a first pipeline run, saving results and checking that further pipeline runs show the same results, this can also be done with nf-tests I believe).

@luisas luisas added this to the 0.1 - IBIS milestone May 3, 2024
@alessiovignoli
Copy link
Contributor

liked to #40

@alessiovignoli
Copy link
Contributor

PR #166 is setting the basis for testing reproducibility. Throught the debug mode. The point is that this issue is much bigger than just checking if output are identical. Because how close to reproducible you are likely depends on the ammount of data, tha size of the model, how long until convergence in learning and the complexity of the problem.

@mathysgrapotte mathysgrapotte removed this from the 0.1 - IBIS milestone Sep 12, 2024
@mathysgrapotte mathysgrapotte changed the title Assure reproducibility [nf-tests] Assure reproducibility Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants