New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tests for Fed EVal #1296

Merged

tanwarsh merged 24 commits into securefederatedai:develop from payalcha:develop

Jan 27, 2025

Contributor

payalcha commented Jan 22, 2025 •

edited

Loading

Add tests for FedEval
Steps -

Create a workspace and train the data using task_group learning
Note down the last round aggregated_model_validation/accuracy
Create a new workspace for evaluation
Copy the best model of prev round to initial model in the workspace
Run with task_group evaluation
Validate that both accuracy is +- 1%
Also change the mounting process of dockerize job

Create workflow file for the tests

Jobs -
Regression -
https://github.com/payalcha/openfl/actions/runs/12926470885
https://github.com/payalcha/openfl/actions/runs/12926488973

Added tests -
https://github.com/payalcha/openfl/actions/runs/12943482432
https://github.com/payalcha/openfl/actions/runs/12943483520

payalcha added 2 commits

January 22, 2025 02:52


          Add tests for Fed EVal

56d8817

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Merge branch 'develop' into develop

6092a65

github-advanced-security bot found potential problems

View reviewed changes

tests/end_to_end/test_suites/tr_with_eval_tests.py Fixed Show fixed Hide fixed

tests/end_to_end/test_suites/tr_with_eval_tests.py Fixed Show fixed Hide fixed

tests/end_to_end/test_suites/tr_with_eval_tests.py Fixed Show fixed Hide fixed

tests/end_to_end/test_suites/tr_with_eval_tests.py Fixed Show fixed Hide fixed

tests/end_to_end/test_suites/tr_with_eval_tests.py Fixed Show fixed Hide fixed

tests/end_to_end/test_suites/tr_with_eval_tests.py Fixed Show fixed Hide fixed


          Small fix

61d867c

Signed-off-by: Chaurasiya, Payal <[email protected]>

payalcha marked this pull request as draft

January 22, 2025 11:07

payalcha added 15 commits

January 22, 2025 03:12


          Fix docker run

6658aa8

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Run for only 3.10

2a59ce5

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Docker test mount the file

1766e3b

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Docker test mount the file

4e7f7d7

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Merge branch 'develop' into develop

ac8f73d


          Docker test mount the file

d20b23c

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Fix fixture name

1e5931a

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Fix docker job

c98c4e2

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Fix docker job

c2100e8

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Fix docker job

7de3a1f

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Fix docker job

72ef593

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Add use_tls flag

87ba787

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Fix docker job

195e6a8

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Fix docker job

204e0e0

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Merge branch 'develop' into develop

b258c3f

payalcha marked this pull request as ready for review

January 23, 2025 10:19

noopurintel reviewed

View reviewed changes

tests/end_to_end/utils/tr_workspace.py Show resolved Hide resolved

tests/end_to_end/utils/tr_workspace.py Show resolved Hide resolved

tests/end_to_end/utils/tr_workspace.py Outdated Show resolved Hide resolved

tests/end_to_end/utils/tr_workspace.py Outdated Show resolved Hide resolved

tests/end_to_end/models/aggregator.py Show resolved Hide resolved

tests/end_to_end/models/model_owner.py Outdated Show resolved Hide resolved

tests/end_to_end/models/model_owner.py Outdated Show resolved Hide resolved

tests/end_to_end/test_suites/tr_with_eval_tests.py Outdated Show resolved Hide resolved

tests/end_to_end/test_suites/tr_with_eval_tests.py Outdated Show resolved Hide resolved

.github/workflows/task_runner_eval_dws_e2e.yml Outdated Show resolved Hide resolved

payalcha added 2 commits

January 23, 2025 05:58


          Review comments

914ffc3

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Eval --> FedEval

af8ed27

Signed-off-by: Chaurasiya, Payal <[email protected]>

github-advanced-security bot found potential problems

View reviewed changes

tests/end_to_end/test_suites/tr_with_fedeval_tests.py

Comment on lines +31 to +32

		assert fed_helper.verify_federation_run_completion(
		fx_federation_tr,

Check notice

Code scanning / Bandit

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

tests/end_to_end/test_suites/tr_with_fedeval_tests.py

Comment on lines +48 to +49

		assert fed_helper.verify_federation_run_completion(
		new_fed_obj,

Check notice

Code scanning / Bandit

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

tests/end_to_end/test_suites/tr_with_fedeval_tests.py

+                  log.info(f"Model accuracy during evaluation only on prev trained model is : {model_accuracy_eval})")
+                  # verify that the model accuracy is similar to the previous model accuracy max of 1% difference
+                  assert abs(model_accuracy - model_accuracy_eval) <= 0.01, "Model accuracy is not similar to the previous model accuracy"

Check notice

Code scanning / Bandit

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

tests/end_to_end/test_suites/tr_with_fedeval_tests.py

Comment on lines +77 to +78

		assert fed_helper.verify_federation_run_completion(
		fx_federation_tr_dws,

Check notice

Code scanning / Bandit

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

tests/end_to_end/test_suites/tr_with_fedeval_tests.py

Comment on lines +95 to +96

		assert fed_helper.verify_federation_run_completion(
		new_fed_obj,

Check notice

Code scanning / Bandit

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

tests/end_to_end/test_suites/tr_with_fedeval_tests.py

+                  log.info(f"Model accuracy during evaluation only on prev trained model is : {model_accuracy_eval})")
+                  # verify that the model accuracy is similar to the previous model accuracy max of 1% difference
+                  assert abs(model_accuracy - model_accuracy_eval) <= 0.01, "Model accuracy is not similar to the previous model accuracy"

Check notice

Code scanning / Bandit

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.

noopurintel approved these changes

View reviewed changes

ishaileshpant reviewed

View reviewed changes

tests/end_to_end/test_suites/memory_logs_tests.py Outdated Show resolved Hide resolved

ishaileshpant reviewed

View reviewed changes

tests/end_to_end/test_suites/tr_with_fedeval_tests.py Show resolved Hide resolved

ishaileshpant reviewed

View reviewed changes

tests/end_to_end/test_suites/tr_with_fedeval_tests.py Show resolved Hide resolved

ishaileshpant reviewed

View reviewed changes

tests/end_to_end/utils/tr_workspace.py Outdated Show resolved Hide resolved

ishaileshpant approved these changes

View reviewed changes

Collaborator

ishaileshpant left a comment

Some minor changes requested - rest LGTM 👍

payalcha added 4 commits

January 24, 2025 20:15


          Merge branch 'develop' into develop

9921f18


          Merge branch 'develop' into develop

1fc884c


          Set initial model during plan initialization

2e89649

Signed-off-by: Chaurasiya, Payal <[email protected]>


          Safe yaml load

7093a22

Signed-off-by: Chaurasiya, Payal <[email protected]>

tanwarsh approved these changes

View reviewed changes

tanwarsh merged commit fd1a23d into securefederatedai:develop

22 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet