WIP: GenerateCoverage Experiment #718

danjujan · 2022-11-29T15:22:48Z

No description provided.

codecov · 2022-12-09T18:31:24Z

Codecov Report

Attention: 472 lines in your changes are missing coverage. Please review.

Comparison is base (3fa246b) 68.55% compared to head (6def304) 69.10%.
Report is 1 commits behind head on vara-dev.

Additional details and impacted files

@@             Coverage Diff              @@
##           vara-dev     #718      +/-   ##
============================================
+ Coverage     68.55%   69.10%   +0.54%     
============================================
  Files           338      344       +6     
  Lines         26434    28639    +2205     
============================================
+ Hits          18122    19791    +1669     
- Misses         8312     8848     +536

Files	Coverage Δ
tests/experiment/test_workload_util.py	`100.00% <100.00%> (ø)`
tests/utils/test_doc_util.py	`100.00% <ø> (ø)`
varats-core/varats/experiment/workload_util.py	`92.85% <100.00%> (+0.21%)`	⬆️
varats-core/varats/project/varats_command.py	`83.01% <100.00%> (+1.38%)`	⬆️
.../varats/provider/feature/feature_model_provider.py	`82.85% <100.00%> (ø)`
varats-core/varats/utils/git_util.py	`85.78% <100.00%> (ø)`
varats/varats/projects/c_projects/bzip2.py	`50.00% <100.00%> (ø)`
varats/varats/projects/c_projects/gzip.py	`61.40% <100.00%> (+1.40%)`	⬆️
varats/varats/projects/c_projects/picosat.py	`59.25% <100.00%> (+1.56%)`	⬆️
varats/varats/projects/c_projects/xz.py	`66.03% <100.00%> (ø)`
... and 8 more

... and 10 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

vulder

Hmm, I'm not 100% sure about the zip files for the tests. On the one side, it's good for testing, on the other, we don't know how to update/fix them when stuff breaks.
Maybe, we should at least put some doc/wording in a file that specifies how the information can be regenerated?

I hope, this does not make the tests brittel

@boehmseb what's your take on that?

Overall, the implementation already looks quite good.

tests/helper_utils.py

vulder · 2023-04-06T09:30:45Z

tests/plots/test_llvm_coverage_plot.py

+
+class TestCodeRegion(unittest.TestCase):
+
+    def test_feature_config_report_map(self):


I'm not sure what this tests, the beginning looks a bit like testing the config handling part of the configuration, which should be tested in separation and in the config module.

This only tests if ConfigCoverageReportMapping will use the right configs for diffing.

vulder · 2023-04-06T09:32:10Z

varats/varats/data/reports/llvm_coverage_report.py

@@ -0,0 +1,708 @@
+"""Code region tree and coverage report."""
+
+from __future__ import annotations


From which python version is this? As we have update to min 3.9 we maybe don't need this anymore

This is still necessary. Otherwise the following is not possible. See https://peps.python.org/pep-0563/ and https://peps.python.org/pep-0649/

class CodeRegion def from(cls, x) -> CodeRegion: ...

varats/varats/data/reports/llvm_coverage_report.py

varats/varats/experiments/vara/llvm_coverage_experiment.py

vulder · 2023-04-06T09:47:45Z

varats/varats/plots/llvm_coverage_plot.py

+        return repr(self.x)
+
+
+class RunConfig(tp.FrozenSet[tp.Tuple[str, ConfigValue]]):


Why do you need to build a config class here, is the one that parses the config format not working. If not, can we maybe adapt it?

boehmseb · 2023-04-06T10:40:22Z

Hmm, I'm not 100% sure about the zip files for the tests. On the one side, it's good for testing, on the other, we don't know how to update/fix them when stuff breaks. Maybe, we should at least put some doc/wording in a file that specifies how the information can be regenerated?
* I hope, this does not make the tests brittle
@boehmseb what's your take on that?

We kind of brought this on ourselves by introducing .zip files as reports.
The corresponding report class should make obvious why there is a .zip lying around.
So technically, this is no different to including a normal report as test input.
I understand your concern, but I'm not sure whether an additional readme of some sorts can provide additional information that is not already encoded by the test and the report class.

danjujan · 2023-04-11T12:09:24Z

Hmm, I'm not 100% sure about the zip files for the tests. On the one side, it's good for testing, on the other, we don't know how to update/fix them when stuff breaks. Maybe, we should at least put some doc/wording in a file that specifies how the information can be regenerated?
* I hope, this does not make the tests brittle
@boehmseb what's your take on that?
We kind of brought this on ourselves by introducing .zip files as reports. The corresponding report class should make obvious why there is a .zip lying around. So technically, this is no different to including a normal report as test input. I understand your concern, but I'm not sure whether an additional readme of some sorts can provide additional information that is not already encoded by the test and the report class.

As @boehmseb said. The zip files were obtained by copy pasting them from the results folder after running the GenerateCoverage experiment.

vulder · 2023-04-13T08:58:11Z

That part I know and "buy", I'm more concerned to how we can update them if needed. We should add docu to the tests how to regenerate the files, project + experiment.
But what also concerns me, it looks like some of the paths where edited by had, which could introduce brittleness as this rewriting is not done automatically. I would be fine with the zips if we have a small description in the test files that tells a developer how to regenerate these files, because from what I see, it's a bit more difficult than "just" running the experiment.

By the way, as soon as we have a "docu format" or idea how to specify these, we should probably also document the other ones.

danjujan · 2023-04-19T14:24:14Z

That part I know and "buy", I'm more concerned to how we can update them if needed. We should add docu to the tests how to regenerate the files, project + experiment. But what also concerns me, it looks like some of the paths where edited by had, which could introduce brittleness as this rewriting is not done automatically. I would be fine with the zips if we have a small description in the test files that tells a developer how to regenerate these files, because from what I see, it's a bit more difficult than "just" running the experiment.

By the way, as soon as we have a "docu format" or idea how to specify these, we should probably also document the other ones.

I regenerated the files and noted down my steps. https://github.com/se-sic/VaRA-Tool-Suite/blob/fba20d72770514086ea65262a491378c5ea11963/tests/TEST_INPUTS/results/FeaturePerfCSCollection/HowToReproduce.md. Feedback welcome.

…ow) or similar lines in plots,

…notated, once with profiling instructions.

…zip2, gzip, picosat, xz

…ature Model

…uced xz casestudy

…extSensitivity

danjujan requested a review from vulder January 11, 2023 09:11

danjujan marked this pull request as ready for review April 5, 2023 09:04

vulder requested changes Apr 6, 2023

View reviewed changes

danjujan mentioned this pull request May 2, 2023

Hashable Configurations #789

Merged

danjujan and others added 21 commits June 1, 2023 17:30

Diff only available features in plot. Removes unnecessary -(header^sl…

9b9ab74

…ow) or similar lines in plots,

Coverage Experiemnt: Compile twice once with VaRAs feature regions an…

52595e5

…notated, once with profiling instructions.

Add test for correctly annotating vara_instrs to code regions

4652f8f

Working threshold in confusion matrix

c0fd405

Merge branch 'vara-dev' into jan-dev

847f7ed

Fix annotating empty "" feature in norm instr.

821ae1f

Merge branch 'vara-dev' into jan-dev

ae51164

Presence Condition seem to work. Cleanup still required

2fb21b1

Presence Conditions work. No diffing/merging needed for them

277130d

Fix mypy + pylint errors

d1d6272

Merge branch 'vara-dev' into jan-dev

d13221a

Fix mypy errors

70fbf58

Fix docs generation failing due to missing pyeda import

fdc8685

Merge branch 'vara-dev' into jan-dev

7b59bf3

Implements classification evaluation helper

36cc057

Updates naming and error handling

be4624f

Renaming

a0ed8cb

Removes unnecessary import

ac3a3d5

Add pyeda dependency for container

2b14c3f

Set container image for FeaturePerfCSCollections

feb5129

Start removing merging + diffing code

ae877f4

danjujan added 30 commits October 20, 2023 10:40

Switch to gzip case study to reduced

f77e449

Add bzip2 reduced case study

ceb25c5

Added specificity to tables + added ignore parsing code regions for b…

b0f2661

…zip2, gzip, picosat, xz

Restore workarounds

f2bfb37

Regenerater case studies after vara-feature fixed

902ad51

Dont fail if we use --stdout args

ded06e2

Update xz configs after ensuring --compress is always generated by Fe…

75cc5df

…ature Model

Merge branch 'vara-dev' into jan-dev

70ccd19

Fix import in experiment

06a4311

Fix mypy + pylint

6488b0d

Fix pre-commit

cac8c82

Fix broken merge

c4f0f15

Add SynthIPRuntime support

d48dc46

Add workload.small again to fix unittest

3ee8614

Fix tests + ignore feature-dependent functions in SynthIPRuntime

d544bde

Add SynthDARecursion support

f779dfb

Add SynthDADynamicDispatch, SynthOVInsideLoop

006fe4b

Add SynthSA support

ac767cf

Fix pylint

cd00040

Plot instrs for FNs and FPs in confusion matrices

0f3aad0

Fix mypy

2cc3195

Get rid of duplicates in feature_combinations generation + Update red…

716e2d5

…uced xz casestudy

Get rid of duplicates in feature_model_formula

5ed9157

Use ConfigParams in more projects

550f1f5

Reduce duplicates in gzip case study

f35bbbe

Get SynthDADynamicDispatch working

0438383

Use FeatureModel when generating matrices...

a2d13e3

Add alternative options workaround for SynthIPRuntime and SynthSACont…

117bede

…extSensitivity

Add excluded options workaround for bzip2

12f1d09

Add excluded options workaround for xz

6def304

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: GenerateCoverage Experiment #718

WIP: GenerateCoverage Experiment #718

danjujan commented Nov 29, 2022

codecov bot commented Dec 9, 2022 •

edited

Loading

vulder left a comment

vulder Apr 6, 2023

danjujan Apr 11, 2023

vulder Apr 6, 2023

danjujan Apr 11, 2023

vulder Apr 6, 2023

boehmseb commented Apr 6, 2023

danjujan commented Apr 11, 2023

vulder commented Apr 13, 2023

danjujan commented Apr 19, 2023


		class TestCodeRegion(unittest.TestCase):

		def test_feature_config_report_map(self):

		@@ -0,0 +1,708 @@
		"""Code region tree and coverage report."""

		from __future__ import annotations

		return repr(self.x)


		class RunConfig(tp.FrozenSet[tp.Tuple[str, ConfigValue]]):

WIP: GenerateCoverage Experiment #718

Are you sure you want to change the base?

WIP: GenerateCoverage Experiment #718

Conversation

danjujan commented Nov 29, 2022

codecov bot commented Dec 9, 2022 • edited Loading

Codecov Report

vulder left a comment

Choose a reason for hiding this comment

vulder Apr 6, 2023

Choose a reason for hiding this comment

danjujan Apr 11, 2023

Choose a reason for hiding this comment

vulder Apr 6, 2023

Choose a reason for hiding this comment

danjujan Apr 11, 2023

Choose a reason for hiding this comment

vulder Apr 6, 2023

Choose a reason for hiding this comment

boehmseb commented Apr 6, 2023

danjujan commented Apr 11, 2023

vulder commented Apr 13, 2023

danjujan commented Apr 19, 2023

codecov bot commented Dec 9, 2022 •

edited

Loading