Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reworks the table to group together similar categories #873

Merged
merged 255 commits into from
Feb 20, 2024
Merged
Changes from all commits
Commits
Show all changes
255 commits
Select commit Hold shift + click to select a range
61cfa54
* Finished first tests for revision renges for patches
Jun 27, 2023
bba044a
* Fixed tests to explicitly copy all revisions
Jun 27, 2023
12168fa
* Added handling for creating project specific providers
Jun 28, 2023
4fae498
* Added test for getting project specific patch provider
Jun 28, 2023
8e74338
* Added test for getting a patch by short name
Jun 28, 2023
8a8ae4b
* Added general conceptual outline for running experiments with patches
Jun 28, 2023
25eda17
* Added actions for patch apply/revert
Jun 29, 2023
ae36e82
* Restructured patch infrastructure as separate experiment
Jul 3, 2023
074f4f2
* Added patch support to JustCompile Project
Jul 3, 2023
e5d6e5f
* Added proper path handling to PatchProvider
Jul 3, 2023
7ac995d
* Small prototype for just compile project
Jul 3, 2023
34141bd
* Added proper descriptions to analysis actions
Jul 4, 2023
0cf1e90
* Updated patch integration
Jul 4, 2023
b8655a6
* Formatting and linting
Jul 4, 2023
a8d84c8
Merge branch 'vara-dev' into f-PatchFiles
Jul 4, 2023
8999d3b
Formatting
Jul 4, 2023
4854e79
Adapts FeaturePerfCSCollection to new setup process
vulder Jul 6, 2023
a6cf7d7
Adds Synthetic case studies as projects
vulder Jul 6, 2023
2e1f720
Refactor helper method into function
vulder Jul 6, 2023
208d30a
Inital work on implementing eval table
vulder Jul 7, 2023
4947187
Implements profiler split
vulder Jul 7, 2023
f4428bd
Adapts pre-commit config to new version
vulder Jul 7, 2023
44bb430
Work on new experiment
vulder Jul 7, 2023
ef52e4e
First draft of a experiment with old/new + patching
vulder Jul 7, 2023
f40223d
First working pipeline
vulder Jul 7, 2023
9bed01e
Adds black box GT computation
vulder Jul 8, 2023
db600bf
Adapt wxray measurement to repeat and adds significatc test
vulder Jul 8, 2023
2cd4cbb
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Jul 8, 2023
b2db084
Adds extra func. to latex table rendering
vulder Jul 9, 2023
da7a1f3
Polish latex for precision table
vulder Jul 9, 2023
75212eb
Adds overhead measurement experiments with eval table
vulder Jul 9, 2023
d15918a
Implements experiments and table support for PIM tracer
vulder Jul 9, 2023
39f604e
Fixes latex table gen for multiple profilers
vulder Jul 9, 2023
05e2d1a
Refactors classification calculation into separate helper file
vulder Jul 9, 2023
14a0697
Removes unnecessary braces
vulder Jul 10, 2023
c77dd9e
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Jul 10, 2023
cd0a75a
Adds total to precision table
vulder Jul 11, 2023
9c0e1ae
Adds docs to classification results
vulder Jul 11, 2023
feee86d
Adds test for precision score
vulder Jul 11, 2023
236982b
Separated XML tests to yaml format
Jul 13, 2023
aa68405
Added yaml parsing for patch information
Jul 13, 2023
b4fee57
Revert irrelevant changes in JustCompile Project
Jul 15, 2023
9751a5e
Changed PatchProvider to initialize itself from multiple info files f…
Jul 15, 2023
a7776f3
Added patch accessors to Patch provider
Jul 15, 2023
49770c4
Removed obsolete ProjectPatchesConfiguration
Jul 15, 2023
89705c0
remove unnecessary functions
Jul 15, 2023
d8975c5
Fixed unit tests for new yaml based structure
Jul 15, 2023
4a35b28
Removed old test XML
Jul 15, 2023
062a29a
Basis for PatchSet class
Jul 15, 2023
39fd77c
Changed get_patches_for_revision to return a PatchSet
Jul 15, 2023
c2be0af
Small fixes to get new synth case studies working
vulder Jul 16, 2023
09fb392
Merge branch 'f-PatchFiles' into f-StaticAnalysisMotivatedSynthBenchm…
vulder Jul 16, 2023
fec0175
Small adaptions
vulder Jul 16, 2023
63ede6b
Roll back exp util changes
vulder Jul 16, 2023
514e253
Fixes formatting in tests
vulder Jul 16, 2023
f4a5689
Get patch provider working
vulder Jul 16, 2023
efd60a6
Adapt profiler step creator
vulder Jul 16, 2023
afdba46
Adds revert support
vulder Jul 16, 2023
b407c7b
Implements multi patch handling
vulder Jul 17, 2023
dbb04ac
Move recompile to own file
vulder Jul 17, 2023
47281da
Reworks output folder steps into own base class
vulder Jul 17, 2023
eb6479e
Adapts code to new output folder steps
vulder Jul 17, 2023
e80a707
Implements plot and refactors data handling
vulder Jul 19, 2023
50e9fde
Fixes example plotting and pareto bug
vulder Jul 19, 2023
10e773c
Plot polishing
vulder Jul 20, 2023
f93fc73
More plot RQ2 tweaks
vulder Jul 20, 2023
2d3b3d1
Strange pareto plot with a lot of extras :D
vulder Jul 20, 2023
8a5b6a4
Refactor f1_score plotting
vulder Jul 22, 2023
2b9c6b0
Moves precision calc of plots to database
vulder Jul 22, 2023
4003e60
Implements correct file loading for overhead data
vulder Jul 22, 2023
7669a09
Removes "stuff"
vulder Jul 22, 2023
65c368b
Removes debug prints and updates error messages
vulder Jul 22, 2023
163cac9
Convert to multiplot
vulder Jul 22, 2023
05b02ed
Fixes typing issue
vulder Jul 23, 2023
59b6308
Fixes pareto for equal x-values
vulder Jul 23, 2023
3843b59
Implements bpftrace experiment
vulder Jul 23, 2023
4db2211
Test experiment for BCC
vulder Jul 23, 2023
dd1c8a6
Implements basic perf report
vulder Jul 23, 2023
3fa09b5
Ports overhead measurements to perf for more metrics
vulder Jul 23, 2023
8d4c384
Removes debug print
vulder Jul 23, 2023
0ada9d8
Adapts remote refs
vulder Jul 24, 2023
570713a
Fixes commit lookup
vulder Jul 24, 2023
32dc069
Correctly sets provider remote
vulder Jul 24, 2023
ec079f5
Removes comments
vulder Jul 24, 2023
f609f07
Fixes packaging tool
vulder Jul 24, 2023
7d4ed1e
Adds experiment to bb config gen
vulder Jul 25, 2023
ae7d67a
Fixes packaging tool to work with config ids
vulder Jul 25, 2023
cfccc47
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Jul 26, 2023
3beca52
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Aug 22, 2023
4bad911
Cleans up patch provider
vulder Aug 24, 2023
16f7fe5
Updates code to merged changes
vulder Aug 24, 2023
cc8955f
More clean up for patch provider
vulder Aug 24, 2023
649449a
Fixes PIM computation
vulder Aug 31, 2023
f64598a
Fixes multi step error tracking
vulder Aug 31, 2023
b1a26d1
Fixes missing folder creation
vulder Aug 31, 2023
d53aecf
Adapts interfaces
vulder Aug 31, 2023
a7cfb7c
Fixes renaming
vulder Aug 31, 2023
a22a17b
Implements special handling for ebpftrace to correctly handle nfs shares
vulder Sep 2, 2023
f0c9295
Fixes packaging bug with multiple exp that have the same report type
vulder Sep 2, 2023
a855c75
Fixes base path for nfs fixes
vulder Sep 2, 2023
fcb04c3
Changes perf_cs collection github ref to https
vulder Sep 2, 2023
8e646bc
Adds more test inputs
vulder Sep 2, 2023
f8ad750
Enbales cli-based debug support for running containers
vulder Sep 4, 2023
e2a68b8
Ensure that BB config is loaded before calling BB
vulder Sep 4, 2023
e6ac56c
Adds container specifiers to perf-cs collection
vulder Sep 4, 2023
5368cb7
Introduces debian12 containers
vulder Sep 4, 2023
e40ca2f
Encode FeatureExperiment cluster requirements
vulder Sep 4, 2023
9cfd84c
Add container desc to experiments
vulder Sep 4, 2023
95fce58
Prevent loading bug
vulder Sep 4, 2023
001f8c9
Migrate from perf to time
vulder Sep 4, 2023
c66d15e
Report project name in error
vulder Sep 4, 2023
fdbaf89
Extend time report parsing
vulder Sep 4, 2023
80c70ca
Extend overhead overview plot by more metrics
vulder Sep 4, 2023
a4f6f9f
Fixes container decl for experiments
vulder Sep 5, 2023
2ac1036
Implements basic subject system overview table
vulder Sep 7, 2023
986fef9
Adds fp/fn ids to precision table output
vulder Sep 7, 2023
3e27539
Adds placeholder dist plot impl
vulder Sep 7, 2023
924a0db
* First skeleton for HyTeg project
Sep 7, 2023
4ddb9a5
Implements profiler comp with distributations
vulder Sep 7, 2023
52319e1
Adds exception handling for some reports that currently don't parse
vulder Sep 7, 2023
93d782b
Polish dist plot
vulder Sep 14, 2023
3382450
* Added Profiling Binary to HyTeg
Sep 15, 2023
cbb9582
Adapt overhead plot for paper
vulder Sep 15, 2023
b1f3ebc
Adds error tracking for incomplete reports (+ preciser region tracking)
vulder Sep 19, 2023
99982bb
Adds checking for pid/tid to ensure we close the right measurement
vulder Sep 24, 2023
68fb8c0
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Sep 24, 2023
b953cf5
Merge branch 'vara-dev' into f-hyteg-project
Sep 26, 2023
3a6c5d3
Implements alternative comparsion techniques
vulder Sep 26, 2023
3ea74fe
* Added cflags to disable phasar for hyteg
Sep 26, 2023
fe03800
Adds extra option call to vara xray runner
vulder Sep 28, 2023
4de2e50
Adds result table
vulder Sep 29, 2023
52fbf56
* Disabled MPI build for HyTeG/Walberla
Oct 4, 2023
a0ba845
Merge branch 'vara-dev' into f-hyteg-project
Oct 4, 2023
85c986c
* Disabled Doc build for HyTeG
Oct 4, 2023
66ae93b
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Oct 5, 2023
bbceb1a
Adds patch tags and workload for Runtime
vulder Oct 5, 2023
3b3e300
Fixes workload file selection for SynthIP* case studies
vulder Oct 5, 2023
4bb973b
Rework workload selection
vulder Oct 5, 2023
9ccd2cf
Rework new patch selection
vulder Oct 5, 2023
083c66b
Prepare integration of template bases systems
vulder Oct 5, 2023
47197e7
Adds missing init py
vulder Oct 5, 2023
968d069
Removes debug line
vulder Oct 5, 2023
ef967ef
* Add HyTeG Workload
Oct 6, 2023
4d1f22d
Try fixing bzip2
vulder Oct 6, 2023
1d8e2ac
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Oct 6, 2023
4fde0f1
A few tweaks for dune
vulder Oct 6, 2023
e4171e5
Adds token for runtime config params for workload commands.
boehmseb Oct 7, 2023
070db2e
Fix typing
boehmseb Oct 7, 2023
0dce960
Merge
vulder Oct 7, 2023
4b75cd5
Allow default values for ConfigParams()
boehmseb Oct 8, 2023
9399aa2
Refactor workload filtering
boehmseb Oct 8, 2023
bcd97e6
Move get_config* functions to utils.config
boehmseb Oct 8, 2023
c51899b
Fix tests
boehmseb Oct 8, 2023
c5c36e4
Merge branch 'f-RuntimeConfigOptionsToken' into f-StaticAnalysisMotiv…
vulder Oct 8, 2023
8d3bba4
Updates config handling to new VCommand API
vulder Oct 8, 2023
039144f
Work on bzip2
vulder Oct 8, 2023
7295486
Adds stdin/stdout redirection to VCommand.
boehmseb Oct 9, 2023
7505811
Fix test
boehmseb Oct 9, 2023
61aab0c
Merge branch 'f-RuntimeConfigOptionsToken' into f-StaticAnalysisMotiv…
vulder Oct 9, 2023
42afc5e
Convert paths to strings in stream redirection
boehmseb Oct 9, 2023
ff06107
Merge branch 'f-RuntimeConfigOptionsToken' into f-StaticAnalysisMotiv…
vulder Oct 9, 2023
83a4ac8
Add VCommand wrapper function
boehmseb Oct 9, 2023
d48ec5e
More von Sebi Runtime example
vulder Oct 9, 2023
b4596de
Adds hints for dune
vulder Oct 9, 2023
f7be54e
Adapts workloads for Sebi CS
vulder Oct 9, 2023
ab0d13b
A few dune tweaks
vulder Oct 9, 2023
8d8d203
Adapt dune to VCommand and disable MPI
vulder Oct 9, 2023
3fd73e7
Larger ring buffer
vulder Oct 9, 2023
72328aa
Adapts bpftrace exp. to new wrapper interface
vulder Oct 10, 2023
929f0ee
Adapts bpftrace handling
vulder Oct 10, 2023
7deddd9
Fixes parsing errors in json
vulder Oct 10, 2023
5943e76
Fixes print
vulder Oct 10, 2023
bf17bca
Update workloads after reg. fix
vulder Oct 10, 2023
6d36d3e
Generalizes workloads for IP projects
vulder Oct 10, 2023
407aa08
Refactor workload handling for SynthIPs
vulder Oct 10, 2023
d117005
Disable warnings
vulder Oct 10, 2023
de4cf20
Fixes wrong sourceroots
vulder Oct 10, 2023
cf89b11
Adds field sense
vulder Oct 10, 2023
ec4b87a
Fixes SynthIP bug
vulder Oct 11, 2023
9d03619
Adds special handling for LocalCommands
vulder Oct 11, 2023
84eaf5a
Merge branch 'vara-dev' into f-hyteg-project
Oct 11, 2023
932b02b
- Added suggestions from code review
Oct 11, 2023
9c98e59
- Added suggestions from code review
Oct 11, 2023
ebd6767
- Pylint and MyPy
Oct 11, 2023
24655c6
Adapt instr. threshold for sebi CS
vulder Oct 12, 2023
4ae03cf
Merge branch 'vara-dev' into f-hyteg-project
vulder Oct 12, 2023
6909287
Fixes threshold checking problem
vulder Oct 12, 2023
dea50ee
Implements loc calculation for dune
vulder Oct 12, 2023
8ad5975
Small fixes and error reduction
vulder Oct 17, 2023
f569a36
Hyteg fixes
Oct 17, 2023
80fc3d0
Adapts instr threshold
vulder Oct 18, 2023
359c5df
Adapts workload selection
vulder Oct 18, 2023
b174027
Merge commit '593ab64c5fb760f21525ab831437a0ef8cc37728' into f-Static…
vulder Oct 18, 2023
88cacca
Introduces new CS
vulder Oct 18, 2023
78257ea
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Oct 18, 2023
1a773ca
Fixes Synth commit ranges
vulder Oct 18, 2023
627bcad
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Oct 18, 2023
3da6c13
Fixes command
vulder Oct 18, 2023
d2f6b5b
Straiten out checking
vulder Oct 20, 2023
4c1a1bf
Merge branch 'f-hyteg-project' into f-StaticAnalysisMotivatedSynthBen…
vulder Oct 20, 2023
8fab50f
Adapts hyteg setup
vulder Oct 23, 2023
dfabb66
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Oct 23, 2023
f581984
Disable phasar for hyteg
vulder Oct 25, 2023
5669bd1
Checkout submodules for HyTeg
vulder Oct 25, 2023
ff7f303
Apply suggestions from code review
vulder Oct 25, 2023
1108d50
Merge branch 'vara-dev' into f-PatchHyteg
vulder Oct 25, 2023
d0c92c1
Merge branch 'vara-dev' into f-PatchHyteg
vulder Oct 29, 2023
b303116
- Adapt HyTeg to be able to compile with external Eigen directory set…
Oct 30, 2023
775a8fd
- Added information on special handling with HyTeg and configurations
Oct 30, 2023
930b870
- Pre-commit
Oct 30, 2023
34ede35
Apply suggestions from code review
vulder Oct 31, 2023
f483841
Merge branch 'f-PatchHyteg' into f-StaticAnalysisMotivatedSynthBenchm…
vulder Nov 2, 2023
dab6199
Switch to new disable compiler flag
vulder Nov 14, 2023
0c9df7e
Adds synthetic feature projects
boehmseb Nov 14, 2023
5d3b8ca
Fixes feature interaction name cleaning for pim traces
vulder Nov 14, 2023
9f3a912
Catch exceptions and report errors instead of failing
vulder Nov 14, 2023
a28b3fb
Adds raw memory stats to table
vulder Nov 14, 2023
57a8c1d
Better debug printing
vulder Nov 14, 2023
cd38950
Report parse errors
vulder Nov 14, 2023
00bd1c2
Merge remote-tracking branch 'origin/f-FeatureCaseStudies' into f-Sta…
vulder Nov 14, 2023
3620593
Adds workloads to new CSs
vulder Nov 14, 2023
c897595
Min reg on BB side + cut off for regressions
vulder Nov 16, 2023
28ac7b6
Instrument all HyTeg regions
vulder Nov 16, 2023
438e8fd
Final fixes for plots and tables
vulder Nov 27, 2023
de76d14
Clean up left over code
vulder Nov 27, 2023
530d5e1
Fixes print
vulder Nov 27, 2023
6b48c93
Bump matplotlib and seaborn versions
vulder Nov 27, 2023
5e2efb1
Cleans up tef report changes
vulder Nov 27, 2023
fa1dcb5
More clean up
vulder Nov 27, 2023
a3354d4
More clean up
vulder Nov 27, 2023
13a2270
More clean up
vulder Nov 27, 2023
1c604cb
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Nov 27, 2023
7fd6936
Cleans up more mypy/pylint issues
vulder Nov 27, 2023
1d3f2b1
More fixes
vulder Nov 27, 2023
97283c9
Apply suggestions from code review
vulder Nov 27, 2023
27ac0b6
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Nov 27, 2023
afe6f04
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
vulder Nov 29, 2023
bb90d66
Refactor baseline regression check into profiler
vulder Dec 14, 2023
1848abc
Refactors regression checking to unify baseline and statistical checking
vulder Jan 23, 2024
78d7a3d
More clean up
vulder Jan 23, 2024
d054c40
Remove wrong disable
vulder Jan 23, 2024
1a5e295
Merge branch 'vara-dev' into f-StaticAnalysisMotivatedSynthBenchmarks…
boehmseb Jan 25, 2024
f2659b9
Apply suggestions from code review
vulder Jan 25, 2024
7350cb0
Clean up from feedback
vulder Jan 25, 2024
174490c
Mypy "rework"
vulder Jan 25, 2024
b64fcdf
Group synthetics into categories
vulder Jan 26, 2024
7ce8ace
Refactors cat grouping to include stats table
vulder Jan 28, 2024
e40070e
Merge branch 'vara-dev' into f-GroupedTable
vulder Jan 30, 2024
194b228
Fixes pylint warning
vulder Jan 30, 2024
7e0bad1
Merge branch 'vara-dev' into f-GroupedTable
vulder Feb 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 64 additions & 1 deletion varats/varats/tables/feature_perf_precision.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,33 @@
from varats.table.tables import TableFormat, TableGenerator
from varats.utils.git_util import calc_repo_loc, ChurnConfig

GROUP_SYNTHETIC_CATEGORIES = True
LuAbelt marked this conversation as resolved.
Show resolved Hide resolved

SYNTH_CATEGORIES = [
"Static Analysis", "Dynamic Analysis", "Configurability",
"Implementation Pattern"
]


def compute_cs_category_grouping(case_study_name: str) -> str:
"""Mapping function to transform individual project names to their synthtic
categories."""
if case_study_name.startswith("SynthSA"):
return "Static Analysis"

if case_study_name.startswith("SynthDA"
) or case_study_name.startswith("SynthOV"):
return "Dynamic Analysis"

if case_study_name.startswith("SynthFeature"):
return "Configurability"

if case_study_name.startswith("SynthCT"
) or case_study_name.startswith("SynthIP"):
return "Implementation Pattern"

return case_study_name


def cmap_map(
function: tp.Callable[[npt.NDArray[np.float64]], npt.NDArray[np.float64]],
Expand Down Expand Up @@ -304,6 +331,21 @@ def tabulate(self, table_format: TableFormat, wrap_table: bool) -> str:
precision_df, overhead_df, on=["CaseStudy", "Profiler"]
)

if GROUP_SYNTHETIC_CATEGORIES:

merged_df["CaseStudy"] = merged_df["CaseStudy"].apply(
compute_cs_category_grouping
)
merged_df = merged_df.groupby(['CaseStudy', "Profiler"],
as_index=False).agg({
'precision': 'mean',
'recall': 'mean',
'overhead_time': 'mean',
'overhead_time_rel': 'mean',
'overhead_memory_rel': 'mean',
'overhead_memory': 'mean'
})

pivot_df = merged_df.pivot(
index='CaseStudy',
columns='Profiler',
Expand All @@ -324,7 +366,18 @@ def tabulate(self, table_format: TableFormat, wrap_table: bool) -> str:
],
axis=1)

pivot_df.loc["Total"] = pivot_df.mean()
# All means need to be computed before they are added as rows
overall_mean = pivot_df.mean()
if GROUP_SYNTHETIC_CATEGORIES:
synth_mean = pivot_df.loc[pivot_df.index.isin(SYNTH_CATEGORIES)
].mean()
real_world_mean = pivot_df.loc[~pivot_df.index.
isin(SYNTH_CATEGORIES)].mean()

pivot_df.loc["SynthMean"] = synth_mean
pivot_df.loc["RealWorldMean"] = real_world_mean

pivot_df.loc["OverallMean"] = overall_mean

# Rename columns
# pylint: disable=anomalous-backslash-in-string
Expand Down Expand Up @@ -516,6 +569,16 @@ def tabulate(self, table_format: TableFormat, wrap_table: bool) -> str:
cs_data.append(pd.DataFrame.from_dict(cs_dict, orient='index'))

df = pd.concat(cs_data).sort_index()
df.index.name = 'CaseStudy'

if GROUP_SYNTHETIC_CATEGORIES:
df.index = df.index.map(compute_cs_category_grouping)

df = df.groupby(df.index.name, as_index=True).agg({
'NumConfig': 'sum',
'Locs': 'sum',
'Regressions': 'sum'
})

style = df.style
kwargs: tp.Dict[str, tp.Any] = {}
Expand Down
Loading