diff --git a/README.md b/README.md index ba70171..21a8519 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,107 @@ -# dd_java +## SIVAND: Prediction-Preserving Program Simplification + +This repository contains the code of prediction-preserving simplification and the simplified data using DD module for our paper '[Understanding Neural Code Intelligence Through Program Simplification](https://github.com/mdrafiqulrabin/SIVAND/)' accepted at ESEC/FSE'21. + +--- + +## Structure + +``` +├── ./ # code for model-agnostic DD framework +├── data/ + ├── selected_input # randomly selected test inputs from different datasets + ├── simplified_input # traces of simplified inputs for different models + ├── summary_result # summary results of all experiments as csv +├── models/ + ├── dd-code2seq # DD module with code2seq model + ├── dd-code2vec # DD module with code2vec model + ├── dd-great # DD module with RNN/Transformer model +├── others/ # related helper functions +├── save/ # images of SIVAND +``` + +--- + +## Workflow + +|Workflow in SIVAND| +:-------------------------: + +[Delta Debugging (DD)](https://www.st.cs.uni-saarland.de/dd/) was implemented with Python 2. We have modified the core modules ([DD.py](https://www.st.cs.uni-saarland.de/dd/DD.py), [MyDD.py](https://www.st.cs.uni-saarland.de/dd/MyDD.py)) to run in [Python 3](https://github.com/mdrafiqulrabin/dd-py3), and then adopted the DD modules for prediction-preserving program simplification using different models. The approach, SIVAND, is model-agnostic and can be applied to any model, but by loading a model and making a prediction with the model for a task. + +**How to Start**: To apply SIVAND (for MethodName task as an example), first update `` (path to a file that contains all selected inputs) and `` (select token or char type delta for DD) in `helper.py`. Then, write your code into `load_model_M()` to load a target model from ``, and into `prediction_with_M()` to get the predicted name, score, and loss value with `` for an input ``. Also, load method by language (i.e. Java) from `load_method()` and check whether `` is parsable into `is_parsable()`. Finally, run `MyDD.py` that will simplify programs one by one and save all simplified traces in the `dd_data/` folder. + +**Check More**: For more detailed code, check `models/dd-code2vec/` and `models/dd-code2seq/` folders to see how SIVAND works with code2vec and code2seq models for MethodName task on Java program. Similarly, for VarMisuse task (RNN & Transformer models, Python program), check the `models/dd-great/` folder for our modified code. + +--- + +## Motivating Example + +|Motivating Example|Traces of Reduction| +:-------------------------:|:-------------------------: +|Example of an original and minimized method in which the target is to predict `onCreate`.| Reduction of a program while preserving the predicted method name `OnCreate` by the code2vec model.| + +The minimized example clearly shows that the model has learned to take shortcuts, in this case looking for the name in the function's body. + +--- + +## Experimental Settings + + * Tasks: + * [MethodName (MN)](https://arxiv.org/abs/1602.03001) + * [VarMisuse (VM)](https://arxiv.org/abs/1904.01720) + + * Models: + * [MN] [code2vec](https://github.com/tech-srl/code2vec/) & [code2seq](https://github.com/tech-srl/code2seq/) + * [VM] [RNN & Transformer](https://github.com/VHellendoorn/ICLR20-Great) + + * Datasets: + * [MN] [Java-Large](https://github.com/tech-srl/code2seq/#java) + * [VM] [Py150](https://github.com/google-research-datasets/great) + + * Sample Inputs: + * [MN] Correctly predicted samples, Wrongly predicted samples + * [VM] Buggy (correct location and target; wrong location), Non-buggy (bug-free) + + * Delta Types: + * [MN] Token & Char + * [VM] Token + --- -Steps: -- Path of test java files, i.e. test_file = 'data/path.txt'. -- Code for load/predict model (i.e. load_model_hc33/prediction_with_hc33) in "helper.py". -- Run "MyDD.py", output (simplified codes) will be saved as "data/simplified/method_n.txt". +## Results + +The `data/summary_result/` folder contains summary results of all experiments as csv, each file has the following fields: + +* `filename`: ID for the input file of `data/simplified_input` folder +* `model`: {code2vec, code2seq, RNN, or Transformer} +* `task`: METHOD_NAME or VARIABLE_MISUSE +* `filter_type`: + * {token_correct, char_correct or token_wrong} for task == METHOD_NAME + * {buggy_correct, non_buggy_correct, or buggy_wrong_location} for task == VARIABLE_MISUSE +* `initial_score`: score of actual program +* `final_score`: score of minimal program +* `initial_loss`: loss of actual program +* `final_loss`: loss of minimal program +* `dd_pass`: total/valid/correct DD stepss for reduction +* `dd_time`: total time spent for reduction +* `initial_program`: actual raw program +* `final_program`: minimal simplified program +* `initial_tokens`: tokens in actual program +* `final_tokens`: tokens in minimal program +* `len_{initial/final/minimal}_{tokens/chars}`: number of corresponding {tokens/chars} +* `per_removed_{chars/tokens}`: percentage of removed {chars/tokens} +* `attn_nodes`: top-k AST nodes based on attention score {k ~= len_final_nodes} +* `final_nodes`: all AST nodes in minimal program +* `common_nodes`: common nodes between attention & reduction +* `len_{attn/final/common}_nodes`: number of corresponding AST nodes +* `per_common_tokens`: percentage of common nodes between attention & reduction +* `ground_truth`: True (for correct prediction) or False (for wrong prediction) + +Note that the ``, `-1`, or `` value represents that the value is not available for that particular input/experiment. + + +|Summary of Results| +:-------------------------: +|Summary of reduction results in correctly predicted samples.| + diff --git a/save/example.png b/save/example.png new file mode 100644 index 0000000..d2371e4 Binary files /dev/null and b/save/example.png differ diff --git a/save/sivand.png b/save/sivand.png new file mode 100644 index 0000000..abe5021 Binary files /dev/null and b/save/sivand.png differ diff --git a/save/summary.png b/save/summary.png new file mode 100644 index 0000000..682a9e7 Binary files /dev/null and b/save/summary.png differ diff --git a/save/traces.png b/save/traces.png new file mode 100644 index 0000000..64f711e Binary files /dev/null and b/save/traces.png differ