Grading Strategy to grade deductively (#18)

* strategy built in and working as was

* no longer ignoring iml file

* graded test results track if they passed instead of inspecting the score

* deductive grader strategy code and tests for it

* pulled out method to deduct

* cleaned up code again

* docs and pom update for v1.1

* wording tweak, ready to merge

.gitignore

@@ -4,7 +4,9 @@
 # JetBrains Stuff
 # decided just to ignore entirely
 .idea/
-jgrade.iml
+
+jgrade.iml  # leaving this in
+
 #.idea/**/workspace.xml
 #.idea/**/tasks.xml
 #.idea/**/usage.statistics.xml

README.md

@@ -37,6 +37,7 @@ JGrade is a helper tool with various classes designed to assist in course instru
   - `Grade`: An annotation for anything to do to grade.
   - `Grader`: An object that holds data specific to general purpose grading (a list of `GradedTestResult`s, output for the whole run, etc.)
   - `GraderObserver`: Just an observer interface to observe a `Grader` and produce output.
+  - `GraderStrategy`: Strategy interface for anything to do to modify `GradedTestResults` that were run as part of a JUnit suite. For example, this contains a `DeductiveGraderStrategy` to treat them as grading deductively from some max score down to some floor.
   - `CheckstyleGrader`: A class to assist in running checkstyle for a `GradedTestResult`.
   - `CLITester`: An abstract class to assist in running the main method of programs with a command line interface for output.
   - `CLIResult`: An interface for an object to hold multiple streams of captured output from the run.
@@ -64,6 +65,8 @@ When repackaging and making a jar that can be run as an executable, I modeled Pr
 
 ## Use
 
+<!-- TODO - Move a lot of this to Wiki -->
+
 The [examples](https://github.com/tkutche1/jgrade/tree/development/examples) directory highlights most of the core functionality. There are two ways to utilize this library.
 
 First, the client can instantiate/extend their own `Grader` (which is a public class). They can implement then attach their own observers (or just use the provided `GradescopeJsonObserver`) and produce their desired output. They can use whatever other `Grader` methods as desired.

docs/CHANGELOG.md

@@ -1,9 +1,15 @@
 # CHANGELOG
 
+Go To:
+
+**[Most Recent (1.1.0)](#1.1.0)**
+
 - [1.0.0-alpha](#1.0.0-(alpha))
 - [1.0.0](#1.0.0)
-- [1.0.1](#1.0.1)
-- [1.0.2](#1.0.2)
+  - [1.0.1](#1.0.1)
+  - [1.0.2](#1.0.2)
+  - [1.0.3](#1.0.3)
+- [1.1.0](#1.1.0)
 
 ---
 
@@ -38,10 +44,23 @@ _1.28.2019_
 - `getOutput()` without parameters defaults to returning the standard out.
 - Some (very basic) unit tests in `CLITesterExecutionResultTest` to test these tweaks and that they work as expected.
 
-### 1.0.3
+#### 1.0.3
 _1.29.2019_
 - Checkstyle compliance in all files.
 - Docs based on GitHub's community standards.
 - Maven checkstyle plugin added to object model.
 - Updated naming of jars to **not** include patch number.
 
+
+### 1.1.0
+_1.31.2019_
+- Added strategy design pattern to `Grader` so when a client calls `runJUnitGradedTests` the strategy for those test results treats them accordingly.
+  - This is the `GraderStrategy` interface class
+  - Added a private class `DefaultGraderStrategy` that does nothing so as to preserve original behavior for tests, etc.
+  - _Note this pattern should probably eventually be moved out to be a strategy for the `GradedTestListener` class rather than the `Grader` but would be a little messier work and wanted to get it working quickly to use this semester._
+- Added `DeductiveGraderStrategy` class to go through graded test results from JUnit tests and treat their scores as deductive.
+  - Constructor sets a starting score, failed tests deduct by the amount of points they are worth up to some floor (0 by default).
+  - Tracks the total points deducted.
+  - Appends a message to the test result's output if not all (or no) points were deducted so as to not go beyond the floor.
+- `DeductiveGraderStrategyTest` unit tests
+  - Added to `AllJGradeTests`

jgrade.iml

@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module version="4">
+  <component name="ExternalSystem" externalSystem="Maven" />
+  <component name="NewModuleRootManager" LANGUAGE_LEVEL="JDK_1_8">
+    <output url="file://$MODULE_DIR$/target/classes" />
+    <output-test url="file://$MODULE_DIR$/target/test-classes" />
+    <content url="file://$MODULE_DIR$">
+      <sourceFolder url="file://$MODULE_DIR$/src/main/java" isTestSource="false" />
+      <sourceFolder url="file://$MODULE_DIR$/src/test/java" isTestSource="true" />
+      <sourceFolder url="file://$MODULE_DIR$/examples/gradescope/src/main/java" isTestSource="false" />
+      <excludeFolder url="file://$MODULE_DIR$/examples/gradescope/classes" />
+      <excludeFolder url="file://$MODULE_DIR$/target" />
+    </content>
+    <orderEntry type="inheritedJdk" />
+    <orderEntry type="sourceFolder" forTests="false" />
+    <orderEntry type="library" name="Maven: junit:junit:4.12" level="project" />
+    <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-core:1.3" level="project" />
+    <orderEntry type="library" name="Maven: org.hamcrest:hamcrest-library:1.3" level="project" />
+    <orderEntry type="library" name="Maven: org.json:org.json:chargebee-1.0" level="project" />
+    <orderEntry type="library" name="Maven: commons-cli:commons-cli:1.4" level="project" />
+  </component>
+</module>

pom.xml

@@ -7,7 +7,7 @@
     <groupId>edu.jhu.cs</groupId>
     <artifactId>jgrade</artifactId>
     <packaging>jar</packaging>
-    <version>1.0.3</version>
+    <version>1.1.0</version>
 
     <name>jgrade</name>
     <url>https://github.com/tkutche1/jgrade</url>
@@ -84,7 +84,7 @@
                             <goal>single</goal>
                         </goals>
                         <configuration>
-                            <finalName>jgrade-1.0-all</finalName>
+                            <finalName>jgrade-1.1-all</finalName>
                             <descriptorRefs>
                                 <descriptorRef>jar-with-dependencies</descriptorRef>
                             </descriptorRefs>

src/main/java/com/github/tkutche1/jgrade/DeductiveGraderStrategy.java

@@ -0,0 +1,69 @@
+package com.github.tkutche1.jgrade;
+
+import com.github.tkutche1.jgrade.gradedtest.GradedTestResult;
+
+import java.util.List;
+
+/**
+ * Strategy to grade deductively. This strategy will take a point value,
+ * all tests will be worth 0, and failed tests will count negative.
+ */
+public class DeductiveGraderStrategy implements GraderStrategy {
+
+    private double floor;
+    private double startingScore;
+    private double deductedPoints;
+
+    /**
+     * Create a new DeductiveGradingStrategy.
+     * @param startingScore The score to deduct from.
+     */
+    public DeductiveGraderStrategy(double startingScore) {
+        this.startingScore = startingScore;
+        this.floor = 0;
+        this.deductedPoints = 0;
+    }
+
+    /**
+     * Set the floor to deduct to.
+     * @param floor The floor to deduct to.
+     */
+    public void setFloor(double floor) {
+        this.floor = floor;
+    }
+
+    /**
+     * Get the amount of points deducted from running this.
+     * @return The amount of points deducted.
+     */
+    public double getDeductedPoints() {
+        return this.deductedPoints;
+    }
+
+    @Override
+    public void grade(List<GradedTestResult> l) {
+        for (GradedTestResult r : l) {
+            if (!r.passed()) {
+                this.deductedPoints += this.deduct(r);
+            }
+            r.setPoints(0);
+        }
+    }
+
+    // Deduct from r, return the amount deducted
+    private double deduct(GradedTestResult r) {
+        double amountToDeduct = r.getPoints();
+        if ((this.deductedPoints + r.getPoints()) > potentialDeductions()) {
+            r.addOutput("Failed test but deductive grading did not subtract"
+                    + "points below floor");
+            amountToDeduct = potentialDeductions() - this.deductedPoints;
+        }
+        r.setScore(0 - amountToDeduct);
+        return amountToDeduct;
+    }
+
+    // Get the amount of possible points that can be deducted in total.
+    private double potentialDeductions() {
+        return this.startingScore - this.floor;
+    }
+}

src/main/java/com/github/tkutche1/jgrade/Grader.java

@@ -22,7 +22,14 @@
 public class Grader {
     private static final int NOT_SET = -1;
 
+    private class DefaultGraderStrategy implements GraderStrategy {
+        public void grade(List<GradedTestResult> l) {
+            // Nothing to do.
+        }
+    }
+
     private List<GraderObserver> observers;
+    private GraderStrategy graderStrategy;
     private List<GradedTestResult> gradedTestResults;
     private long startTime;
     private long executionTime;
@@ -36,6 +43,7 @@ public Grader() {
         this.gradedTestResults = new ArrayList<>();
         this.executionTime = NOT_SET;
         this.output = new StringBuilder();
+        this.graderStrategy = new DefaultGraderStrategy();
     }
 
     // <editor-fold desc="accessors">
@@ -179,6 +187,16 @@ public void notifyOutputObservers() {
         }
     }
 
+    /**
+     * Set the strategy to use to grade. By default, the strategy is to
+     * add {@link GradedTestResult}s as they are. To change that, you can
+     * alter them via the strategy and it's grade method.
+     * @param s The strategy to set.
+     */
+    public void setGraderStrategy(GraderStrategy s) {
+        this.graderStrategy = s;
+    }
+
     /** Starts (or resumes) the timer for the Grader. */
     public void startTimer() {
         this.startTime = System.currentTimeMillis();
@@ -202,14 +220,17 @@ public void stopTimer() throws IllegalStateException {
      * created {@link GradedTestResult}s. If class <code>MyTests</code> has
      * graded test JUnit test methods, then call this method with
      * <code>MyTests.class</code>. Similarly can use JUnit's
-     * {@link org.junit.runners.Suite}.
+     * {@link org.junit.runners.Suite}. Can alter the list of results added from the
+     * run by setting the {@link GraderStrategy}.
      * @param testSuite The class containing the tests.
      */
     public void runJUnitGradedTests(Class testSuite) {
         GradedTestListener listener = new GradedTestListener();
         JUnitCore runner = new JUnitCore();
         runner.addListener(listener);
         runner.run(testSuite);
-        this.gradedTestResults.addAll(listener.getGradedTestResults());
+        List<GradedTestResult> results = listener.getGradedTestResults();
+        this.graderStrategy.grade(results);
+        this.gradedTestResults.addAll(results);
     }
 }

src/main/java/com/github/tkutche1/jgrade/GraderStrategy.java

@@ -0,0 +1,22 @@
+package com.github.tkutche1.jgrade;
+
+import com.github.tkutche1.jgrade.gradedtest.GradedTestResult;
+
+import java.util.List;
+
+/**
+ * Interface for the strategy design pattern on top of a {@link Grader} object.
+ * If you want to change the grading strategy, for example to grade deductively
+ * instead of simply adding the tests as they are, you can call
+ * {@link Grader#setGraderStrategy(GraderStrategy)}.
+ */
+public interface GraderStrategy {
+
+    /**
+     * Do work on the list of {@link GradedTestResult}s before they get
+     * added to the overall list of tests.
+     * @param l The list of {@link GradedTestResult}s to modify (or not
+     *          in the case of the default strategy).
+     */
+    void grade(List<GradedTestResult> l);
+}

src/main/java/com/github/tkutche1/jgrade/gradedtest/GradedTestListener.java

@@ -143,6 +143,7 @@ public void testFailure(Failure failure) throws Exception {
             this.currentGradedTestResult.addOutput(failure.getMessage() == null
                     ? failure.toString() : failure.getMessage());
             numFailedGradedTests++;
+            this.currentGradedTestResult.setPassed(false);
         }
     }
 

src/main/java/com/github/tkutche1/jgrade/gradedtest/GradedTestResult.java

@@ -38,6 +38,7 @@ public class GradedTestResult {
 
     private double score;
     private StringBuilder output;
+    private boolean passed;
 
     /**
      * Create a new GradedTestResult, setting the initial score to 0.
@@ -62,6 +63,7 @@ public GradedTestResult(String name, String number, double points, String visibi
         this.visibility = visibility;
         this.score = 0;
         this.output = new StringBuilder();
+        this.passed = true;
     }
 
     // <editor-fold "desc="accessors">
@@ -86,6 +88,22 @@ public void setScore(double score) {
         this.score = score;
     }
 
+    /**
+     * Set the number of points the test is worth.
+     * @param points The number of points to set.
+     */
+    public void setPoints(double points) {
+        this.points = points;
+    }
+
+    /**
+     * Set whether or not this result passed.
+     * @param passed The value to set.
+     */
+    public void setPassed(boolean passed) {
+        this.passed = passed;
+    }
+
     /**
      * Get the name of the test.
      * @return The name of the test.
@@ -134,6 +152,16 @@ public String getOutput() {
         return this.output.toString();
     }
 
-    // <editor-fold "desc="accessors">
 
+    /**
+     * Determine if the test for this result was considered to have passed
+     * or not.
+     * @return True if the test passed (student got full credit), false
+     *         otherwise.
+     */
+    public boolean passed() {
+        return this.passed;
+    }
+
+    // <editor-fold "desc="accessors">
 }

src/test/java/com/github/tkutche1/jgrade/AllJGradeTests.java

@@ -12,5 +12,6 @@
         AllGraderTests.class,
         GradescopeJsonObserverTest.class,
         CLITesterExecutionResultTest.class,
+        DeductiveGraderStrategyTest.class,
 })
 public class AllJGradeTests { }