#105: Refactoring Measurement, from Agent side #104

lsulak · 2023-11-01T11:56:19Z

Addressing Measurement and MeasurementResult - I tried to make it much more simpler and added some validations even. Now Measure contains only 2 attributes; the result itself is separated.

I improved the MeasurementBuilder and now the Measures and Measurements are Set but with some additional validation on uniqueness - i.e. a user can't define the same Measures against the same columns with different result values - it's not a valid case.

Closes #105

github-actions · 2023-11-01T11:59:25Z

JaCoCo agent module code coverage report - spark:2 - scala 2.12.12

Build Failed

github-actions · 2023-11-01T11:59:26Z

JaCoCo server module code coverage report - scala 2.12.12

Build Failed

… order there is not important

TebaleloS

This looks good. The only problem is that it cannot compile successfully, it has a type mismatch on found : scala.collection.immutable.Set[String] [error] required: Seq[Any] [error] val measureNames = scalaIterableToSQLArray(values.measurements.map(_.measure.measureName))

lsulak · 2023-11-09T08:52:54Z

This looks good. The only problem is that it cannot compile successfully, it has a type mismatch on found : scala.collection.immutable.Set[String] [error] required: Seq[Any] [error] val measureNames = scalaIterableToSQLArray(values.measurements.map(_.measure.measureName))

Fixed, thanks, it's already in master now

github-actions · 2023-11-10T13:58:02Z

JaCoCo agent module code coverage report - spark:2 - scala 2.12.18

Build Failed

github-actions · 2023-11-10T13:58:04Z

JaCoCo server module code coverage report - scala 2.12.18

Build Failed

…-refactoring # Conflicts: # agent/src/main/scala/za/co/absa/atum/agent/AtumContext.scala # agent/src/main/scala/za/co/absa/atum/agent/model/Measure.scala # agent/src/main/scala/za/co/absa/atum/agent/model/MeasurementBuilder.scala # model/src/main/scala/za/co/absa/atum/model/dto/CheckpointDTO.scala

…-refactoring # Conflicts: # agent/src/main/scala/za/co/absa/atum/agent/AtumContext.scala # agent/src/main/scala/za/co/absa/atum/agent/exception/AtumAgentException.scala # agent/src/main/scala/za/co/absa/atum/agent/model/Measurement.scala # agent/src/test/scala/za/co/absa/atum/agent/model/MeasurementTest.scala # model/src/main/scala/za/co/absa/atum/model/dto/CheckpointDTO.scala

…-refactoring # Conflicts: # agent/src/main/scala/za/co/absa/atum/agent/AtumAgent.scala # agent/src/main/scala/za/co/absa/atum/agent/dispatcher/HttpDispatcher.scala # server/src/main/scala/za/co/absa/atum/server/api/database/Runs.scala

lsulak · 2024-02-01T16:38:41Z

agent/src/main/scala/za/co/absa/atum/agent/AtumAgent.scala

@@ -64,7 +64,7 @@ class AtumAgent private[agent] () {
  }

  /**
-   *  Provides an AtumContext given a `AtumPartitions` instance. Retrieves the data from AtumService API.


https://docs.scala-lang.org/style/scaladoc.html

github-actions · 2024-02-01T16:47:18Z

JaCoCo server module code coverage report - scala 2.13.11

Overall Project	61.33% `-13.71%`	❌
Files changed	16.6%	❌

File	Coverage
PlayJsonImplicits.scala	30.18% `-71.27%`	❌

github-actions · 2024-02-12T14:27:58Z

JaCoCo agent module code coverage report - scala 2.12.18

Overall Project	NaN% `NaN%`	🍏

There is no coverage information present for the Files changed

github-actions · 2024-02-12T14:28:00Z

JaCoCo model module code coverage report - scala 2.12.18

Overall Project	NaN% `NaN%`	🍏

There is no coverage information present for the Files changed

…n more

benedeki

Let's merge it. A major improvement.

agent/src/main/scala/za/co/absa/atum/agent/dispatcher/HttpDispatcher.scala

agent/src/main/scala/za/co/absa/atum/agent/model/Measurement.scala

benedeki · 2024-02-28T08:14:46Z

.github/workflows/build.yml

      - name: Build and run tests
-        run: sbt ++${{matrix.scala}} test doc
+        run: sbt "project server" ++${{matrix.scala}} test doc


Aren't missing the db module (tests)?

…ture using probably circe

… Java 11 uses different algorithm - JLS3 "Binary Compatibility" you can read more about it if interested; this PR addresses that and removes reflection completely from this SerDe

github-actions · 2024-03-05T08:45:17Z

JaCoCo agent module code coverage report - scala 2.13.11

Overall Project	73.43% `-48.98%`	❌
Files changed	52.57%	❌

File	Coverage
MeasureResult.scala	100% `-260%`	❌
MeasurementBuilder.scala	94.74% `-5.26%`	🍏
AtumContext.scala	91.11%	🍏
Measure.scala	88.38% `-67.84%`	❌
MeasuresBuilder.scala	83.87% `-12.9%`	🍏
AtumAgent.scala	77.44%	🍏
AtumAgentException.scala	64.71% `-347.06%`	❌
HttpDispatcher.scala	0% `-19.64%`	❌

github-actions · 2024-03-05T08:45:19Z

JaCoCo model module code coverage report - scala 2.13.11

Overall Project	60.65% `-64.91%`	❌
Files changed	32.49%	❌

File	Coverage
CheckpointDTO.scala	100% `-20.69%`	❌
MeasureResultDTO.scala	75.68% `-475.68%`	❌
SerializationUtils.scala	47.91% `-64.19%`	❌

benedeki

Let's get it in 😉

Refactoring Measurement and a lot of stuff from Agent side

7b013f9

lsulak requested review from benedeki, TebaleloS, Zejnilovic, dk1844 and salamonpavel as code owners November 1, 2023 11:56

lsulak changed the title ~~Refactoring Measurement and a lot of stuff from Agent side~~ Refactoring Measurement, from Agent side Nov 1, 2023

lsulak self-assigned this Nov 1, 2023

lsulak added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Nov 1, 2023

lsulak marked this pull request as draft November 1, 2023 11:57

Better validation handling - Set for measurements and measures as the…

0b3b5b4

… order there is not important

lsulak removed the work in progress Work on this item is not yet finished (mainly intended for PRs) label Nov 1, 2023

lsulak changed the title ~~Refactoring Measurement, from Agent side~~ #105: Refactoring Measurement, from Agent side Nov 1, 2023

lsulak marked this pull request as ready for review November 2, 2023 10:07

lsulak added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Nov 2, 2023

lsulak added 2 commits November 7, 2023 16:46

Merge branch 'master' into feature/measurement-refactoring

e9b771d

Merge branch 'master' into feature/measurement-refactoring

bb72f40

TebaleloS reviewed Nov 8, 2023

View reviewed changes

lsulak mentioned this pull request Nov 8, 2023

Feature/99 custom measure 2 #108

Closed

Merge branch 'master' into feature/measurement-refactoring

fe62225

Merge branch 'master' into feature/measurement-refactoring

6a299b6

lsulak added 4 commits November 15, 2023 10:46

post-merge fix, set can't be understood by the fa-db

093c6e0

fixing the build, after merge conflict resolution

7a0b52e

lsulak added 2 commits February 1, 2024 17:33

post-review changes

8a92e94

lsulak commented Feb 1, 2024

View reviewed changes

lsulak added 2 commits February 1, 2024 17:44

post-merge fixes

d85c85e

post-merge fixes

a80956a

lsulak added 3 commits February 8, 2024 17:22

#105: refactoring to have a generic type of the measurement result

fcfe3fb

Merge branch 'master' into feature/measurement-refactoring

cae400d

Merge branch 'master' into feature/measurement-refactoring

23ff72b

lsulak added 2 commits February 13, 2024 17:04

Merge branch 'master' into feature/measurement-refactoring

e6df0d7

Merge branch 'master' into feature/measurement-refactoring

bdf8906

salamonpavel previously approved these changes Feb 16, 2024

View reviewed changes

lsulak added 2 commits February 26, 2024 17:36

Merge branch 'master' into feature/measurement-refactoring

8beceef

#105: dropping Measurement in agent and thus simplifying the code eve…

830be6e

…n more

lsulak dismissed salamonpavel’s stale review via 830be6e February 27, 2024 16:29

benedeki previously approved these changes Feb 28, 2024

View reviewed changes

#105: changing Enumeration to case objects for easier SerDe in the fu…

098d40b

…ture using probably circe

lsulak dismissed benedeki’s stale review via 098d40b February 28, 2024 16:38

lsulak added 4 commits February 28, 2024 17:38

fix

6d838d7

Merge branch 'master' into feature/measurement-refactoring

79a0bed

#105: getSimpleName is using reflection, and reflection in Java 8 and…

4a34b7f

… Java 11 uses different algorithm - JLS3 "Binary Compatibility" you can read more about it if interested; this PR addresses that and removes reflection completely from this SerDe

Merge branch 'master' into feature/measurement-refactoring

98d10e0

Merge branch 'master' into feature/measurement-refactoring

595a972

benedeki approved these changes Mar 8, 2024

View reviewed changes

lsulak merged commit a7f3f64 into master Mar 8, 2024
8 of 10 checks passed

lsulak deleted the feature/measurement-refactoring branch March 8, 2024 11:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#105: Refactoring Measurement, from Agent side #104

#105: Refactoring Measurement, from Agent side #104

lsulak commented Nov 1, 2023 •

edited

Loading

github-actions bot commented Nov 1, 2023 •

edited

Loading

github-actions bot commented Nov 1, 2023 •

edited

Loading

TebaleloS left a comment

lsulak commented Nov 9, 2023

github-actions bot commented Nov 10, 2023 •

edited

Loading

github-actions bot commented Nov 10, 2023 •

edited

Loading

lsulak Feb 1, 2024

github-actions bot commented Feb 1, 2024 •

edited

Loading

github-actions bot commented Feb 12, 2024 •

edited

Loading

github-actions bot commented Feb 12, 2024 •

edited

Loading

benedeki left a comment

benedeki Feb 28, 2024

lsulak Feb 29, 2024

github-actions bot commented Mar 5, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024 •

edited

Loading

benedeki left a comment

#105: Refactoring Measurement, from Agent side #104

#105: Refactoring Measurement, from Agent side #104

Conversation

lsulak commented Nov 1, 2023 • edited Loading

github-actions bot commented Nov 1, 2023 • edited Loading

JaCoCo agent module code coverage report - spark:2 - scala 2.12.12

Build Failed

github-actions bot commented Nov 1, 2023 • edited Loading

JaCoCo server module code coverage report - scala 2.12.12

Build Failed

TebaleloS left a comment

Choose a reason for hiding this comment

lsulak commented Nov 9, 2023

github-actions bot commented Nov 10, 2023 • edited Loading

JaCoCo agent module code coverage report - spark:2 - scala 2.12.18

Build Failed

github-actions bot commented Nov 10, 2023 • edited Loading

JaCoCo server module code coverage report - scala 2.12.18

Build Failed

lsulak Feb 1, 2024

Choose a reason for hiding this comment

github-actions bot commented Feb 1, 2024 • edited Loading

JaCoCo server module code coverage report - scala 2.13.11

github-actions bot commented Feb 12, 2024 • edited Loading

JaCoCo agent module code coverage report - scala 2.12.18

github-actions bot commented Feb 12, 2024 • edited Loading

JaCoCo model module code coverage report - scala 2.12.18

benedeki left a comment

Choose a reason for hiding this comment

benedeki Feb 28, 2024

Choose a reason for hiding this comment

lsulak Feb 29, 2024

Choose a reason for hiding this comment

github-actions bot commented Mar 5, 2024 • edited Loading

JaCoCo agent module code coverage report - scala 2.13.11

github-actions bot commented Mar 5, 2024 • edited Loading

JaCoCo model module code coverage report - scala 2.13.11

benedeki left a comment

Choose a reason for hiding this comment

lsulak commented Nov 1, 2023 •

edited

Loading

github-actions bot commented Nov 1, 2023 •

edited

Loading

github-actions bot commented Nov 1, 2023 •

edited

Loading

github-actions bot commented Nov 10, 2023 •

edited

Loading

github-actions bot commented Nov 10, 2023 •

edited

Loading

github-actions bot commented Feb 1, 2024 •

edited

Loading

github-actions bot commented Feb 12, 2024 •

edited

Loading

github-actions bot commented Feb 12, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024 •

edited

Loading

github-actions bot commented Mar 5, 2024 •

edited

Loading