optimize temporary and permanent limits db writings and db reading #82

EtienneLt · 2024-11-21T14:58:48Z

Please check if the PR fulfills these requirements

The commit message follows our guidelines
Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

Does this PR already have an issue describing the problem?

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior (if this is a feature change)?

Does this PR introduce a breaking change or deprecate an API?

Yes
No

If yes, please check if the following requirements are fulfilled

The Breaking Change or Deprecated label has been added
The migration steps are described in the following section

What changes might users need to make in their application due to this PR? (migration steps)

Other information:

Signed-off-by: Etienne LESOT <[email protected]>

etiennehomer · 2024-12-02T09:48:51Z

network-store-server/src/main/resources/db/changelog/changesets/changelog_20241121T110000Z.xml

+        <customChange class="com.powsybl.network.store.server.migrations.PermanentLimitsMigration"/>
+        <customChange class="com.powsybl.network.store.server.migrations.TemporaryLimitsMigration"/>
+        <renameTable oldTableName="newpermanentlimits"
+                     newTableName="permanentlimits"/>


Before this, drop permanentlimit table.

And rename to permanentlimit

I rename to permanentlimits and it was permanentlimit before do you agree ? same for temporarylimits

We can keep permanentlimit I think. All table names use the singular

Discussed with Etienne L: we keep the plurial as one row of permanentLimits contains several permanent limits

etiennehomer · 2024-12-02T09:49:01Z

network-store-server/src/main/resources/db/changelog/changesets/changelog_20241121T110000Z.xml

+        <renameTable oldTableName="newpermanentlimits"
+                     newTableName="permanentlimits"/>
+        <renameTable oldTableName="newtemporarylimits"
+                     newTableName="temporarylimits"/>


network-store-server/src/main/java/com/powsybl/network/store/server/QueryCatalog.java

network-store-server/src/main/java/com/powsybl/network/store/server/TemporaryLimitSqlData.java

network-store-server/src/main/java/com/powsybl/network/store/server/QueryCatalog.java

network-store-server/src/main/java/com/powsybl/network/store/server/PermanentLimitSqlData.java

network-store-server/src/main/resources/db/changelog/changesets/changelog_20241121T110000Z.xml

network-store-server/src/main/java/com/powsybl/network/store/server/PermanentLimitSqlData.java

network-store-server/src/main/java/com/powsybl/network/store/server/TemporaryLimitSqlData.java

...rver/src/main/java/com/powsybl/network/store/server/migrations/PermanentLimitsMigration.java

Signed-off-by: Etienne LESOT <[email protected]>

jonenst · 2024-12-13T16:50:23Z

...store-server/src/main/resources/db/changelog/changesets/refactor_limits_20241121T110000Z.sql

@@ -0,0 +1,22 @@
+-- will be used in next deployment


I was thinking of doing the liquibase migration by doing a java change that just calls the same method as the one used for users

Done. But the spring context is not available when the liquibase migration is executed and it makes things a bit more complicated to code. I built an "handmade" NetworkStoreRepository, not by Spring injection. To do so, I needed to build a javax.sql.DataSource and to retrieve the java.sql.Connection...

jonenst · 2024-12-13T16:52:54Z

network-store-server/src/main/java/com/powsybl/network/store/server/QueryCatalog.java

-    static final String SELECTED_OPERATIONAL_LIMITS_GROUP_ID_COLUMN = "selectedOperationalLimitsGroupId";
+    static final String TEMPORARY_LIMITS_TABLE = "newtemporarylimits";
+    static final String TEMPORARY_LIMITS = "temporarylimits";
+    static final String OLD_TEMPORARY_LIMITS = "temporarylimit";


need to find a better name than old because this code is going to stay if it goes in the liquibase changeset. Maybe "V211LimitMigration"

I prefix all code related to old tables with V211

Signed-off-by: Etienne Homer <[email protected]>

jonenst · 2025-01-14T08:59:04Z

...ore-server/src/main/java/com/powsybl/network/store/server/migration/V211LimitsMigration.java

+                migrateV211Limits(repository, networkUuid, variantNum);
+            }
+        } catch (SQLException | DatabaseException e) {
+            LOGGER.error(e.getMessage(), e);


maybe we shouldn't abort the migration on an error, migrate as much as possible ?

jonenst · 2025-01-14T09:01:51Z

...-server/src/main/java/com/powsybl/network/store/server/migration/V211LimitsQueryCatalog.java

+/**
+ * @author Etienne Homer <etienne.homer at rte-france.com>
+ */
+//Class to be deprecated when limits are fully migrated - should be after v2.13 deployment


remove comment, we keep this for the migration for eternity

jonenst · 2025-01-14T09:13:42Z

network-store-server/src/main/resources/db/changelog/changesets/changelog_20241223T130000Z.xml

+    <changeSet author="homereti" id="8428118131015-1">
+        <customChange class="com.powsybl.network.store.server.migration.V211LimitsMigration"/>
+    </changeSet>
+


add delete v211 tables here or even in the next version 2.14 ??

I added a new changelog to be executed in 2.14

Signed-off-by: Etienne Homer <[email protected]>

jonenst · 2025-01-15T14:08:28Z

...src/main/java/com/powsybl/network/store/server/migration/v211limits/V211LimitsMigration.java

+                migrateV211Limits(repository, networkUuid, variantNum);
+            }
+        } catch (SQLException | DatabaseException e) {
+            LOGGER.error(e.getMessage(), e);


maybe need to be a bit smarter here. We should do the same exception handling as regular sql/xml changesets

Signed-off-by: Etienne Homer <[email protected]>

sonarqubecloud · 2025-01-15T16:32:12Z

Quality Gate passed

Issues
16 New issues
0 Accepted issues

Measures
0 Security Hotspots
89.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

optimize temporary limits db writings and db reading

031bc5c

Signed-off-by: Etienne LESOT <[email protected]>

EtienneLt requested review from jonenst and etiennehomer November 21, 2024 14:58

EtienneLt self-assigned this Nov 21, 2024

EtienneLt added 7 commits November 22, 2024 15:47

replace array sql type by jackson text

db4983e

Signed-off-by: Etienne LESOT <[email protected]>

fix

0287fd3

Signed-off-by: Etienne LESOT <[email protected]>

fix

b3889a9

Signed-off-by: Etienne LESOT <[email protected]>

disable sonar on migration package

778429a

Signed-off-by: Etienne LESOT <[email protected]>

fix

d611ece

Signed-off-by: Etienne LESOT <[email protected]>

log temporary limits insertion

4847027

Signed-off-by: Etienne LESOT <[email protected]>

refactor permanent limit too and clean

d0319e5

Signed-off-by: Etienne LESOT <[email protected]>

jonenst changed the title ~~optimize temporary limits db writings and db reading~~ optimize temporary and permanent limits db writings and db reading Nov 29, 2024

EtienneLt added 4 commits November 29, 2024 14:29

exclude sonar duplication for migrations server

38ffe4c

Signed-off-by: Etienne LESOT <[email protected]>

fix disable duplication on migrations

45bbcec

Signed-off-by: Etienne LESOT <[email protected]>

Merge branch 'main' into optimize-temporarylimits-in-db

5a4efb2

fix sonar

ba52b95

Signed-off-by: Etienne LESOT <[email protected]>

etiennehomer requested changes Dec 2, 2024

View reviewed changes

network-store-server/src/main/java/com/powsybl/network/store/server/PermanentLimitSqlData.java Outdated Show resolved Hide resolved

network-store-server/src/main/java/com/powsybl/network/store/server/TemporaryLimitSqlData.java Outdated Show resolved Hide resolved

etiennehomer requested changes Dec 2, 2024

View reviewed changes

...rver/src/main/java/com/powsybl/network/store/server/migrations/PermanentLimitsMigration.java Outdated Show resolved Hide resolved

review and fixes

042ee22

Signed-off-by: Etienne LESOT <[email protected]>

EtienneLt requested a review from etiennehomer December 3, 2024 09:59

EtienneLt added 7 commits December 5, 2024 09:06

fix sonar

bb527ff

Signed-off-by: Etienne LESOT <[email protected]>

add index

07e71b7

Signed-off-by: Etienne LESOT <[email protected]>

fix index

78bdf52

Signed-off-by: Etienne LESOT <[email protected]>

change java migration to sql migration

e434daa

Signed-off-by: Etienne LESOT <[email protected]>

refactor and annd transfer method when getting

4bd3ce4

Signed-off-by: Etienne LESOT <[email protected]>

fix and migrate in java

577bdad

Signed-off-by: Etienne LESOT <[email protected]>

fix

7547a85

Signed-off-by: Etienne LESOT <[email protected]>

jonenst reviewed Dec 13, 2024

View reviewed changes

etiennehomer added 11 commits January 13, 2025 11:09

Do no exclude migration from coverage

671aa1a

Signed-off-by: Etienne Homer <[email protected]>

Exclude migration again from coverage

a7d1017

Signed-off-by: Etienne Homer <[email protected]>

Refacto for minimum changes in PR

5befc61

Signed-off-by: Etienne Homer <[email protected]>

Refacto for minimum changes in PR 2

22c2cef

Signed-off-by: Etienne Homer <[email protected]>

Refacto for minimum changes in PR 3

2efa9d0

Signed-off-by: Etienne Homer <[email protected]>

Refacto for minimum changes in PR 4

06ff49c

Signed-off-by: Etienne Homer <[email protected]>

Refacto for minimum changes in PR 5

217b588

Signed-off-by: Etienne Homer <[email protected]>

Migrate by networkId and variantNum

c7b4603

Signed-off-by: Etienne Homer <[email protected]>

Clean

e80880e

Signed-off-by: Etienne Homer <[email protected]>

CLean

25338ea

Signed-off-by: Etienne Homer <[email protected]>

CLean

4329dd7

Signed-off-by: Etienne Homer <[email protected]>

jonenst reviewed Jan 14, 2025

View reviewed changes

etiennehomer added 6 commits January 14, 2025 14:26

Fix clone v211 limits + fix delete network + clean up

2aa4960

Signed-off-by: Etienne Homer <[email protected]>

Remove comments

3562463

Signed-off-by: Etienne Homer <[email protected]>

Rename v211limits package

4136d90

Signed-off-by: Etienne Homer <[email protected]>

Improve migration logs

9809102

Signed-off-by: Etienne Homer <[email protected]>

Merge branch 'main' into optimize-temporarylimits-in-db

b066133

Clean

0b61119

Signed-off-by: Etienne Homer <[email protected]>

jonenst reviewed Jan 15, 2025

View reviewed changes

etiennehomer added 4 commits January 15, 2025 15:47

Manage exceptions of execute()

74178ef

Signed-off-by: Etienne Homer <[email protected]>

Fix sonar warnings

f295e08

Signed-off-by: Etienne Homer <[email protected]>

Remove logger in catch

5c70460

Signed-off-by: Etienne Homer <[email protected]>

Clean

85cc31c

Signed-off-by: Etienne Homer <[email protected]>

jonenst approved these changes Jan 15, 2025

View reviewed changes

etiennehomer approved these changes Jan 15, 2025

View reviewed changes

jonenst merged commit f8bd618 into main Jan 15, 2025
5 checks passed

jonenst deleted the optimize-temporarylimits-in-db branch January 15, 2025 16:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

optimize temporary and permanent limits db writings and db reading #82

optimize temporary and permanent limits db writings and db reading #82

EtienneLt commented Nov 21, 2024

etiennehomer Dec 2, 2024

EtienneLt Dec 3, 2024

etiennehomer Dec 3, 2024

etiennehomer Dec 4, 2024

etiennehomer Dec 2, 2024

jonenst Dec 13, 2024

etiennehomer Jan 7, 2025

jonenst Dec 13, 2024

etiennehomer Jan 14, 2025

jonenst Jan 14, 2025

jonenst Jan 14, 2025

etiennehomer Jan 14, 2025

jonenst Jan 14, 2025

etiennehomer Jan 14, 2025

jonenst Jan 15, 2025

sonarqubecloud bot commented Jan 15, 2025

optimize temporary and permanent limits db writings and db reading #82

optimize temporary and permanent limits db writings and db reading #82

Conversation

EtienneLt commented Nov 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 15, 2025

Quality Gate passed