-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
optimize temporary and permanent limits db writings and db reading #82
Merged
Merged
Changes from 20 commits
Commits
Show all changes
66 commits
Select commit
Hold shift + click to select a range
031bc5c
optimize temporary limits db writings and db reading
EtienneLt db4983e
replace array sql type by jackson text
EtienneLt 0287fd3
fix
EtienneLt b3889a9
fix
EtienneLt 778429a
disable sonar on migration package
EtienneLt d611ece
fix
EtienneLt 4847027
log temporary limits insertion
EtienneLt d0319e5
refactor permanent limit too and clean
EtienneLt 38ffe4c
exclude sonar duplication for migrations server
EtienneLt 45bbcec
fix disable duplication on migrations
EtienneLt 5a4efb2
Merge branch 'main' into optimize-temporarylimits-in-db
EtienneLt ba52b95
fix sonar
EtienneLt 042ee22
review and fixes
EtienneLt bb527ff
fix sonar
EtienneLt 07e71b7
add index
EtienneLt 78bdf52
fix index
EtienneLt e434daa
change java migration to sql migration
EtienneLt 4bd3ce4
refactor and annd transfer method when getting
EtienneLt 577bdad
fix and migrate in java
EtienneLt 7547a85
fix
EtienneLt 3758ae5
Migration endpoint + java changeset (not working) + reorganize limits…
etiennehomer e033295
clean
etiennehomer ce59bb4
Merge branch 'main' into optimize-temporarylimits-in-db
EtienneLt 8168f22
Build NetworkStoreRepository bean in migration class
etiennehomer e3ee705
exclude migration package from coverage
etiennehomer d7ef98c
move MigrationController + move sonar exclusion property
etiennehomer d430611
Clean MigrationController
etiennehomer eb0e117
Merge branch 'main' into optimize-temporarylimits-in-db
etiennehomer 0e0947d
Fix sonar exclusion
etiennehomer 8ef0f55
Fix sonar exclusion
etiennehomer 0dca16c
Add test for migration + clean licence and author + put V211 methods …
etiennehomer 22bb345
Sonar fixes + rename
etiennehomer fcacba6
End of migration test
etiennehomer e8809bd
improve logs + remove java changeset from changelog
etiennehomer 9220880
Fix migration
etiennehomer 465719d
No on the fly migration
etiennehomer 8ed5c71
FIX
etiennehomer aefea20
Fix migration test
etiennehomer 33821aa
Enrich test
etiennehomer 53150ca
clean
etiennehomer f25eb73
Clean
etiennehomer 4e82707
Rename tables
etiennehomer 7a74234
Clean
etiennehomer bbfbf82
clean
etiennehomer e76302b
Clean
etiennehomer 671aa1a
Do no exclude migration from coverage
etiennehomer a7d1017
Exclude migration again from coverage
etiennehomer 5befc61
Refacto for minimum changes in PR
etiennehomer 22c2cef
Refacto for minimum changes in PR 2
etiennehomer 2efa9d0
Refacto for minimum changes in PR 3
etiennehomer 06ff49c
Refacto for minimum changes in PR 4
etiennehomer 217b588
Refacto for minimum changes in PR 5
etiennehomer c7b4603
Migrate by networkId and variantNum
etiennehomer e80880e
Clean
etiennehomer 25338ea
CLean
etiennehomer 4329dd7
CLean
etiennehomer 2aa4960
Fix clone v211 limits + fix delete network + clean up
etiennehomer 3562463
Remove comments
etiennehomer 4136d90
Rename v211limits package
etiennehomer 9809102
Improve migration logs
etiennehomer b066133
Merge branch 'main' into optimize-temporarylimits-in-db
etiennehomer 0b61119
Clean
etiennehomer 74178ef
Manage exceptions of execute()
etiennehomer f295e08
Fix sonar warnings
etiennehomer 5c70460
Remove logger in catch
etiennehomer 85cc31c
Clean
etiennehomer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
245 changes: 211 additions & 34 deletions
245
...k-store-server/src/main/java/com/powsybl/network/store/server/NetworkStoreRepository.java
Large diffs are not rendered by default.
Oops, something went wrong.
295 changes: 186 additions & 109 deletions
295
network-store-server/src/main/java/com/powsybl/network/store/server/QueryCatalog.java
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
51 changes: 51 additions & 0 deletions
51
...ore-server/src/main/java/com/powsybl/network/store/server/json/PermanentLimitSqlData.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
/** | ||
* Copyright (c) 2024, RTE (http://www.rte-france.com) | ||
* This Source Code Form is subject to the terms of the Mozilla Public | ||
* License, v. 2.0. If a copy of the MPL was not distributed with this | ||
* file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
*/ | ||
package com.powsybl.network.store.server.json; | ||
|
||
import com.powsybl.iidm.network.LimitType; | ||
import com.powsybl.network.store.server.dto.PermanentLimitAttributes; | ||
import lombok.AllArgsConstructor; | ||
import lombok.Builder; | ||
import lombok.Getter; | ||
import lombok.ToString; | ||
|
||
/** | ||
* @author Etienne Lesot <etienne.lesot at rte-france.com> | ||
*/ | ||
@ToString | ||
@Builder | ||
@AllArgsConstructor | ||
@Getter | ||
public class PermanentLimitSqlData { | ||
|
||
private String operationalLimitsGroupId; | ||
private double value; | ||
private Integer side; | ||
private LimitType limitType; | ||
|
||
public PermanentLimitSqlData() { | ||
// empty constructor for Jackson | ||
} | ||
|
||
public static PermanentLimitSqlData of(PermanentLimitAttributes permanentLimitAttributes) { | ||
return PermanentLimitSqlData.builder() | ||
.operationalLimitsGroupId(permanentLimitAttributes.getOperationalLimitsGroupId()) | ||
.value(permanentLimitAttributes.getValue()) | ||
.side(permanentLimitAttributes.getSide()) | ||
.limitType(permanentLimitAttributes.getLimitType()) | ||
.build(); | ||
} | ||
|
||
public PermanentLimitAttributes toPermanentLimitAttributes() { | ||
return PermanentLimitAttributes.builder() | ||
.operationalLimitsGroupId(operationalLimitsGroupId) | ||
.value(value) | ||
.side(side) | ||
.limitType(limitType) | ||
.build(); | ||
} | ||
} |
59 changes: 59 additions & 0 deletions
59
...ore-server/src/main/java/com/powsybl/network/store/server/json/TemporaryLimitSqlData.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
/** | ||
* Copyright (c) 2024, RTE (http://www.rte-france.com) | ||
* This Source Code Form is subject to the terms of the Mozilla Public | ||
* License, v. 2.0. If a copy of the MPL was not distributed with this | ||
* file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
*/ | ||
package com.powsybl.network.store.server.json; | ||
|
||
import com.powsybl.iidm.network.LimitType; | ||
import com.powsybl.network.store.model.TemporaryLimitAttributes; | ||
import lombok.AllArgsConstructor; | ||
import lombok.Builder; | ||
import lombok.Getter; | ||
import lombok.ToString; | ||
|
||
/** | ||
* @author Etienne Lesot <etienne.lesot at rte-france.com> | ||
*/ | ||
@ToString | ||
@Builder | ||
@AllArgsConstructor | ||
@Getter | ||
public class TemporaryLimitSqlData { | ||
private String operationalLimitsGroupId; | ||
private Integer side; | ||
private LimitType limitType; | ||
private String name; | ||
private double value; | ||
private Integer acceptableDuration; | ||
private boolean fictitious; | ||
|
||
public TemporaryLimitSqlData() { | ||
// empty constructor for Jackson | ||
} | ||
|
||
public static TemporaryLimitSqlData of(TemporaryLimitAttributes temporaryLimit) { | ||
return TemporaryLimitSqlData.builder() | ||
.operationalLimitsGroupId(temporaryLimit.getOperationalLimitsGroupId()) | ||
.side(temporaryLimit.getSide()) | ||
.limitType(temporaryLimit.getLimitType()) | ||
.name(temporaryLimit.getName()) | ||
.value(temporaryLimit.getValue()) | ||
.acceptableDuration(temporaryLimit.getAcceptableDuration()) | ||
.fictitious(temporaryLimit.isFictitious()) | ||
.build(); | ||
} | ||
|
||
public TemporaryLimitAttributes toTemporaryLimitAttributes() { | ||
return TemporaryLimitAttributes.builder() | ||
.operationalLimitsGroupId(operationalLimitsGroupId) | ||
.side(side) | ||
.limitType(limitType) | ||
.name(name) | ||
.value(value) | ||
.acceptableDuration(acceptableDuration) | ||
.fictitious(fictitious) | ||
.build(); | ||
} | ||
} |
48 changes: 48 additions & 0 deletions
48
...rk-store-server/src/main/resources/db/changelog/changesets/changelog_20241121T110000Z.xml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" | ||
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog | ||
https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd"> | ||
|
||
<changeSet id="2986331049275-5" author="lesoteti"> | ||
<createTable tableName="newtemporarylimits"> | ||
<column name="networkuuid" type="VARCHAR(255)"> | ||
<constraints nullable="false" primaryKey="true" primaryKeyName="newtemporarylimits_pkey"/> | ||
</column> | ||
<column name="variantnum" type="INTEGER"> | ||
<constraints nullable="false" primaryKey="true" primaryKeyName="newtemporarylimits_pkey"/> | ||
</column> | ||
<column name="equipmentid" type="VARCHAR(255)"> | ||
<constraints nullable="false" primaryKey="true" primaryKeyName="newtemporarylimits_pkey"/> | ||
</column> | ||
<column name="equipmenttype" type="VARCHAR(255)"> | ||
</column> | ||
<column name="temporarylimits" type="TEXT"/> | ||
</createTable> | ||
<createTable tableName="newpermanentlimits"> | ||
<column name="networkuuid" type="VARCHAR(255)"> | ||
<constraints nullable="false" primaryKey="true" primaryKeyName="newpermanentlimits_pkey"/> | ||
</column> | ||
<column name="variantnum" type="INTEGER"> | ||
<constraints nullable="false" primaryKey="true" primaryKeyName="newpermanentlimits_pkey"/> | ||
</column> | ||
<column name="equipmentid" type="VARCHAR(255)"> | ||
<constraints nullable="false" primaryKey="true" primaryKeyName="newpermanentlimits_pkey"/> | ||
</column> | ||
<column name="equipmenttype" type="VARCHAR(255)"> | ||
</column> | ||
<column name="permanentlimits" type="TEXT"/> | ||
</createTable> | ||
<createIndex indexName="permanentlimits_networkuuid_variantnum_equipmenttype_idx" tableName="newpermanentlimits"> | ||
<column name="networkuuid"/> | ||
<column name="variantnum"/> | ||
<column name="equipmenttype"/> | ||
</createIndex> | ||
<createIndex indexName="temporarylimits_networkuuid_variantnum_equipmenttype_idx" tableName="newtemporarylimits"> | ||
<column name="networkuuid"/> | ||
<column name="variantnum"/> | ||
<column name="equipmenttype"/> | ||
</createIndex> | ||
</changeSet> | ||
|
||
</databaseChangeLog> |
22 changes: 22 additions & 0 deletions
22
...re-server/src/main/resources/db/changelog/changesets/refactor_limits_20241121T110000Z.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
-- will be used in next deployment | ||
INSERT INTO newtemporarylimits (equipmentid, equipmenttype, networkuuid, variantnum, temporarylimits) | ||
SELECT equipmentid, equipmenttype, networkuuid, variantnum, | ||
json_agg(json_build_object( | ||
'operationalLimitsGroupId', operationalLimitsGroupId, | ||
'side', side, | ||
'limitType', limitType, | ||
'name', name, | ||
'value', value_, | ||
'acceptableDuration', acceptableduration, | ||
'fictitious', fictitious | ||
)) as temporarylimits FROM temporarylimit | ||
GROUP BY equipmentid, equipmenttype, networkuuid, variantnum; | ||
INSERT INTO newpermanentlimits (equipmentid, equipmenttype, networkuuid, variantnum, permanentlimits) | ||
SELECT equipmentid, equipmenttype, networkuuid, variantnum, | ||
json_agg(json_build_object( | ||
'operationalLimitsGroupId', operationalLimitsGroupId, | ||
'value', value_, | ||
'side', side, | ||
'limitType', limitType | ||
)) as permanentlimits FROM permanentlimit | ||
GROUP BY equipmentid, equipmenttype, networkuuid, variantnum; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking of doing the liquibase migration by doing a java change that just calls the same method as the one used for users
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. But the spring context is not available when the liquibase migration is executed and it makes things a bit more complicated to code. I built an "handmade" NetworkStoreRepository, not by Spring injection. To do so, I needed to build a javax.sql.DataSource and to retrieve the java.sql.Connection...