Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimize temporary and permanent limits db writings and db reading #82

Merged
merged 66 commits into from
Jan 15, 2025
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
031bc5c
optimize temporary limits db writings and db reading
EtienneLt Nov 21, 2024
db4983e
replace array sql type by jackson text
EtienneLt Nov 22, 2024
0287fd3
fix
EtienneLt Nov 22, 2024
b3889a9
fix
EtienneLt Nov 22, 2024
778429a
disable sonar on migration package
EtienneLt Nov 22, 2024
d611ece
fix
EtienneLt Nov 22, 2024
4847027
log temporary limits insertion
EtienneLt Nov 28, 2024
d0319e5
refactor permanent limit too and clean
EtienneLt Nov 29, 2024
38ffe4c
exclude sonar duplication for migrations server
EtienneLt Nov 29, 2024
45bbcec
fix disable duplication on migrations
EtienneLt Nov 29, 2024
5a4efb2
Merge branch 'main' into optimize-temporarylimits-in-db
EtienneLt Nov 29, 2024
ba52b95
fix sonar
EtienneLt Nov 29, 2024
042ee22
review and fixes
EtienneLt Dec 3, 2024
bb527ff
fix sonar
EtienneLt Dec 5, 2024
07e71b7
add index
EtienneLt Dec 5, 2024
78bdf52
fix index
EtienneLt Dec 5, 2024
e434daa
change java migration to sql migration
EtienneLt Dec 6, 2024
4bd3ce4
refactor and annd transfer method when getting
EtienneLt Dec 12, 2024
577bdad
fix and migrate in java
EtienneLt Dec 13, 2024
7547a85
fix
EtienneLt Dec 13, 2024
3758ae5
Migration endpoint + java changeset (not working) + reorganize limits…
etiennehomer Dec 31, 2024
e033295
clean
etiennehomer Dec 31, 2024
ce59bb4
Merge branch 'main' into optimize-temporarylimits-in-db
EtienneLt Jan 6, 2025
8168f22
Build NetworkStoreRepository bean in migration class
etiennehomer Jan 7, 2025
e3ee705
exclude migration package from coverage
etiennehomer Jan 7, 2025
d7ef98c
move MigrationController + move sonar exclusion property
etiennehomer Jan 7, 2025
d430611
Clean MigrationController
etiennehomer Jan 7, 2025
eb0e117
Merge branch 'main' into optimize-temporarylimits-in-db
etiennehomer Jan 7, 2025
0e0947d
Fix sonar exclusion
etiennehomer Jan 7, 2025
8ef0f55
Fix sonar exclusion
etiennehomer Jan 7, 2025
0dca16c
Add test for migration + clean licence and author + put V211 methods …
etiennehomer Jan 8, 2025
22bb345
Sonar fixes + rename
etiennehomer Jan 9, 2025
fcacba6
End of migration test
etiennehomer Jan 9, 2025
e8809bd
improve logs + remove java changeset from changelog
etiennehomer Jan 9, 2025
9220880
Fix migration
etiennehomer Jan 9, 2025
465719d
No on the fly migration
etiennehomer Jan 9, 2025
8ed5c71
FIX
etiennehomer Jan 10, 2025
aefea20
Fix migration test
etiennehomer Jan 10, 2025
33821aa
Enrich test
etiennehomer Jan 10, 2025
53150ca
clean
etiennehomer Jan 10, 2025
f25eb73
Clean
etiennehomer Jan 10, 2025
4e82707
Rename tables
etiennehomer Jan 10, 2025
7a74234
Clean
etiennehomer Jan 10, 2025
bbfbf82
clean
etiennehomer Jan 10, 2025
e76302b
Clean
etiennehomer Jan 10, 2025
671aa1a
Do no exclude migration from coverage
etiennehomer Jan 13, 2025
a7d1017
Exclude migration again from coverage
etiennehomer Jan 13, 2025
5befc61
Refacto for minimum changes in PR
etiennehomer Jan 13, 2025
22c2cef
Refacto for minimum changes in PR 2
etiennehomer Jan 13, 2025
2efa9d0
Refacto for minimum changes in PR 3
etiennehomer Jan 13, 2025
06ff49c
Refacto for minimum changes in PR 4
etiennehomer Jan 13, 2025
217b588
Refacto for minimum changes in PR 5
etiennehomer Jan 13, 2025
c7b4603
Migrate by networkId and variantNum
etiennehomer Jan 13, 2025
e80880e
Clean
etiennehomer Jan 13, 2025
25338ea
CLean
etiennehomer Jan 13, 2025
4329dd7
CLean
etiennehomer Jan 13, 2025
2aa4960
Fix clone v211 limits + fix delete network + clean up
etiennehomer Jan 14, 2025
3562463
Remove comments
etiennehomer Jan 14, 2025
4136d90
Rename v211limits package
etiennehomer Jan 14, 2025
9809102
Improve migration logs
etiennehomer Jan 14, 2025
b066133
Merge branch 'main' into optimize-temporarylimits-in-db
etiennehomer Jan 15, 2025
0b61119
Clean
etiennehomer Jan 15, 2025
74178ef
Manage exceptions of execute()
etiennehomer Jan 15, 2025
f295e08
Fix sonar warnings
etiennehomer Jan 15, 2025
5c70460
Remove logger in catch
etiennehomer Jan 15, 2025
85cc31c
Clean
etiennehomer Jan 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion network-store-server/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,6 @@
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
@Schema(description = "limits attributes")
public class LimitsInfos {

@Schema(description = "List of permeant limits")
@Schema(description = "List of permanent limits")
private List<PermanentLimitAttributes> permanentLimits = new ArrayList<>();

@Schema(description = "List of temporary limits")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/**
* Copyright (c) 2024, RTE (http://www.rte-france.com)
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/.
*/
package com.powsybl.network.store.server.json;

import com.powsybl.iidm.network.LimitType;
import com.powsybl.network.store.server.dto.PermanentLimitAttributes;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.ToString;

/**
* @author Etienne Lesot <etienne.lesot at rte-france.com>
*/
@ToString
@Builder
@AllArgsConstructor
@Getter
public class PermanentLimitSqlData {

private String operationalLimitsGroupId;
private double value;
private Integer side;
private LimitType limitType;

public PermanentLimitSqlData() {
// empty constructor for Jackson
}

public static PermanentLimitSqlData of(PermanentLimitAttributes permanentLimitAttributes) {
return PermanentLimitSqlData.builder()
.operationalLimitsGroupId(permanentLimitAttributes.getOperationalLimitsGroupId())
.value(permanentLimitAttributes.getValue())
.side(permanentLimitAttributes.getSide())
.limitType(permanentLimitAttributes.getLimitType())
.build();
}

public PermanentLimitAttributes toPermanentLimitAttributes() {
return PermanentLimitAttributes.builder()
.operationalLimitsGroupId(operationalLimitsGroupId)
.value(value)
.side(side)
.limitType(limitType)
.build();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/**
* Copyright (c) 2024, RTE (http://www.rte-france.com)
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/.
*/
package com.powsybl.network.store.server.json;

import com.powsybl.iidm.network.LimitType;
import com.powsybl.network.store.model.TemporaryLimitAttributes;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.ToString;

/**
* @author Etienne Lesot <etienne.lesot at rte-france.com>
*/
@ToString
@Builder
@AllArgsConstructor
@Getter
public class TemporaryLimitSqlData {
private String operationalLimitsGroupId;
private Integer side;
private LimitType limitType;
private String name;
private double value;
private Integer acceptableDuration;
private boolean fictitious;

public TemporaryLimitSqlData() {
// empty constructor for Jackson
}

public static TemporaryLimitSqlData of(TemporaryLimitAttributes temporaryLimit) {
return TemporaryLimitSqlData.builder()
.operationalLimitsGroupId(temporaryLimit.getOperationalLimitsGroupId())
.side(temporaryLimit.getSide())
.limitType(temporaryLimit.getLimitType())
.name(temporaryLimit.getName())
.value(temporaryLimit.getValue())
.acceptableDuration(temporaryLimit.getAcceptableDuration())
.fictitious(temporaryLimit.isFictitious())
.build();
}

public TemporaryLimitAttributes toTemporaryLimitAttributes() {
return TemporaryLimitAttributes.builder()
.operationalLimitsGroupId(operationalLimitsGroupId)
.side(side)
.limitType(limitType)
.name(name)
.value(value)
.acceptableDuration(acceptableDuration)
.fictitious(fictitious)
.build();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">

<changeSet id="2986331049275-5" author="lesoteti">
<createTable tableName="newtemporarylimits">
<column name="networkuuid" type="VARCHAR(255)">
<constraints nullable="false" primaryKey="true" primaryKeyName="newtemporarylimits_pkey"/>
</column>
<column name="variantnum" type="INTEGER">
<constraints nullable="false" primaryKey="true" primaryKeyName="newtemporarylimits_pkey"/>
</column>
<column name="equipmentid" type="VARCHAR(255)">
<constraints nullable="false" primaryKey="true" primaryKeyName="newtemporarylimits_pkey"/>
</column>
<column name="equipmenttype" type="VARCHAR(255)">
</column>
<column name="temporarylimits" type="TEXT"/>
</createTable>
<createTable tableName="newpermanentlimits">
<column name="networkuuid" type="VARCHAR(255)">
<constraints nullable="false" primaryKey="true" primaryKeyName="newpermanentlimits_pkey"/>
</column>
<column name="variantnum" type="INTEGER">
<constraints nullable="false" primaryKey="true" primaryKeyName="newpermanentlimits_pkey"/>
</column>
<column name="equipmentid" type="VARCHAR(255)">
<constraints nullable="false" primaryKey="true" primaryKeyName="newpermanentlimits_pkey"/>
</column>
<column name="equipmenttype" type="VARCHAR(255)">
</column>
<column name="permanentlimits" type="TEXT"/>
</createTable>
<createIndex indexName="permanentlimits_networkuuid_variantnum_equipmenttype_idx" tableName="newpermanentlimits">
<column name="networkuuid"/>
<column name="variantnum"/>
<column name="equipmenttype"/>
</createIndex>
<createIndex indexName="temporarylimits_networkuuid_variantnum_equipmenttype_idx" tableName="newtemporarylimits">
<column name="networkuuid"/>
<column name="variantnum"/>
<column name="equipmenttype"/>
</createIndex>
</changeSet>

</databaseChangeLog>
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
-- will be used in next deployment
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking of doing the liquibase migration by doing a java change that just calls the same method as the one used for users

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. But the spring context is not available when the liquibase migration is executed and it makes things a bit more complicated to code. I built an "handmade" NetworkStoreRepository, not by Spring injection. To do so, I needed to build a javax.sql.DataSource and to retrieve the java.sql.Connection...

INSERT INTO newtemporarylimits (equipmentid, equipmenttype, networkuuid, variantnum, temporarylimits)
SELECT equipmentid, equipmenttype, networkuuid, variantnum,
json_agg(json_build_object(
'operationalLimitsGroupId', operationalLimitsGroupId,
'side', side,
'limitType', limitType,
'name', name,
'value', value_,
'acceptableDuration', acceptableduration,
'fictitious', fictitious
)) as temporarylimits FROM temporarylimit
GROUP BY equipmentid, equipmenttype, networkuuid, variantnum;
INSERT INTO newpermanentlimits (equipmentid, equipmenttype, networkuuid, variantnum, permanentlimits)
SELECT equipmentid, equipmenttype, networkuuid, variantnum,
json_agg(json_build_object(
'operationalLimitsGroupId', operationalLimitsGroupId,
'value', value_,
'side', side,
'limitType', limitType
)) as permanentlimits FROM permanentlimit
GROUP BY equipmentid, equipmenttype, networkuuid, variantnum;
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,7 @@ databaseChangeLog:
- include:
file: changesets/changelog_20241031T110000Z.xml
relativeToChangelogFile: true

- include:
file: changesets/changelog_20241121T110000Z.xml
relativeToChangelogFile: true
1 change: 0 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@
<properties>
<sirocco.version>1.0</sirocco.version>
<powsybl-ws-dependencies.version>2.16.0</powsybl-ws-dependencies.version>

<!-- FIXME : to remove when sonar version is updated on github actions -->
<!-- https://community.sonarsource.com/t/stackoverflowerror-at-defaultinputcomponent-equals/20324 -->
<!-- The versions are very different from this post. But the fix works again. Maybe a similar problem in sonar code -->
Expand Down
Loading