PowSyBl - AFS

AFS stands for Application FileSystem.

An AFS is meant to be used to organize your business data and store them, like a file system does for plain files.

The structure of an AFS looks like:

   AppData
     +-- FileSystem1
     |   +-- File1
     |   +-- File2
     |   +-- Project1
     |   |   +-- RootFolder
     |   |       +-- ProjectFile1
     |   |       +-- ProjectFolder1
     |   |       |   +-- ProjectFile2
     |   |       +-- ProjectFolder2
     |   |           +-- ProjectFile3
     |   +-- Project2
     |      ...
     |
     +-- FileSystem2
         ...

where each "project file" may represent a business object, for instance a network or a list of contingencies, or even a computation.

Below, you will learn:

• to use the AFS API in your application
• to extend and customize AFS for your needs

1- Using AFS

Using AFS in your java application

In order to use AFS in your application, you will first need to add some dependencies to your project:

• powsybl-afs-core to use the core API in your code
• an actual implementation available at runtime: powsybl comes with powsyl-afs-mapdb for prototyping
• basic file types defined as extensions of the core: powsybl-afs-ext-base.

For instance, if you use maven, in the dependencies section:

<dependency>
  <groupId>com.powsybl</groupId>
  <artifactId>powsybl-afs-core</artifactId>
</dependency>
<dependency>
  <groupId>com.powsybl</groupId>
  <artifactId>powsybl-afs-ext-base</artifactId>
</dependency>
<dependency>
  <groupId>com.powsybl</groupId>
  <artifactId>powsybl-afs-mapdb</artifactId>
  <scope>runtime</scope>
</dependency>

By default, powsybl will load application file system depending on your platform configuration (file ${HOME}/.itools/config.yml). To tell AFS where it should store its data, you will need to configure your mapdb file system:

mapdb-app-file-system:
  drive-name : my-first-fs
  db-file : /path/to/my/mapdb/file

or in XML:

<mapdb-app-file-system>
  <drive-name>my-first-fs</drive-name>
  <db-file>/path/to/my/mapdb/file</db-file>
</mapdb-app-file-system>

Here, we have defined a mapdb based file system, which will be named my-first-fs in your application, and stored in the file /path/to/my/mapdb/file.

Now, from your application, you will be able to interact with that file system. For example, you can create directories and projects:

//Build an instance of AppData
ComputationManager c = LocalComputationManager.getDefault(); //Do not pay attention to this part
AppData appData = new AppData(c, c);

//Get your file system
AppFileSystem myFirstFs = appData.getFileSystem("my-first-fs");
//Create a new folder at the root of you file system, and a new project in that folder.
Project myFirstProject = myFirstFs.getRootFolder()
         .createFolder("my-first-folder")
         .createProject("my-first-project");

Everything that we have just created will be persisted to your mapdb file. Your file system tree now looks like:

my-first-fs
  +-- my-first-folder
    +--my-first-project

Using AFS from groovy scripts

Your configured AFS is also accessible from groovy. This comes in 2 flavours, either with an interactive console using the powsybl shell powsyblsh:

powsyblsh
groovy:000> :register com.powsybl.scripting.groovy.InitPowsybl
groovy:000> :init_powsybl
groovy:000> import com.powsybl.contingency.*
groovy:000>

or using the itools command to execute a groovy script:

itools run-script --file my_script.groovy

From groovy code, powsybl provides a variable called afs which exposes base methods to access configured file systems. You can then simply perform the same thing as in the java section this way:

//Create a new folder at the root of you file system, and a new project in that folder.
myFirstProject = afs.getRootFolder("my-first-fs")
                    .createFolder("my-first-folder")
                    .createProject("my-first-project");

Using actual business objects

All this is fine, but the primary goal of AFS is to manage your business objects, which we have not seen so far.

AFS is fully extendable with your own type of files, but it already comes with a few basic types for grid studies. The most basic one may be the ImportedCase type, which expose a Network object to the API.

Such files may only be created inside a project. Projects may be seen as a kind of workspace for a particular study or computation. Inside of a project, we can import a case from a file representing a network, for example an XIIDM file or a UCTE file.

In java:

ImportedCase myImportedCase = myFirstProject.getRootFolder()
              .fileBuilder(ImportedCaseBuilder.class)
              .withName("my-imported-case")
              .withFile(Paths.get("path/to/network.xiidm"))
              .build();

or in groovy with a nice, simplified syntax:

myImportedCase = myFirstProject.getRootFolder().buildImportedCase {
              name "my-imported-case"
              file Paths.get("path/to/network.xiidm")
            }

Now our tree looks like:

my-first-fs
  +-- my-first-folder
    +-- my-first-project
      +-- "my-imported-case"

You can then use the methods exposed by your imported case to carry out some business related logic:

//use stored network
Network network = myImportedCase.getNetwork();
//Carry out some computations ...
...
//Query stored network to get all substations names
System.out.println(myImportedCase.queryNetwork(ScriptType.GROOVY,
                            "network.substationStream.map {it.name} collect()"))

Of course, you can later retrieve your imported case from another execution or from another application, once it has been persisted to the underlying storage:

//From another application:
ImportedCase importedCase = appData.getFileSystem("my-first-fs")
                                   .getRootFolder()
                                   .getChild(Project.class, "my-first-folder/my-first-project").get()
                                   .getRootFolder().getChild("my-imported-case")
                                   .orElseThrow(() -> new RuntimeException("Not found"));

Network network = myImportedCase.getNetwork();
//Do some stuff with network
...

Using a remote file system

Powsybl provides a special implementation of application file system storage which forwards calls, through a REST API, to a remote AFS server. The server may use any storage implementation itself, for example the mapdb implementation.

That feature makes it easy to store data on a remote server.

In order to use it you will need to:

• package in a war and deploy powsybl-afs-ws-server in a JEE server, like Wildfly
• configure app file systems in the server powsybl configuration file, for instance defining a mapdb file system as above. You will need to add support for remote acces by setting the additional remotely-accessible parameter to true:

 mapdb-app-file-system:
   drive-name : my-first-fs
   db-file : /path/to/my/mapdb/file
   remotely-accessible: true

• add powsybl-afs-ws-client to the runtime dependencies of your client application
• configure in the powsybl configuration of your application the following rest file system:

  remote-service:
    host-name: my-afs-server
    app-name: my-server-app
    port: 8080
    secure: false

Now all file systems defined in the server configurations will be transparently accessible from your client application, without changing any of your code!

This allows for great flexibility in the deployment of your application, for instance to run the same application as standalone or client/server.

Note: The underlying REST API is documented at http://my-afs-server:8080/my-server-app/rest/swagger

Listeners

TO BE COMPLETED

Services

TO BE COMPLETED

Dependencies

TO BE COMPLETED

2- Extending AFS for your needs

2.1 Adding your own file types

Powsybl-afs already comes with a number of project file types:

• Imported cases and virtual cases
• Contingency stores
• Action scripts
• Security analysis runner: an object from which you can launch a security analysis

However, you will probably want to extend that list with your own type of data or computation. AFS provides an easy way to do so.

Each project file type in AFS relies on the definition of 3 classes:

• The actual new type, which must extends ProjectFile
• A builder class for the new type, which must extends ProjectFile
• An extension class, in charge of registering the new type and its builder to AFS. It must implement the ProjectFileExtension interface, and be registered through the use of @AutoService annotation.

First define you new type :

class FooFile extends ProjectFile {

    FooFile(ProjectFileCreationContext context) {
        super(context, 0);
    }

    public void doSomeCoolStuff() { ... }
    public Foo getSomeCoolData() { ... }
    public FooResult runSomeCoolComputation() { ... }
}

Then define how to build it. This involves at least creating a new node in the underlying storage, and may involve writing some data blob or defining depencencies to other nodes:

class FooFileBuilder implements ProjectFileBuilder<FooFile> {

    private final ProjectFileBuildContext context;
    private String name;

    FooFileBuilder(ProjectFileBuildContext context) {
        this.context = Objects.requireNonNull(context);
    }

    public FooFileBuilder withName(String name) {
        this.name = name;
        return this;
    }

    //Storeds new node information in underlying storage, and builds actual object
    @Override
    public FooFile build() {
        if (name == null) {
            throw new IllegalArgumentException("name is not set");
        }
        String pseudoClass = "foo";

        //Create new node in storage
        NodeInfo info = context.getStorage().createNode(context.getFolderInfo().getId(), name, pseudoClass, "", 0, new NodeGenericMetadata());

        //Possibly, write data to storage
        try (OutputStream os = context.getStorage().writeBinaryData(info.getId(), "my_data")) {
            os.write(...);
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }

        //Possibly, add dependencies to other nodes
        context.getStorage().addDependency(info.getId(), "my_dependency", dependencyNodeId);

        //Flush modifications
        context.getStorage().flush();

        //Return new object
        return new FooFile(new ProjectFileCreationContext(info,
                                                          context.getStorage(),
                                                          context.getProject()));
    }
}

Finally, register your new type and its builder type through and extension class:

@AutoService(ProjectFileExtension.class)
public class FooFileExtension implements ProjectFileExtension<FooFile, FooFileBuilder> {
    @Override
    public Class<FooFile> getProjectFileClass() {
        return FooFile.class;
    }

    //Define the name of this type for AFS
    @Override
    public String getProjectFilePseudoClass() {
        return "foo";
    }

    @Override
    public Class<FooFileBuilder> getProjectFileBuilderClass() {
        return FooFileBuilder.class;
    }

    @Override
    public FooFile createProjectFile(ProjectFileCreationContext context) {
        return new FooFile(context);
    }

    @Override
    public FooFileBuilder createProjectFileBuilder(ProjectFileBuildContext context) {
        return new FooFileBuilder(context);
    }
}

Notice the definition of a "pseudo class" name, which is the name of the type of this project file as known by the AFS storage.

And that's it! You will now be able to use this new type of project file as any other. For instance, to create one in groovy:

myFooFile = myFirstProject.getRootFolder().buildFoo { name "my-foo-file" }
//use it!
myFooFile.runSomeCoolComputation()

2.2 Adding your own storage implementation

TODO

Using Maven Wrapper

If you don't have a proper Maven installed, you could use the Apache Maven Wrapper scripts provided. They will download a compatible maven distribution and use it automatically.

Configuration

Configure the access to the maven distributions

In order to work properly, Maven Wrapper needs to download 2 artifacts: the maven distribution and the maven wrapper distribution. By default, these are downloaded from the online Maven repository, but you could use an internal repository instead.

Using a Maven Repository Manager

If you prefer to use an internal Maven Repository Manager instead of retrieving the artefacts from the internet, you should define the following variable in your environment:

• MVNW_REPOURL: the URL to your repository manager (for instance https://my_server/repository/maven-public)

Note that if you need to use this variable, it must be set for each maven command. Else, the Maven Wrapper will try to retrieve the maven distribution from the online Maven repository (even if one was already downloaded from another location).

Using a proxy to access the Internet

If you don't use an internal Maven Repository, and need to use a proxy to access the Internet, you should:

1. configure the proxy in your terminal (on Linux/MacOS, you can do it via the http_proxy and https_proxy environment variables). This is needed to download the Maven Wrapper distribution ;

2. execute at least once the following command:

./mvnw -DproxyHost=XXX -DproxyPort=XXX -Dhttp.proxyUser=XXX -Dhttp.proxyPassword=XXX -Djdk.http.auth.tunneling.disabledSchemes= clean

Notes:

• The 4 XXX occurrences should be replaced with your configuration;
• The -Djdk.http.auth.tunneling.disabledSchemes= option should be left empty;
• Windows users should use mvnw.cmd instead of ./mwn.

This second step is required to download the Maven distribution.

Once both distributions are retrieved, the proxy configuration isn't needed anymore to use ./mvnw or mvnw.cmd commands.

Checking your access configuration

You could check your configuration with the following command:

./mvnw -version

If you encounter any problem, you could specify MVNW_VERBOSE=true and relaunch the command to have further information.

Configuring install.sh to use maven wrapper

To indicate install.sh to use Maven Wrapper, you need to configure it with the --mvn option:

./install.sh clean --mvn ./mvnw

You can revert this configuration with the following command:

./install.sh clean --mvn mvn

Usage

Once the configuration is done, you just need to use ./mvnw instead of mvn in your commands.