Merge pull request #1461 from SamMayWork/1.3-docs

docs: Prepare documentation for FireFly v1.3.0

.github/workflows/docs.yml

@@ -50,7 +50,7 @@ jobs:
       - name: Update doc site for latest release
         if: ${{ github.event_name == 'release' && env.IS_LATEST_RELEASE == 'true' }}
         working-directory: doc-site
-        run: mike deploy ${{ github.event.release.tag_name }} latest --push
+        run: mike deploy ${{ github.event.release.tag_name }} latest -u --push
 
       - name: Update doc site for `main` branch
         if: ${{ github.event_name == 'push' }}

.ruby-version

@@ -0,0 +1 @@
+3.1.2

doc-site/docs/contributors/docs_setup.md

@@ -82,18 +82,10 @@ The commands you will usually use with `mkdocs` are:
 - `mkdocs build` - Build the documentation site.
 - `mkdocs -h` - Print help message and exit.
 
-## Adding Content
+## Directory layout
 
-The basic process for adding content to the site is:
-
-- Create a new markdown file under the `docs` folder
-- Add the new file to the table of contents (`nav` section in the `mkdocs.yml` file)
-
-If you are using this as a template for creating your own documentation, please see [the instructions for customization](./docs/index.md).
-
-## Repository layout
-
-    mkdocs.yml    # The configuration file.
+    mkdocs.yml     # The configuration file.
     docs/
-        index.md  # The documentation homepage.
-        ...       # Other markdown pages, images and other files.
+        index.md   # The documentation homepage.
+        SUMMARY.md # The main left nav
+        ...        # Other markdown pages, images and other files.

doc-site/docs/gettingstarted/sandbox.md

@@ -89,7 +89,7 @@ The Messages tab is where we can send broadcast and private messages to other me
 
 ## Tokens
 
-The Tokens tab is where you can create token pools, and mint, burn, or transfer tokens. This works with both fungible and non-fungible tokens (NFTs). For more details, please see the [Tokens tutorials](../tutorials/tokens).
+The Tokens tab is where you can create token pools, and mint, burn, or transfer tokens. This works with both fungible and non-fungible tokens (NFTs). For more details, please see the [Tokens tutorials](../tutorials/tokens/index.md).
 
 ### Things to try out
 

doc-site/docs/index.md

@@ -7,15 +7,3 @@ Hyperledger FireFly is an [open source Supernode](./overview/supernode_concept.m
 The easiest way to understand a FireFly Supernode is to think of it like a toolbox. Connect your existing apps and/or back office systems to the toolbox and within it there are two different sets of tools. One set of tools helps you connect to the Web3 world that already exists, and the other set allows you to build new decentralized applications quickly with security and scalability.
 
 Head to the [Understanding FireFly](./overview/supernode_concept.md) section for more details.
-
-## Documentation Sections
-
-- [Understanding FireFly](./overview/)
-- [Getting Started](./gettingstarted/)
-- [Tutorials](./tutorials/)
-- [Reference](./reference/)
-- [Architecture](./architecture/)
-- [Contributors](./contributors/)
-- [API Spec](./swagger/)
-- [FAQs](./faqs/)
-- [Release notes](./release_notes/)

doc-site/docs/overview/key_components/apps.md

@@ -30,7 +30,7 @@ FireFly provides APIs that:
 - Conform as closely as possible to the principles of REST
 - Do not pretend to be RESTful in cases when it is impossible to be
 
-> Learn more about **deploying APIs for custom smart contracts** in [this tutorial](../../tutorials/custom_contracts/)
+> Learn more about **deploying APIs for custom smart contracts** in [this tutorial](../../tutorials/custom_contracts/index.md)
 
 ### Event Streams
 

doc-site/docs/releasenotes/1.3_migration_guide.md

@@ -0,0 +1,106 @@
+---
+title: v1.3.0 Migration Guide
+---
+
+## Overview
+
+Hyperledger FireFly v1.3.0 is a feature release that includes changes around event streaming, contract listeners, define/publish APIs as well as a range of general fixes.
+
+For the most part, upgrading from v1.2.x to v1.3.0 should be a seamless experience, but there are several important things to note about changes between the two versions, which are described in detail on this page.
+
+## Docker image file permission considerations
+
+Following security best practices, the official published Docker images for FireFly Core and all of its microservices now run as a non-root user by default. If you are running a FireFly release prior to v1.3.0, depending on how you were running your containers, you may need to adjust file permissions inside volumes that these containers write to. If you have overridden the default user for your containers (for example though a Kubernetes deployment) you may safely ignore this section.
+
+> ⚠️ **Warning**: If you have been using the default root user and upgrade to FireFly v1.3.0 without changing these file permissions your services may fail to start.
+
+The new default user is `1001`. If you are not overriding the user for your container, this user or group needs to have write permissions in several places. The list of services and directories you should specifically check are:
+
+- **firefly-evmconnect**
+  - `persistence.leveldb.path` directory set in the config file
+- **firefly-ethconnect**
+  - `rest.rest-gateway.openapi.storagePath` directory in the config file
+  - `rest.rest-gateway.openapi.eventsDB` directory in the config file
+- **firefly-fabconnect**
+  - `receipts.leveldb.path` directory in the config file
+  - `events.leveldb.path` directory in the config file
+- **firefly-dataexchange-https**
+  - Data directory set by the `DATA_DIRECTORY` environment variable (default `/data`)
+
+## API definition/publication considerations
+
+**As of FireFly v1.3.0 in multi-party namespaces, by default, contract interfaces, contracts APIs, and token pools have distinct steps in their creation flow and by default they are unpublished.**
+
+These following described changes impact contract interfaces, contract APIs, and token pools.
+
+Previously, when creating one of the affected resources in a multi-party network, if successful, the resource would be automatically broadcasted to other namespaces. In FireFly v1.3.0, this behaviour has changed, now when one of the resources is created there are 2 distinct states for the resource, _published_ and _unpublished_. The default state for a resource (provided FireFly is not told otherwise) after creation is _unpublished_.
+
+When a resource is _unpublished_ it is not broadcasted to other namespaces in the multi-party network, and it is not pinned to the blockchain. In this state, it is possible to call the `DELETE` APIs to remove the resource (such as in the case where configuration needs to be changed) and reclaim the name that has been provided to it, so that it can be recreated.
+
+When a resource is _published_ it is broadcasted to other namespaces in the multi-party network, and it is pinned to the blockchain. In this state, it is no longer possible to call the `DELETE` APIs to remove the resource.
+
+In FireFly v1.2.0 to create one of the affected resources and publish it to other parties, a `POST` call would be made to its respective API route and the broadcast would happen immediately. To achieve the same behaviour in FireFly v1.3.0, there are 2 options for all impacted resources, either providing a query parameter at creation to signal immediate publish, or a subsequent API call to publish the resources.
+
+### Contract interfaces
+
+Previously, to create a contract interface a `POST` call would be made to `/contracts/interfaces` and the interface would be broadcasted to all other namepsaces. In FireFly v1.3.0, this same call can be made with the `publish=true` query parameter, or a subsequent API call can be made on an unpublished interface on `POST /contracts/interfaces/{name}/{version}/publish` specifying the name and version of the interface.
+
+[For an exact view of the changes to contract interfaces, see PR #1279.](https://github.com/hyperledger/firefly/pull/1279)
+
+### Contract APIs
+
+Previously, to create a contract API a `POST` call would be made to `/apis` and the API would be broadcasted to all other namepsaces. In FireFly v1.3.0, this same call can be made with the `publish=true` query parameter, or a subsequent API call can be made on an unpublished API on `/apis/{apiName}/publish` specifying the name of the API.
+
+[For an exact view of the changes to contract APIs, see PR #1322.](https://github.com/hyperledger/firefly/pull/1322)
+
+### Token pools
+
+Previously, to create a token pool a `POST` call would be made to `/tokens/pools` and the token pool would be broadcasted to all other namepsaces. In FireFly v1.3.0, this same call can be made with the `publish=true` query parameter, or a subsequent API call can be made on an unpublished token pool on `/tokens/pools/{nameOrId}/publish` specifying the name or ID of the token pool.
+
+[For an exact view of the changes to token pools, see PR #1261.](https://github.com/hyperledger/firefly/pull/1261)
+
+## Event stream considerations
+
+### Single event stream per namespace
+
+In this release, the model for event streams in a multi-party network has fundamentally changed. Previously, there was a single event stream for each blockchain plugin, even if this plugin served multiple namespaces. In FireFly v1.3.0 there is now a single event stream per namespace in the network.
+
+When migrating from FireFly v1.2.X to v1.3.0, due to these changes, existing event streams will be rebuilt. This means that connectors will replay past events to FireFly, but FireFly will automatically de-duplicate them by design so this is a safe operation.
+
+The migration to individual event streams promotes high-availability capability but is not itself a breaking change, however the ID format for event streams has changed. Event streams now follow the format `<plugin_topic_name>/<namespace_name>`. For example, an event stream for the default namespace with a plugin topic of 0 would now be: `0/default`.
+
+Summarily, these changes _should not_ impact end-users of FireFly, but they're noted here as they are significant architectural changes to the relationships between namespaces, plugins, and connectors.
+
+[For an exact view of the changes, see PR #1388.](https://github.com/hyperledger/firefly/pull/1388)
+
+## Configuration considerations
+
+### Deprecated configuration
+
+In FireFly v1.3.0 deprecated configuration options for the `blockchain`, `database`, `dataexchange`, `sharedstorage` and `tokens` plugins have been removed, and can no longer be provided.
+
+[For an exact view of the changes, see PR #1289](https://github.com/hyperledger/firefly/pull/1289).
+
+## Token pool considerations
+
+### Activity indicator changes
+
+Token pools have a status, when creating a token pool previously, it would go into a pending state immediately following creation, and then into a confirmed state when it has been confirmed on the chain. This behaviour is still consistent in FireFly v1.3.0, but the representation of the data has changed.
+
+Previously, token pools had a `state` field with an enumerated value which was either `pending`, or `confirmed`, this has been replaced with an `active` boolean field, where `true` indicates the token pool has been committed onto chain, and `false` indicated the transaction has not yet been confirmed.
+
+[For an exact view of the changes, see PR #1305](https://github.com/hyperledger/firefly/pull/1305).
+
+## FabConnect event considerations
+
+### FabConnect Protocol ID format changes
+
+Prior to FireFly v1.3.0, when the FabConnect client indexed events submitted by the Fabric SDK, FireFly would deduplicate events into a single event because the protocol ID of the events compiled into a single block would evaluate to be the same. In this release, we have changed the format of the calculated protocol ID so that is unique across events even if they are located within the same block. Crucially, **the new format includes the transaction hash, so events are no longer alphanumerically sortable.**
+
+[For an exact view of the changes, see PR #1345](https://github.com/hyperledger/firefly/pull/1345).
+
+## Local development considerations
+
+### Go version upgrade
+
+FireFly v1.3.0 now uses Go 1.21 across all modules.

doc-site/docs/releasenotes/index.md

@@ -4,13 +4,25 @@ title: Release Notes
 
 [Full release notes](https://github.com/hyperledger/firefly/releases)
 
+## [v1.3.0 - April 25, 2024](https://github.com/hyperledger/firefly/releases/tag/v1.1.0)
+
+[Migration guide](1.3_migration_guide.md)
+
+What's New:
+
+- Enhancements to FireFly Messaging capabilities, allowing off-chain data to be linked to on-chain transactions from custom smart contracts
+- Sample contract implementation for custom pin contracts
+- Contract interfaces, contract APIs, and token pools can now be separately defined and published
+- Support for batching events when delivering over websockets
+- Lots of bug fixes and miscellaneous enhancements
+
 ## [v1.2.0 - February 6, 2023](https://github.com/hyperledger/firefly/releases/tag/v1.2.0)
 
 [Migration guide](1.2_migration_guide.md)
 
 What's New:
 
-- Enhanced support for token contracts generated by the OpenZepplin Wizard
+- Enhanced support for token contracts generated by the OpenZeppelin Wizard
 - Custom smart contract error types are now returned on the API
 - Data objects and associated blobs can now be deleted
 - Optional dynamic reload of core configuration file

doc-site/docs/tutorials/chains/tezos_testnet.md

@@ -88,4 +88,4 @@ On the **Transfers** tab from you account page you will see the actual transfer
 
 ## Use the public testnet
 
-Now that you have everything set up, you can follow one of the other FireFly guides such as [Custom Smart Contracts](../custom_contracts/tezos.md). For detailed instructions on deploying a custom smart contract to Tezos, please see the [Tezos docs](https://docs.tezos.com/smart-contracts/deploying) for instructions using various tools.
+Now that you have everything set up, you can follow one of the other FireFly guides such as [Custom Smart Contracts](../custom_contracts/index.md). For detailed instructions on deploying a custom smart contract to Tezos, please see the [Tezos docs](https://docs.tezos.com/smart-contracts/deploying) for instructions using various tools.

doc-site/docs/tutorials/custom_contracts/ethereum.md

@@ -342,7 +342,9 @@ We will take the output from the previous HTTP response above, **fill in the nam
 
 ### Request
 
-`POST` `http://localhost:5000/api/v1/namespaces/default/contracts/interfaces`
+`POST` `http://localhost:5000/api/v1/namespaces/default/contracts/interfaces?publish=true`
+
+> *NOTE*: Without passing the query parameter `publish=true` when the interface is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the interface, a subsequent API call would need to be made to `/contracts/interfaces/{name}/{version}/publish`
 
 ```json
 {
@@ -506,7 +508,8 @@ We will take the output from the previous HTTP response above, **fill in the nam
         }
       ]
     }
-  ]
+  ],
+  "published": true
 }
 ```
 
@@ -520,7 +523,9 @@ We need to copy the `id` field we got in the response from the previous step to
 
 ### Request
 
-`POST` `http://localhost:5000/api/v1/namespaces/default/apis`
+`POST` `http://localhost:5000/api/v1/namespaces/default/apis?publish=true`
+
+> *NOTE*: Without passing the query parameter `publish=true` when the API is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the API, a subsequent API call would need to be made to `/apis/{apiName}/publish`
 
 ```json
 {
@@ -551,7 +556,8 @@ We need to copy the `id` field we got in the response from the previous step to
   "urls": {
     "openapi": "http://127.0.0.1:5000/api/v1/namespaces/default/apis/simple-storage/api/swagger.json",
     "ui": "http://127.0.0.1:5000/api/v1/namespaces/default/apis/simple-storage/api"
-  }
+  },
+  "published": true
 }
 ```
 

doc-site/docs/tutorials/custom_contracts/fabric.md

@@ -187,7 +187,9 @@ We will use the FFI JSON constructed above and `POST` that to the `/contracts/in
 
 ### Request
 
-`POST` `http://localhost:5000/api/v1/namespaces/default/contracts/interfaces`
+`POST` `http://localhost:5000/api/v1/namespaces/default/contracts/interfaces?publish=true`
+
+> *NOTE*: Without passing the query parameter `publish=true` when the interface is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the interface, a subsequent API call would need to be made to `/contracts/interfaces/{name}/{version}/publish`
 
 ```json
 {
@@ -374,7 +376,9 @@ We need to copy the `id` field we got in the response from the previous step to
 
 ### Request
 
-`POST` `http://localhost:5000/api/v1/namespaces/default/apis`
+`POST` `http://localhost:5000/api/v1/namespaces/default/apis?publish=true`
+
+> *NOTE*: Without passing the query parameter `publish=true` when the API is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the API, a subsequent API call would need to be made to `/apis/{apiName}/publish`
 
 ```json
 {

doc-site/docs/tutorials/tokens/erc1155.md

@@ -39,7 +39,9 @@ At this point you could open the Sandbox at [http://127.0.0.1:5109/home?action=t
 
 After your stack is up and running, the first thing you need to do is create a token pool. Every application will need at least one token pool. At a minimum, you must always specify a `name` and `type` (`fungible` or `nonfungible`) for the pool.
 
-`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools`
+`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools?publish=true`
+
+> *NOTE*: Without passing the query parameter `publish=true` when the token pool is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the token pool, a subsequent API call would need to be made to `/tokens/pools/{nameOrId}/publish`
 
 ```json
 {
@@ -62,7 +64,9 @@ See the [README](https://github.com/hyperledger/firefly-tokens-erc1155/blob/main
 
 You can pass a `config` object with an `address` when you make the request to create the token pool, and if you created a contract interface, you can include the `interface` ID as well.
 
-`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools`
+`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools?publish=true`
+
+> *NOTE*: Without passing the query parameter `publish=true` when the token pool is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the token pool, a subsequent API call would need to be made to `/tokens/pools/{nameOrId}/publish`
 
 ```json
 {

doc-site/docs/tutorials/tokens/erc20.md

@@ -29,7 +29,9 @@ If you're using the default ERC-20 / ERC-721 token connector and its sample toke
 
 #### Request
 
-`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools`
+`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools?publish=true`
+
+> _NOTE_: Without passing the query parameter `publish=true` when the token pool is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the token pool, a subsequent API call would need to be made to `/tokens/pools/{nameOrId}/publish`
 
 ```json
 {
@@ -51,7 +53,8 @@ If you're using the default ERC-20 / ERC-721 token connector and its sample toke
   "tx": {
     "type": "token_pool",
     "id": "e901921e-ffc4-4776-b20a-9e9face70a47"
-  }
+  },
+  "published": true
 }
 ```
 
@@ -106,7 +109,9 @@ You can pass a `config` object with an `address` and `blockNumber` when you make
 
 #### Request
 
-`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools`
+`POST` `http://127.0.0.1:5000/api/v1/namespaces/default/tokens/pools?publish=true`
+
+> _NOTE_: Without passing the query parameter `publish=true` when the token pool is created, it will initially be unpublished and not broadcasted to other members of the network (if configured in multi-party). To publish the token pool, a subsequent API call would need to be made to `/tokens/pools/{nameOrId}/publish`
 
 ```json
 {