Merge branch 'main' of github.com:txpipe/oura

docs/pages/v2/advanced.mdx

@@ -2,10 +2,8 @@
 
 This section provides detailed information on the some of the advanced features available in Oura:
 
-- [Stateful Cursor](advanced/stateful_cursor.md): provides a mechanism to persist the "position" of the processing pipeline to make it resilient to restarts.
-- [Rollback Buffer](advanced/rollback_buffer.md): provides a way to mitigate the impact of chain rollbacks in downstream stages.
-- [Pipeline Metrics](advanced/pipeline_metrics.md): allows operators to track the progress and performance of long-running Oura sessions.
-- [Mapper Options](advanced/mapper_options.md): A set of "expensive" event mapping procedures that require an explicit opt-in to be activated.
-- [Intersect Options](advanced/intersect_options.md): Advanced options for instructing Oura from which point in the chain to start reading from.
-- [Custom Network](advanced/custom_network.md): Instructions on how to configure Oura for connecting to a custom network.
-- [Retry Policy](advanced/retry_policy.md): Instructions on how to configure retry policies for different operations
+- [Stateful Cursor](advanced/stateful_cursor): provides a mechanism to persist the "position" of the processing pipeline to make it resilient to restarts.
+- [Pipeline Metrics](advanced/pipeline_metrics): allows operators to track the progress and performance of long-running Oura sessions.
+- [Intersect Options](advanced/intersect_options): Advanced options for instructing Oura from which point in the chain to start reading from.
+- [Custom Network](advanced/custom_network): Instructions on how to configure Oura for connecting to a custom network.
+- [Retry Policy](advanced/retry_policy): Instructions on how to configure retry policies for different operations

docs/pages/v2/advanced/custom_network.mdx

@@ -1,53 +1,52 @@
 # Custom networks
 
-Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / testnet).
+Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / preprod / preview).
 
 ## Context
 
-Oura requires certain information about the chain it is reading from. In a way, this is similar to the json config files required to run the Cardano node. These values are used for procedures such as encoding bech32 addresses, computing wall-clock time for blocks, etc. 
+Oura requires certain information about the chain it is reading from. In a way, this is similar to the json config files required to run the Cardano node. These values are used for procedures such as encoding bech32 addresses, computing wall-clock time for blocks, etc.
 
-Since `mainnet` and `testnet` are well-known, heavily used networks, Oura hardcodes these values as part of the binary release so that the user is spared from having to manually specify them. On the other hand, custom networks require the user to configure these values manually for Oura to establish a connection.
+Since `mainnet`, `testnet`, `preprod` and `preview` are well-known, heavily used networks, Oura hardcodes these values as part of the binary release so that the user is spared from having to manually specify them. On the other hand, custom networks require the user to configure these values manually for Oura to establish a connection.
 
 ## Feature
 
 By adding a `[chain]` section in the daemon configuration file, users can provide the information required by Oura to connect to a custom network.
 
 The `[chain]` section has the following propoerties:
 
-| Name                 | DataType | Description                                                |
-| :------------------- | :------- | :--------------------------------------------------------- |
-| byron_epoch_length   | integer  | the length (in seconds) of a Byron epoch in this network   |
-| byron_slot_length    | integer  | the length (in seconds) of a Byron slot in this network    |
-| byron_known_slot     | integer  | the slot of a Byron block known to exist in this network   |
-| byron_known_hash     | string   | the hash of the known Byron block                          |
-| byron_known_time     | integer  | the unix timestamp of the known Byron block                |
-| shelley_epoch_length | integer  | the length (in seconds) of a Shelley epoch in this network |
-| shelley_slot_length  | integer  | the length (in seconds) of a Shelley slot in this network  |
-| shelley_known_slot   | integer  | the slot of a Shelley block known to exist in this network |
-| shelley_known_hash   | String   | the hash of the known Shelley block                        |
-| shelley_known_time   | integer  | the unix timestamp of the known Shelley block              |
-| address_hrp          | string   | the human readable part for addresses of this network      |
-| adahandle_policy     | string   | the minting policy for AdaHandle on this network.          |
-
+| Name                 | DataType | Description                                                   |
+| :------------------- | :------- | :------------------------------------------------------------ |
+| type                 | string   | types of network (mainnet, testnet, preprod, preview, custom) |
+| magic                | integer  | the network number                                            |
+| byron_epoch_length   | integer  | the length (in seconds) of a Byron epoch in this network      |
+| byron_slot_length    | integer  | the length (in seconds) of a Byron slot in this network       |
+| byron_known_slot     | integer  | the slot of a Byron block known to exist in this network      |
+| byron_known_hash     | string   | the hash of the known Byron block                             |
+| byron_known_time     | integer  | the unix timestamp of the known Byron block                   |
+| shelley_epoch_length | integer  | the length (in seconds) of a Shelley epoch in this network    |
+| shelley_slot_length  | integer  | the length (in seconds) of a Shelley slot in this network     |
+| shelley_known_slot   | integer  | the slot of a Shelley block known to exist in this network    |
+| shelley_known_hash   | String   | the hash of the known Shelley block                           |
+| shelley_known_time   | integer  | the unix timestamp of the known Shelley block                 |
 
 ## Examples
 
-### Chain information for Testnet
+### Chain information for mainnet
 
-This example configuration shows the values for Testnet. Since testnet values are hardcoded as part of Oura's release, users are not required to input these exact values anywhere, but it serves as a good example of what the configuration looks like.
+This example configuration shows the values for mainnet. Since testnet values are hardcoded as part of Oura's release, users are not required to input these exact values anywhere, but it serves as a good example of what the configuration looks like.
 
-```toml 
+```toml
 [chain]
+type = "custom"
+magic = 764824073
 byron_epoch_length  = 432000
 byron_slot_length = 20
 byron_known_slot = 0
-byron_known_hash = "8f8602837f7c6f8b8867dd1cbc1842cf51a27eaed2c70ef48325d00f8efb320f"
-byron_known_time = 1564010416
+byron_known_time = 1506203091
+byron_known_hash = "f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f"
 shelley_epoch_length = 432000
 shelley_slot_length = 1
-shelley_known_slot = 1598400
-shelley_known_hash = "02b1c561715da9e540411123a6135ee319b02f60b9a11a603d3305556c04329f"
-shelley_known_time = 1595967616
-address_hrp = "addr_test"
-adahandle_policy = "8d18d786e92776c824607fd8e193ec535c79dc61ea2405ddf3b09fe3"
+shelley_known_slot = 4492800
+shelley_known_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
+shelley_known_time = 1596059091
 ```

docs/pages/v2/advanced/finalize_options.mdx

@@ -0,0 +1,22 @@
+# Finalize Options
+
+Advanced options for instructing Oura to stop on a chain point. If you'd like it to only sync a specific section of the chain, you can also instruct oura to stop syncing when it reaches a specific block hash by defining a `[finalize]` config.
+
+## Configuration
+
+To modify the default behavior the daemon mode uses, a section named `[finalize]` needs to be added to the `daemon.toml` file.
+
+```toml
+[finalize]
+until_hash = <BlockHash>
+max_block_slot = <SlotNumber>
+```
+
+## Examples
+
+The following example show how to configure Oura to stop sync on Byron era
+
+```toml
+[finalize]
+until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
+```

docs/pages/v2/advanced/intersect_options.mdx

@@ -6,57 +6,37 @@ Advanced options for instructing Oura from which point in the chain to start rea
 
 When running in daemon mode, Oura provides 4 different strategies for finding the intersection point within the chain sync process.
 
-- `Origin`: Oura will start reading from the beginning of the chain.
 - `Tip`: Oura will start reading from the current tip of the chain.
-- `Point`: Oura will start reading from a particular point (slot, hash) in the chain. If the point is not found, the process will be terminated with a non-zero exit code.
-- `Fallbacks`: Oura will start reading the first valid point within a set of alternative positions. If point is not valid, the process will fallback into the next available point in the list of options. If none of the points are valid, the process will be terminated with a non-zero exit code.
+- `Origin`: Oura will start reading from the beginning of the chain.
+- `Point`: Oura will start reading from a particular point [slot, hash] in the chain.
+- `Breadcrumbs`: Oura will start reading from a some points [[slot, hash],[slot, hash]] in the chain.
 
 The default strategy use by Oura is `Tip`, unless an alternative option is specified via configuration.
 
 You can also define a finalizing point by providing a block hash at which oura will stop reading from the the chain and exit gracefully.
 
 ## Configuration
 
-To modify the default behaviour used by the daemon mode, a section named `[source.intersect]` needs to be added in the `daemon.toml` file.
+To modify the default behaviour used by the daemon mode, a section named `[intersect]` needs to be added in the `daemon.toml` file.
 
 ```toml
-[source.intersect]
+[intersect]
 type = <Type>
 value = <Value>
 ```
 
-- `type`: Defines which strategy to use. Valid values are `Origin`, `Tip`, `Point`, `Fallbacks`. Default value is `Tip`.
+- `type`: Defines which strategy to use. Valid values are `Origin`, `Tip`, `Point`, `Breadcrumbs`. Default value is `Tip`.
 - `value`: Either a point or an array of points to be used as argument for the selected strategy.
 
-If you'd like it to only sync an specific section of the chain, you can also instruct oura to stop syncing when it reaches an specific block hash by defining a `[source.finalize]` config:
-
-```toml
-[source.finalize]
-until_hash = <BlockHash>
-```
-
-Note that unlike the intersect point, no slot is provided for the finalizer.
-
 ## Examples
 
-The following example show how to configure Oura to use a set of fallback intersection point. The chain sync process will attempt to first intersect at slot `4449598`. If not found, it will continue with slot `43159` and finally with slot `0`.
+The following example show how to configure Oura to use a intersection point.
 
 ```toml
-[source.intersect]
-type = "Fallbacks"
+[intersect]
+type = "Point"
 value = [
-    [4449598, "2c9ba2611c5d636ecdb3077fde754413c9d6141c6288109922790e53bbb938b5"],
-    [43159, "f5d398d6f71a9578521b05c43a668b06b6103f94fcf8d844d4c0aa906704b7a6"],
-    [0, "f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f"],
+    4493860,
+    "ce7f821d2140419fea1a7900cf71b0c0a0e94afbb1f814a6717cff071c3b6afc",
 ]
 ```
-
-This configuration will sync the whole Byron era only:
-
-```toml
-[source.intersect]
-type = "Origin"
-
-[source.finalize]
-until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
-```

docs/pages/v2/filters.mdx

@@ -6,10 +6,10 @@ _Filters_ are intermediate steps in the pipeline that process events as they tra
 
 These are the existing filters that are included as part the main _Oura_ codebase:
 
-- [ParseCbor](filters/parse_cbor.mdx): a filter that maps cbor transactions to a data structure.
-
-### Legacy V1
-- [Fingerprint](filters/fingerprint.mdx): a filter that adds a (probably) unique identifier (fingerprint) to each event.
-- [Selection](filters/selection.mdx): a filter that evaluates different built-in predicates to decide which events should go through the pipeline.
+- [ParseCbor](filters/parse_cbor): a filter that maps cbor transactions to a data structure.
+- [SplitBlock](filters/split_block): a filter that will decode the cbor block and extract all transactions in an event in the format CborTx.
+- [Deno](filters/deno): a filter that allows JS code to be implemented as a stage within the pipeline.
+- [DSL](filters/dsl): a filter that can select which events to block and which to let pass.
+- [Legacy V1](filters/legacy_v1): a filter that transforms the block data to the Oura V1 data structure.
 
 New filters are being developed, information will be added in this documentation to reflect the updated list. Contributions and feature request are welcome in our [Github Repo](https://github.com/txpipe/oura).

docs/pages/v2/filters/deno.mdx

@@ -0,0 +1,26 @@
+# Deno filter
+
+The `deno` filter will allow JS code to be implemented as a stage within the pipeline.
+
+Therefore it will be possible to manipulate the block data dynamically and according to the specific rules of your business.
+
+## Configuration
+
+Adding the following section to the daemon config file will enable the filter as part of the pipeline:
+
+```toml
+[[filters]]
+type = "Deno"
+main_module = "JS file Path"
+use_async = true
+```
+
+The file should export a function that takes the event data as a parameter, for example
+
+```js
+export async function mapEvent(event) {
+    const event2 = { ...event, extraProp: "123abc" };
+    return event2;
+}
+```
+

docs/pages/v2/filters/dsl.mdx

@@ -0,0 +1,3 @@
+# DSL filter
+
+Coming Soon!