From 534ac417b06996c53a96e630cb3baef1bba88b11 Mon Sep 17 00:00:00 2001 From: Paulo Bressan Date: Mon, 14 Aug 2023 21:47:32 -0300 Subject: [PATCH] docs: improve documentation across the board (#669) --- docs/pages/v2/advanced.mdx | 12 +- docs/pages/v2/advanced/custom_network.mdx | 55 ++-- docs/pages/v2/advanced/finalize_options.mdx | 22 ++ docs/pages/v2/advanced/intersect_options.mdx | 42 +-- docs/pages/v2/advanced/mapper_options.mdx | 36 --- docs/pages/v2/advanced/rollback_buffer.mdx | 39 --- docs/pages/v2/filters.mdx | 7 +- docs/pages/v2/filters/deno.mdx | 26 ++ docs/pages/v2/filters/dsl.mdx | 3 + docs/pages/v2/filters/fingerprint.mdx | 22 -- docs/pages/v2/filters/legacy_v1.mdx | 57 ++++ docs/pages/v2/filters/parse_cbor.mdx | 33 +++ docs/pages/v2/filters/selection.mdx | 274 ------------------ docs/pages/v2/filters/split_block.mdx | 31 ++ docs/pages/v2/guides/cardano_2_kafka.mdx | 29 +- docs/pages/v2/index.mdx | 2 +- docs/pages/v2/installation.mdx | 8 +- docs/pages/v2/installation/docker.mdx | 11 +- docs/pages/v2/sinks.mdx | 23 +- docs/pages/v2/sinks/aws_sqs.mdx | 8 +- .../sinks/{elastic.mdx => elasticsearch.mdx} | 2 +- .../v2/sinks/{redis_streams.mdx => redis.mdx} | 0 docs/pages/v2/sinks/stdout.mdx | 12 + docs/pages/v2/sources.mdx | 5 +- docs/pages/v2/sources/n2c.mdx | 68 +---- docs/pages/v2/sources/n2n.mdx | 66 +---- docs/pages/v2/usage.mdx | 6 +- docs/pages/v2/usage/daemon.mdx | 45 ++- docs/pages/v2/usage/dump.mdx | 45 --- docs/pages/v2/usage/watch.mdx | 63 ---- 30 files changed, 302 insertions(+), 750 deletions(-) create mode 100644 docs/pages/v2/advanced/finalize_options.mdx delete mode 100644 docs/pages/v2/advanced/mapper_options.mdx delete mode 100644 docs/pages/v2/advanced/rollback_buffer.mdx create mode 100644 docs/pages/v2/filters/deno.mdx create mode 100644 docs/pages/v2/filters/dsl.mdx delete mode 100644 docs/pages/v2/filters/fingerprint.mdx create mode 100644 docs/pages/v2/filters/legacy_v1.mdx create mode 100644 docs/pages/v2/filters/parse_cbor.mdx delete mode 100644 docs/pages/v2/filters/selection.mdx create mode 100644 docs/pages/v2/filters/split_block.mdx rename docs/pages/v2/sinks/{elastic.mdx => elasticsearch.mdx} (82%) rename docs/pages/v2/sinks/{redis_streams.mdx => redis.mdx} (100%) create mode 100644 docs/pages/v2/sinks/stdout.mdx delete mode 100644 docs/pages/v2/usage/dump.mdx delete mode 100644 docs/pages/v2/usage/watch.mdx diff --git a/docs/pages/v2/advanced.mdx b/docs/pages/v2/advanced.mdx index 3d183cdd..b0fe6c03 100644 --- a/docs/pages/v2/advanced.mdx +++ b/docs/pages/v2/advanced.mdx @@ -2,10 +2,8 @@ This section provides detailed information on the some of the advanced features available in Oura: -- [Stateful Cursor](advanced/stateful_cursor.md): provides a mechanism to persist the "position" of the processing pipeline to make it resilient to restarts. -- [Rollback Buffer](advanced/rollback_buffer.md): provides a way to mitigate the impact of chain rollbacks in downstream stages. -- [Pipeline Metrics](advanced/pipeline_metrics.md): allows operators to track the progress and performance of long-running Oura sessions. -- [Mapper Options](advanced/mapper_options.md): A set of "expensive" event mapping procedures that require an explicit opt-in to be activated. -- [Intersect Options](advanced/intersect_options.md): Advanced options for instructing Oura from which point in the chain to start reading from. -- [Custom Network](advanced/custom_network.md): Instructions on how to configure Oura for connecting to a custom network. -- [Retry Policy](advanced/retry_policy.md): Instructions on how to configure retry policies for different operations +- [Stateful Cursor](advanced/stateful_cursor): provides a mechanism to persist the "position" of the processing pipeline to make it resilient to restarts. +- [Pipeline Metrics](advanced/pipeline_metrics): allows operators to track the progress and performance of long-running Oura sessions. +- [Intersect Options](advanced/intersect_options): Advanced options for instructing Oura from which point in the chain to start reading from. +- [Custom Network](advanced/custom_network): Instructions on how to configure Oura for connecting to a custom network. +- [Retry Policy](advanced/retry_policy): Instructions on how to configure retry policies for different operations diff --git a/docs/pages/v2/advanced/custom_network.mdx b/docs/pages/v2/advanced/custom_network.mdx index 5bfe8280..399687d3 100644 --- a/docs/pages/v2/advanced/custom_network.mdx +++ b/docs/pages/v2/advanced/custom_network.mdx @@ -1,12 +1,12 @@ # Custom networks -Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / testnet). +Instructions on how to configure Oura for connecting to a custom network (aka: other than mainnet / preprod / preview). ## Context -Oura requires certain information about the chain it is reading from. In a way, this is similar to the json config files required to run the Cardano node. These values are used for procedures such as encoding bech32 addresses, computing wall-clock time for blocks, etc. +Oura requires certain information about the chain it is reading from. In a way, this is similar to the json config files required to run the Cardano node. These values are used for procedures such as encoding bech32 addresses, computing wall-clock time for blocks, etc. -Since `mainnet` and `testnet` are well-known, heavily used networks, Oura hardcodes these values as part of the binary release so that the user is spared from having to manually specify them. On the other hand, custom networks require the user to configure these values manually for Oura to establish a connection. +Since `mainnet`, `testnet`, `preprod` and `preview` are well-known, heavily used networks, Oura hardcodes these values as part of the binary release so that the user is spared from having to manually specify them. On the other hand, custom networks require the user to configure these values manually for Oura to establish a connection. ## Feature @@ -14,40 +14,39 @@ By adding a `[chain]` section in the daemon configuration file, users can provid The `[chain]` section has the following propoerties: -| Name | DataType | Description | -| :------------------- | :------- | :--------------------------------------------------------- | -| byron_epoch_length | integer | the length (in seconds) of a Byron epoch in this network | -| byron_slot_length | integer | the length (in seconds) of a Byron slot in this network | -| byron_known_slot | integer | the slot of a Byron block known to exist in this network | -| byron_known_hash | string | the hash of the known Byron block | -| byron_known_time | integer | the unix timestamp of the known Byron block | -| shelley_epoch_length | integer | the length (in seconds) of a Shelley epoch in this network | -| shelley_slot_length | integer | the length (in seconds) of a Shelley slot in this network | -| shelley_known_slot | integer | the slot of a Shelley block known to exist in this network | -| shelley_known_hash | String | the hash of the known Shelley block | -| shelley_known_time | integer | the unix timestamp of the known Shelley block | -| address_hrp | string | the human readable part for addresses of this network | -| adahandle_policy | string | the minting policy for AdaHandle on this network. | - +| Name | DataType | Description | +| :------------------- | :------- | :------------------------------------------------------------ | +| type | string | types of network (mainnet, testnet, preprod, preview, custom) | +| magic | integer | the network number | +| byron_epoch_length | integer | the length (in seconds) of a Byron epoch in this network | +| byron_slot_length | integer | the length (in seconds) of a Byron slot in this network | +| byron_known_slot | integer | the slot of a Byron block known to exist in this network | +| byron_known_hash | string | the hash of the known Byron block | +| byron_known_time | integer | the unix timestamp of the known Byron block | +| shelley_epoch_length | integer | the length (in seconds) of a Shelley epoch in this network | +| shelley_slot_length | integer | the length (in seconds) of a Shelley slot in this network | +| shelley_known_slot | integer | the slot of a Shelley block known to exist in this network | +| shelley_known_hash | String | the hash of the known Shelley block | +| shelley_known_time | integer | the unix timestamp of the known Shelley block | ## Examples -### Chain information for Testnet +### Chain information for mainnet -This example configuration shows the values for Testnet. Since testnet values are hardcoded as part of Oura's release, users are not required to input these exact values anywhere, but it serves as a good example of what the configuration looks like. +This example configuration shows the values for mainnet. Since testnet values are hardcoded as part of Oura's release, users are not required to input these exact values anywhere, but it serves as a good example of what the configuration looks like. -```toml +```toml [chain] +type = "custom" +magic = 764824073 byron_epoch_length = 432000 byron_slot_length = 20 byron_known_slot = 0 -byron_known_hash = "8f8602837f7c6f8b8867dd1cbc1842cf51a27eaed2c70ef48325d00f8efb320f" -byron_known_time = 1564010416 +byron_known_time = 1506203091 +byron_known_hash = "f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f" shelley_epoch_length = 432000 shelley_slot_length = 1 -shelley_known_slot = 1598400 -shelley_known_hash = "02b1c561715da9e540411123a6135ee319b02f60b9a11a603d3305556c04329f" -shelley_known_time = 1595967616 -address_hrp = "addr_test" -adahandle_policy = "8d18d786e92776c824607fd8e193ec535c79dc61ea2405ddf3b09fe3" +shelley_known_slot = 4492800 +shelley_known_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de" +shelley_known_time = 1596059091 ``` diff --git a/docs/pages/v2/advanced/finalize_options.mdx b/docs/pages/v2/advanced/finalize_options.mdx new file mode 100644 index 00000000..8a9a10dd --- /dev/null +++ b/docs/pages/v2/advanced/finalize_options.mdx @@ -0,0 +1,22 @@ +# Finalize Options + +Advanced options for instructing Oura to stop on a chain point. If you'd like it to only sync a specific section of the chain, you can also instruct oura to stop syncing when it reaches a specific block hash by defining a `[finalize]` config. + +## Configuration + +To modify the default behavior the daemon mode uses, a section named `[finalize]` needs to be added to the `daemon.toml` file. + +```toml +[finalize] +until_hash = +max_block_slot = +``` + +## Examples + +The following example show how to configure Oura to stop sync on Byron era + +```toml +[finalize] +until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de" +``` diff --git a/docs/pages/v2/advanced/intersect_options.mdx b/docs/pages/v2/advanced/intersect_options.mdx index d7979fdf..4ae8749b 100644 --- a/docs/pages/v2/advanced/intersect_options.mdx +++ b/docs/pages/v2/advanced/intersect_options.mdx @@ -6,10 +6,10 @@ Advanced options for instructing Oura from which point in the chain to start rea When running in daemon mode, Oura provides 4 different strategies for finding the intersection point within the chain sync process. -- `Origin`: Oura will start reading from the beginning of the chain. - `Tip`: Oura will start reading from the current tip of the chain. -- `Point`: Oura will start reading from a particular point (slot, hash) in the chain. If the point is not found, the process will be terminated with a non-zero exit code. -- `Fallbacks`: Oura will start reading the first valid point within a set of alternative positions. If point is not valid, the process will fallback into the next available point in the list of options. If none of the points are valid, the process will be terminated with a non-zero exit code. +- `Origin`: Oura will start reading from the beginning of the chain. +- `Point`: Oura will start reading from a particular point [slot, hash] in the chain. +- `Breadcrumbs`: Oura will start reading from a some points [[slot, hash],[slot, hash]] in the chain. The default strategy use by Oura is `Tip`, unless an alternative option is specified via configuration. @@ -17,46 +17,26 @@ You can also define a finalizing point by providing a block hash at which oura w ## Configuration -To modify the default behaviour used by the daemon mode, a section named `[source.intersect]` needs to be added in the `daemon.toml` file. +To modify the default behaviour used by the daemon mode, a section named `[intersect]` needs to be added in the `daemon.toml` file. ```toml -[source.intersect] +[intersect] type = value = ``` -- `type`: Defines which strategy to use. Valid values are `Origin`, `Tip`, `Point`, `Fallbacks`. Default value is `Tip`. +- `type`: Defines which strategy to use. Valid values are `Origin`, `Tip`, `Point`, `Breadcrumbs`. Default value is `Tip`. - `value`: Either a point or an array of points to be used as argument for the selected strategy. -If you'd like it to only sync an specific section of the chain, you can also instruct oura to stop syncing when it reaches an specific block hash by defining a `[source.finalize]` config: - -```toml -[source.finalize] -until_hash = -``` - -Note that unlike the intersect point, no slot is provided for the finalizer. - ## Examples -The following example show how to configure Oura to use a set of fallback intersection point. The chain sync process will attempt to first intersect at slot `4449598`. If not found, it will continue with slot `43159` and finally with slot `0`. +The following example show how to configure Oura to use a intersection point. ```toml -[source.intersect] -type = "Fallbacks" +[intersect] +type = "Point" value = [ - [4449598, "2c9ba2611c5d636ecdb3077fde754413c9d6141c6288109922790e53bbb938b5"], - [43159, "f5d398d6f71a9578521b05c43a668b06b6103f94fcf8d844d4c0aa906704b7a6"], - [0, "f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f"], + 4493860, + "ce7f821d2140419fea1a7900cf71b0c0a0e94afbb1f814a6717cff071c3b6afc", ] ``` - -This configuration will sync the whole Byron era only: - -```toml -[source.intersect] -type = "Origin" - -[source.finalize] -until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de" -``` diff --git a/docs/pages/v2/advanced/mapper_options.mdx b/docs/pages/v2/advanced/mapper_options.mdx deleted file mode 100644 index 00ab50d2..00000000 --- a/docs/pages/v2/advanced/mapper_options.mdx +++ /dev/null @@ -1,36 +0,0 @@ -# Mapper Options - -A set of "expensive" event mapping procedures that require an explicit opt-in to be activated. - -## Context - -One of the main concerns of Oura is turning block / tx data into atomic events to send down the pipeline for further processing. The `source` stage is responsible for executing these mapping procedures. - -Most of the time, this logic is generic enough that it can be reused in different scenarios. For example, the `N2N` and the `N2C` sources share the same mapping procedures. If a particular use-case needs to cherry-pick, enrich or alter the data in some way, the recommendation is to handle the transformation in downstream stages, by using any of the built-in filter or by creating new ones. - -There are some exceptions though, whenever a mapping has a heavy impact on performance, it is better to disable it completely at the `source` level to avoid paying the overhead associated with the initial processing of the data. - -## Feature - -We consider a mapping procedure "expensive" if it involves: handling a relative large amount of data, computing some relatively expensive value or generating redundant data required only for very particular use cases. - -For these expensive procedures, we provide configurable options that instructs an Oura instance running in daemon mode to opt-in on each particular rule. - -## Configuration - -The mapper options can be defined by adding the following configuration in the `daemon.toml` file: - -```toml -[source.mapper] -include_block_end_events = -include_transaction_details = -include_transaction_end_events = -include_block_cbor = -include_byron_ebb = -``` - -- `include_block_end_events`: if enabled, the source will output an event signaling the end of a block, duplicating all of the data already sent in the corresponding block start event. Default value is `false`. -- `include_transaction_details`: if enabled, each transaction event payload will contain an nested version of all of the details of the transaction (inputs, outputs, mint, assets, metadata, etc). Useful when the pipeline needs to process the tx as a unit, instead of handling each sub-object as an independent event. Default value is `false`. -- `include_transaction_end_events`: if enabled, the source will output an event signaling the end of a transaction, duplicating all of the data already sent in the corresponding transaction start event. Defaul value is `false`. -- `include_block_cbor`: if enabled, the block event will include the raw, unaltered cbor content received from the node, formatted as an hex string. Useful when some custom cbor decoding is required. Default value is `false`. -- `include_byron_ebb`: if enabled, a block event will be emmitted for legacy epoch boundary block of the Byron era (deprecated in newer eras). Useful when performing validation on previous block hashes. Default value is `false`. diff --git a/docs/pages/v2/advanced/rollback_buffer.mdx b/docs/pages/v2/advanced/rollback_buffer.mdx deleted file mode 100644 index a9364c68..00000000 --- a/docs/pages/v2/advanced/rollback_buffer.mdx +++ /dev/null @@ -1,39 +0,0 @@ -# Rollback Buffer - -The "rollback buffer" feature provides a way to mitigate the impact of chain rollbacks in downstream stages of the data-processing pipeline. - -## Context - -Handling rollbacks in a persistent storage requires clearing the orphaned data / blocks before adding new records. The complexity of this process may vary by concrete storage engine, but it always has an associated impact on performance. In some scenarios, it might even be prohibitive to process events without a reasonable level of confidence about the immutability of the record. - -Rollbacks occur frequently under normal conditions, but the chances of a block becoming orphaned diminishes as the depth of the block increases. Some Oura use-cases may benefit from this property, some pipelines might prefer lesser rollback events, even if it means waiting for a certain number of confirmations. - -## Feature - -Oura provides a "rollback buffer" that will hold blocks in memory until they reach a certain depth. Only blocks above a min depth threshold will be sent down the pipeline. If a rollback occurs and the intersection is within the scope of the buffer, the rollback operation will occur within memory, totally transparent to the subsequent stages of the pipeline. - -If a rollback occurs and the intersection is outside of the scope of the buffer, Oura will fallback to the original behaviour and publish a RollbackEvent so that the "sink" stages may handle the rollback procedure manually. - -## Trade-off - -There's an obvious trade-off to this approach: latency. A pipeline will not process any events until the buffer fills up. Once the initial wait is over, the throughput of the whole pipeline should be equivalent to having no buffer at all (due to Oura's "pipelining" nature). If a rollback occurs, an extra delay will be required to fill the buffer again. - -Notice that even if the throughput isn't affected, the latency (measured as the delta between the timestamp at which the event reaches the "sink" stage and the original timestamp of the block) will always be affected by a fixed value proportional to the size of the buffer. - -## Implementation Details - -The buffer logic is implemented in pallas-miniprotocols library. It works by keeping a VecDeque data structure of chain "points", where roll-forward operations accumulate at the end of the deque and retrieving confirmed points means popping from the front of the deque. - -## Configuration - -The min depth is a configurable setting available when running in daemon mode. Higher min_depth values will lower the chances of experiencing a rollback event, at the cost of adding more latency. An node-to-node source stage config would look like this: - -```toml -[source] -type = "N2N" -address = ["Tcp", "relays-new.cardano-mainnet.iohk.io:3001"] -magic = "mainnet" -min_depth = 6 -``` - -Node-to-client sources provide an equivalent setting. diff --git a/docs/pages/v2/filters.mdx b/docs/pages/v2/filters.mdx index 34c9c412..0b20226d 100644 --- a/docs/pages/v2/filters.mdx +++ b/docs/pages/v2/filters.mdx @@ -6,7 +6,10 @@ _Filters_ are intermediate steps in the pipeline that process events as they tra These are the existing filters that are included as part the main _Oura_ codebase: -- [Fingerprint](filters/fingerprint.md): a filter that adds a (probably) unique identifier (fingerprint) to each event. -- [Selection](filters/selection.md): a filter that evaluates different built-in predicates to decide which events should go through the pipeline. +- [ParseCbor](filters/parse_cbor): a filter that maps cbor transactions to a data structure. +- [SplitBlock](filters/split_block): a filter that will decode the cbor block and extract all transactions in an event in the format CborTx. +- [Deno](filters/deno): a filter that allows JS code to be implemented as a stage within the pipeline. +- [DSL](filters/dsl): a filter that can select which events to block and which to let pass. +- [Legacy V1](filters/legacy_v1): a filter that transforms the block data to the Oura V1 data structure. New filters are being developed, information will be added in this documentation to reflect the updated list. Contributions and feature request are welcome in our [Github Repo](https://github.com/txpipe/oura). diff --git a/docs/pages/v2/filters/deno.mdx b/docs/pages/v2/filters/deno.mdx new file mode 100644 index 00000000..690bf0bd --- /dev/null +++ b/docs/pages/v2/filters/deno.mdx @@ -0,0 +1,26 @@ +# Deno filter + +The `deno` filter will allow JS code to be implemented as a stage within the pipeline. + +Therefore it will be possible to manipulate the block data dynamically and according to the specific rules of your business. + +## Configuration + +Adding the following section to the daemon config file will enable the filter as part of the pipeline: + +```toml +[[filters]] +type = "Deno" +main_module = "JS file Path" +use_async = true +``` + +The file should export a function that takes the event data as a parameter, for example + +```js +export async function mapEvent(event) { + const event2 = { ...event, extraProp: "123abc" }; + return event2; +} +``` + diff --git a/docs/pages/v2/filters/dsl.mdx b/docs/pages/v2/filters/dsl.mdx new file mode 100644 index 00000000..4847265c --- /dev/null +++ b/docs/pages/v2/filters/dsl.mdx @@ -0,0 +1,3 @@ +# DSL filter + +Coming Soon! diff --git a/docs/pages/v2/filters/fingerprint.mdx b/docs/pages/v2/filters/fingerprint.mdx deleted file mode 100644 index d872c445..00000000 --- a/docs/pages/v2/filters/fingerprint.mdx +++ /dev/null @@ -1,22 +0,0 @@ -# Fingerprint Filter - -A filter that computes a (probably) unique identifier for each event and appends it as part of the data passed forward. - -Dealing with duplicated records is a common problem in data pipelines. A common workaround is to use identifiers based on the hash of the data for each record. In this way, a duplicated record would yield the same hash, allowing the storage engine to discard the extra instance. - -The _fingerprint_ filter uses the non-cryptographic hash algorithm `murmur3` to compute an id for each _Oura_ event with a very low collision level. We use a non-cryptographic hash because they are faster to compute and non of the cryptographic properties are required for this use-case. - -When enabled, this filter will set the `fingerprint` property of the `Event` data structure passed through each stage of the pipeline. _Sinks_ at the end of the process might leverage this value as primary key of the corresponding storage mechanism. - -## Configuration - -Adding the following section to the daemon config file will enable the filter as part of the pipeline: - -```toml -[[filters]] -type = "Fingerprint" -``` - -### Section: `filter` - -- `type`: the literal value `Fingerprint`. diff --git a/docs/pages/v2/filters/legacy_v1.mdx b/docs/pages/v2/filters/legacy_v1.mdx new file mode 100644 index 00000000..4aaa1748 --- /dev/null +++ b/docs/pages/v2/filters/legacy_v1.mdx @@ -0,0 +1,57 @@ +# Legacy V1 filter + +Transforms the block data to the Oura V1 data structure. This filter is ideal for compatibility with systems that are using Oura V1 and want to migrate without much work. + +## Configuration + +Adding the following section to the daemon config file will enable the filter as part of the pipeline: + +```toml +[[filters]] +type = "LegacyV1" +include_block_end_events = false +include_transaction_details = false +include_transaction_end_events = false +include_block_details = false +include_block_cbor = false +include_byron_ebb = false +``` + +### Section `LegacyV1` + +- `type`: the literal value `LegacyV1`. +- `include_block_end_events`: if enabled, the source will output an event signaling the end of a block, duplicating all of the data already sent in the corresponding block start event. Default value is `false`. +- `include_transaction_details`: if enabled, each transaction event payload will contain an nested version of all of the details of the transaction (inputs, outputs, mint, assets, metadata, etc). Useful when the pipeline needs to process the tx as a unit, instead of handling each sub-object as an independent event. Default value is `false`. +- `include_transaction_end_events`: if enabled, the source will output an event signaling the end of a transaction, duplicating all of the data already sent in the corresponding transaction start event. Defaul value is `false`. +- `include_block_cbor`: if enabled, the block event will include the raw, unaltered cbor content received from the node, formatted as an hex string. Useful when some custom cbor decoding is required. Default value is `false`. +- `include_byron_ebb`: if enabled, a block event will be emmitted for legacy epoch boundary block of the Byron era (deprecated in newer eras). Useful when performing validation on previous block hashes. Default value is `false`. +- `include_block_details`: If enabled, will be added the basic details of each transaction. Default value is `false`. + +## Examples + +Below is an example of the data that will be sent to the sink with all settings disabled. + +```json +{ + "event": "apply", + "point": { + "slot": 100110525, + "hash": "c808fc4142c5f10a2a6d0922edbd23972100d7d22e2255206bd05e968cc045f1" + }, + "record": { + "context": { + "block_hash": "c808fc4142c5f10a2a6d0922edbd23972100d7d22e2255206bd05e968cc045f1", + "block_number": 9142145, + "slot": 100110525, + "timestamp": 1691676816, + "tx_idx": 6, + "tx_hash": "4329140c6711f2197c8c81bfff4b75fb95892375050dafda30ba146476ca3d65", + "input_idx": null, + "output_idx": null, + "output_address": null, + "certificate_idx": null + }, + ... + } +} +``` diff --git a/docs/pages/v2/filters/parse_cbor.mdx b/docs/pages/v2/filters/parse_cbor.mdx new file mode 100644 index 00000000..c2fd8b98 --- /dev/null +++ b/docs/pages/v2/filters/parse_cbor.mdx @@ -0,0 +1,33 @@ +# Parse CBOR filter + +The `parse_cbor` filter aims to map cbor transactions to a structured transaction. + +However, the filter will only work when the record received in the stage is CborTx in other words a transaction in Cbor format that was previously extracted from a block by another stage, otherwise, parse_cbor will ignore and pass the record to the next stage. When the record is CborTx, parse_cbor will decode and map the Cbor to a structure, so the next stage will receive the ParsedTx record. If no filter is enabled, the stages will receive the record in CborBlock format, and if only the parse_cbor filter is enabled in `daemon.toml`, it will be necessary to enable the [split_cbor](split_block) filter for the stage to receive the CborTx format. + +## Configuration + +Adding the following section to the daemon config file will enable the filter as part of the pipeline: + +```toml +[[filters]] +type = "ParseCbor" +``` + +## Examples + +Below is an example of the data that will be sent to the sink. A block can contain many transactions, so the sink will receive an event for each transaction in json format. + +```json +{ + "event": "apply", + "point": { + "slot": 0, + "hash": "" + }, + "record": { + "inputs": [], + "outputs": [], + ... + } +} +``` diff --git a/docs/pages/v2/filters/selection.mdx b/docs/pages/v2/filters/selection.mdx deleted file mode 100644 index d7359e81..00000000 --- a/docs/pages/v2/filters/selection.mdx +++ /dev/null @@ -1,274 +0,0 @@ -### Table of contents -- [Selection filter](#selection-filter) -- [General predicates](#general-predicates) -- [Variant-restricted predicates](#variant-restricted-predicates) -- [Real world example](#real-world-example) -# Selection filter - -A filter that evaluates a set of configurable predicates against each event in the pipeline to decide which records should be sent to the following stage. - -Not every use-case requires each and every event to be processed. For example, a pipeline interested in creating a 'metadata' search engine might not care about transaction outputs. With a similar logic, a pipeline aggregating transaction amounts might not care about metadata. The _selection filter_ provides a way to optimize the pipeline so that only relevant events are processed. - -The filter works by evaluating a predicate against each event. If the predicate returns `true`, then the event will continue down the pipeline. If the predicate evalutes to `false`, the event will be dopped. We currently provide some common built-in predicate to facilitate common use-cases (eg: matching event type, matching policy id, matching a metadata key, etc.). We also provide some 'connecting' predicates like `all_of`, `any_of`, and `not` which can be used to create complex conditions by composing other predicates. - -## Configuration - -Adding the following section to the daemon config file will enable the filter as part of the pipeline: - -```toml -[[filters]] -type = "Selection" - -[filters.check] -predicate = "" -argument = -``` - -### Section: `filters` - -- `type`: the literal value `Selection`. - -### Section `filters.check` - -- `predicate`: the key of the predicate to use for the evaluation. See the list of available predicates for possible values. -- `argument`: a polimorphic argument that specializes the behavior of the predicate in some way. - -## General predicates -Predicates available for events of any type. - -### `variant_in (string[])` - This predicate will yield true when the variant of the event matches any of the items in the argument array. Available variants: -- [Block](../reference/data_dictionary.md#block-event) -- [CIP15Asset](../reference/data_dictionary.md#cip15asset-event) -- [CIP25Asset](../reference/data_dictionary.md#cip25asset-event) -- [Collateral](../reference/data_dictionary.md#collateral-event) -- [GenesisKeyDelegation](../reference/data_dictionary.md#genesiskeydelegation-event) -- [Metadata](../reference/data_dictionary.md#metadata-event) -- [Mint](../reference/data_dictionary.md#mint-event) -- [MoveInstantaneousRewardsCert](../reference/data_dictionary.md#moveinstantaneousrewardscert-event) -- [NativeScript](../reference/data_dictionary.md#nativescript-event) -- [NativeWitness](../reference/data_dictionary.md#nativewitness-event) -- [OutputAsset](../reference/data_dictionary.md#outputasset-event) -- [PlutusDatum](../reference/data_dictionary.md#plutusdatum-event) -- [PlutusRedeemer](../reference/data_dictionary.md#plutusredeemer-event) -- [PlutusScript](../reference/data_dictionary.md#plutusscript-event) -- [PlutusWitness](../reference/data_dictionary.md#plutuswitness-event) -- [PoolRegistration](../reference/data_dictionary.md#poolregistration-event) -- [PoolRetirement](../reference/data_dictionary.md#poolretirement-event) -- [RollBack](../reference/data_dictionary.md#rollback-event) -- [StakeDelegation](../reference/data_dictionary.md#stakedelegation-event) -- [StakeDeregistration](../reference/data_dictionary.md#stakederegistration-event) -- [StakeRegistration](../reference/data_dictionary.md#stakeregistration-event) -- [Transaction](../reference/data_dictionary.md#transaction-event) -- [TxInput](../reference/data_dictionary.md#txinput-event) -- [TxOutput](../reference/data_dictionary.md#txoutput-event) -- [VKeyWitness](../reference/data_dictionary.md#vkeywitness-event) - -**Example** - Allowing only block and transaction events to pass: - -```toml -[[filters]] -type = "Selection" - -[filters.check] -predicate = "variant_in" -argument = ["Block", "Transaction"] -``` - -### `variant_not_in (string[])` - This predicate will yield true when the variant of the event doesn't match any of the items in the argument array. - -**Example** - Allowing all events except transaction to pass: -```toml -[[filters]] -type = "Selection" - -[filters.check] -predicate = "variant_not_in" -argument = ["Transaction"] -``` - -### `not (predicate)` - This predicate will yield true when the predicate in the arguments yields false. - - -**Example** - Using the `not` predicate to allow all events except the variant `Transaction`: - -```toml -[[filters]] -type = "Selection" - -[filters.check] -predicate = "not" - -[filters.check.argument] -predicate = "variant_in" -argument = ["Transaction"] -``` - -### `any_of (predicate[])` - This predicate will yield true when _any_ of the predicates in the argument yields true. - - -**Example** - Using the `any_of` predicate to filter events presenting any of two different policies (Boolean "or"): - -```toml -[filters.check] -predicate = "any_of" - -[[filters.check.argument]] -predicate = "policy_equals" -argument = "4bf184e01e0f163296ab253edd60774e2d34367d0e7b6cbc689b567d" - -[[filters.check.argument]] -predicate = "policy_equals" -argument = "a5bb0e5bb275a573d744a021f9b3bff73595468e002755b447e01559" -``` - -### `all_of (predicate[])` - This predicate will yield true when _all_ of the predicates in the argument yields true. - -**Example** - Using the `all_of` predicate to filter only "asset" events presenting a particular policy (Boolean "and") : - -```toml -[filters.check] -predicate = "all_of" - -[[filters.check.argument]] -predicate = "variant_in" -argument = ["OutputAsset"] - -[[filters.check.argument]] -predicate = "policy_equals" -argument = "a5bb0e5bb275a573d744a021f9b3bff73595468e002755b447e01559" -``` - -## Variant-restricted predicates -Predicates operating on a subset of event variants. - - -### `policy_equals (string)` - This predicate will yield true when the policy of a mint or output asset matches the value in the argument. - - -**Variants:** `Transaction`, `Mint`, `CIP25Asset`, `OutputAsset` - -**Example** -```toml -[[filters]] -type = "Selection" - -[filters.check] -predicate = "policy_equals" -argument = "" -``` - -### `asset_equals (string)` - This predicate will yield true when the asset (token name) of a mint or output asset matches the value in the argument. - -**Variants:** `CIP25Asset`, `Transaction`, `OutputAsset`, `Mint` - -**Example** -```toml -[[filters]] -type = "Selection" - -[filters.check] -predicate = "asset_equals" -argument = "" -``` - -### `metadata_label_equals (string)` - This predicate will yield true when the root label of a metadata entry matches the value in the argument. - -**Variants:** `Metadata`, `Transaction` - -**Example** -```toml -[[filters]] -type = "Selection" - -[filters.check] -predicate = "metadata_label_equals" -argument = "