diff --git a/client/docs/interface/remote-client.md b/client/docs/interface/remote-client.md index 801bb63b..20a1a6de 100644 --- a/client/docs/interface/remote-client.md +++ b/client/docs/interface/remote-client.md @@ -1,27 +1,37 @@ # Remote Client -This is the default use-case for using Ulixee Client. You supply a connection URI or Object to the datastore you want to query. +This is the default use-case for using Ulixee Client. You supply a connection URI or Object to the datastore you want to query. You can also initialize clients with a local Datastore, Table, Extractor or Crawler instance, however, these clients provide a more limited set of properties and methods than what is shown on this page. See [Local Client](./local-client.md). ## Constructor -### new Client _(uriOrObject)_ {#constructor} +### new Client _(uriOrObject, config?)_ {#constructor} Creates a new Client instance. #### **Arguments**: - uri `string` | `Object`. A connection string in the format of `ulx://USERNAME:PASSWORD@HOST:PORT/DB`. You can also supply -an object with the following properties: - - username `string` + an object with the following properties: + - username `string` - password `string` - host `string` - port `number` - database `string` +- config `Object`. Optional. Configuration options + - paymentService `IPaymentService`. Optional. A [payment service](/docs/datastore/basics/payments#payment-services) to use for transactions. + - argonMainchainUrl `string`. Optional. The RPC URL of an Argon Mainchain node to use for looking up notary information for micropayment channel holds. + - authentication `Object`. Optional. An object with the following properties: + - identity `string`. A bech32 encoded Ed25519 key + - signature `string`. A signature of the identity + - nonce `string`. A nonce to prevent replay attacks + - affiliateId `string`. Optional. An affiliate ID to allow your queries to be tracked (often used in cloned Datastores to request funding) + - onQueryResult `(IDatastoreQueryResult) => void`. Optional. A callback function that will be called with the result of every query. The result has the latestVersion, metadata, and output/error. + - queryId `string`. Optional. A unique identifier for the query. If not provided, one will be generated. ```js -import Client from '@ulixee/client-playground'; +import Client from '@ulixee/client'; const client = new Client({ username: 'test', @@ -31,6 +41,22 @@ const client = new Client({ database: 'test', }); ``` + +Example using a Payment Service: + +```typescript +import { DefaultPaymentService } from '@ulixee/databroker'; +import Client from '@ulixee/client'; + +const paymentService = await DefaultPaymentService.fromBroker('wss://broker.testnet.ulixee.org', { + pemPath: 'path to your Identity pem file', +}); +const client = new Client('ulx://UsCPI.Stats/v1.0.0', { + paymentService, + argonMainchainUrl: 'wss://rpc.testnet.argonprotocol.org', +}); +``` + ## Properties ### client.username {#username} @@ -39,35 +65,30 @@ The username authenticated with the remote server. #### **Type**: `string` - ### client.password {#password} The password used to authenticate with the remote server. #### **Type**: `string` - ### client.host {#host} The host of the remote server. Defaults to localhost. #### **Type**: `string` - ### client.port {#port} The port sent of the remote server. Defaults to 1818. #### **Type**: `number` - ### client.database {#database} The name of the datastore the client is connected. #### **Type**: `string` - ## Methods ### client.query _(sql, boundValues)_ {#query} @@ -118,7 +139,6 @@ const records = await client.fetch('daysUntilWorldDomination', { probability: 5 #### **Returns**: `Promise` - ### client.crawl _(crawlerName, inputFilter)_ {#crawl} Trigger one of the Datastore's crawlers: diff --git a/datastore/docs/advanced/credits.md b/datastore/docs/advanced/credits.md index bfe66046..56ac7961 100644 --- a/datastore/docs/advanced/credits.md +++ b/datastore/docs/advanced/credits.md @@ -12,7 +12,7 @@ To embed credits, you can configure the [remoteDatastoreEmbeddedCredits](../basi ## Denominations -Ulixee Payments (in this case, Credits) come in the following denominations: +Argon Payments (in this case, Credits) come in the following denominations: - _Argon_: ~1 USD adjusted for inflation. - _Milligon_: ~1 thousand of a USD adjusted for inflation ($0.001). diff --git a/datastore/docs/basics/payments.md b/datastore/docs/basics/payments.md new file mode 100644 index 00000000..8fee0c6c --- /dev/null +++ b/datastore/docs/basics/payments.md @@ -0,0 +1,108 @@ +# Datastore Payments + +Ulixee Datastores accept two forms of micropayments out of the box: Argons and a Credit system. A Payment Service allows you to customize how payments are allocated for your queries. + +You can create your own payment service by implementing the [IPaymentService](https://github.com/ulixee/platform/tree/main/datastore/main/interfaces/IPaymentService.ts) interface. + +## Concepts + +### Denominations + +Argon Payments come in the following denominations: + +- _Argon_: ~1 USD adjusted for inflation. +- _Milligon_: ~1 thousand of a USD adjusted for inflation (about $0.001). +- _Microgon_: ~1 millionth of a USD adjusted for inflation (about $0.000,001). This is the denomination used by a query (eg, [Extractor.basePrice](./extractor.md#constructor). + +### Micropayment ChannelHolds + +A micropayment ChannelHold is a temporary hold on a Localchain that reserves a set amount of Argons (let's say, 10 Argon). When a user sets aside funds in their Localchain for a ChannelHold, their account cannot be modified for 1 hour (in Argon, these are 60 "ticks" representing an agreed upon minute of clock time). + +The smallest unit in Argon is a milligon, which is 1/1000th of an Argon. But in Ulixee Micropayments, payments are allowed to go as low as a microgon, which is 1/1,000,000th of an Argon. This allows for a price per query model to work with huge volumes, while still keeping the cost per query reasonable. + +During the hour, the ChannelHold sender and Datastore can exchange data for payment. There is no way to break the agreement early. Every time another milligon is spent (1/1000th of an argon), the Datastore will require an updated "settlement" indicating 1 more milligon is authorized for payment. This way, at all times, the Datastore is assured that the 10 Argons are legitimate, and the user knows the Datastore can only claim the funds that have been authorized. + +After the hour is up, the recipient of the funds (the datastore) submits the signed settlement to a notary and moves those funds to their Localchain. + +The Datastore and User were able to exchange data for payment with volumes only limited by each other's machines and network connections. Only the beginning and final settlement need to be sent to the broader Argon network. This allows Ulixee to achieve a high volume of micropayments without overwhelming the Argon network. + +### Credits + +Datastores come built-in with a credits model. This is a little like a free trial mode if you want to hand out data credits to people trying out your Datastore. Credits are not a payment method, but a way to allocate free queries. A payment service will keep track of how many credits are available and prioritize them before charging for Argons. + +## Payment Services + +Payment services are used by [Clients](/docs/client) and Datastores to manage payments. They can be used to allocate Argons, manage credits, and track payments. + +### DefaultPaymentService + +The [default payment service](https://github.com/ulixee/platform/blob/42bc301bb24f1697ea60bca2db9258fe469e0212/datastore/main/payments/DefaultPaymentService.ts#L25) combines an Argon payment service with a Credit payment service. Argon payments can come from a Localchain on the same computer, a [Data broker](./databrokers) or a Remote Service. + +You will interact with this class in two primary ways: + +#### 1. With a Localchain + +If you are running a Localchain on the same computer, you can use the `fromLocalchain` method to create a payment service that will automatically allocate Argons from the Localchain. + +```typescript +import { + DefaultPaymentService, + IChannelHoldAllocationStrategy, + LocalchainWithSync, +} from '@ulixee/datastore'; + +// This strategy will create batches of 200 queries worth of argons per ChannelHold (1 hour). +const channelHoldAllocationStrategy: IChannelHoldAllocationStrategy = { + type: 'multiplier', + queries: 200, +}; +const bobchain = await LocalchainWithSync.load({ + localchainName: 'bobchain', + channelHoldAllocationStrategy, +}); +const paymentService = DefaultPaymentService.fromLocalchain(bobchain); +``` + +#### 2. With a Data Broker + +A [Databroker](./databrokers) is a service that manages Argons for you. You can use the `fromBroker` method to create a payment service that will automatically allocate Argons from the Data Broker. + +```typescript +import { DefaultPaymentService, IChannelHoldAllocationStrategy } from '@ulixee/datastore'; + +// This strategy will create batches of 200 queries worth of argons per channelHold (1 hour). +const channelHoldAllocationStrategy: IChannelHoldAllocationStrategy = { + type: 'multiplier', + queries: 200, +}; +const paymentService = await DefaultPaymentService.fromBroker( + 'wss://broker.testnet.ulixee.org', + { + pemPath: 'path to your Identity pem file', + }, + channelHoldAllocationStrategy, +); +``` + +### EmbeddedPaymentService + +When you [Clone](./cloning) a Datastore that requires payment, your CloudNode needs to establish Micropayment Channels with any upstream datastore(s). The embedded payment service works with a local (or cluster) Localchain to establish payments with limited permissions. This service will automatically only whitelist the upstream Datastore sources and Datastore IDs listed in the cloned Datastore. + +To enable the EmbeddedPaymentService, you either need to have a configured Localchain on the same machine, or you'll need to configure a Hosted Service to manage the Localchain for you. + +Configure a Localchain with the [`Localchain` configurations](http://localhost:8080/docs/datastore/overview/configuration#payment-configuration), or if you're setting up a CloudNode in a cluster, you would set the `ULX_UPSTREAM_PAYMENTS_SERVICE_HOST` environment variable, pointing to your Hosted Services node. (NOTE: you can also just configure your child with the `ULX_SERVICES_SETUP_HOST` environment variable set to your Hosted Services node). + +Lead node: + +```bash +$ npx @ulixee/cloud start --hosted-services-port 18181 \ + --argon-localchain-path /path/to/localchain \ + --argon-mainchain-url wss://rpc.testnet.argonprotocol.org \ + --argon-block-rewards-address 5DRTmdnaztvtdZ56QbEmHM8rqUR2KiKh7KY1AeMfyvkPSb5S +``` + +Child node: + +```bash +$ npx @ulixee/cloud start --setup-host :18181 +``` diff --git a/datastore/docs/guides/close-argon-blocks.md b/datastore/docs/guides/close-argon-blocks.md new file mode 100644 index 00000000..1cefac2a --- /dev/null +++ b/datastore/docs/guides/close-argon-blocks.md @@ -0,0 +1,63 @@ +# Close Argon Blocks + +> Your Localchain will automatically convert your settled micropayments into Argon Block votes, and you'll close some blocks! Find out here how to set it up. + +## Background + +Ulixee uses the Argon currency for Micropayments. When a datastore consumer runs a query, they will lock their Localchain with a set amount of Argons - this is called a Micropayment ChannelHold. Payment is now settled in tiny increments between the datastore and user. Once complete, the Datastore will send the Settled ChannelHold note to a notary and the Argon network. + +Argon is a blockchain that uses proof of "work" to close blocks. However, in this case the work is the Datastore queries that you just served to users. Proof of work is supplied in the form of tax revenue from Datastore Micropayments, which are able to be converted into a "vote" on which block to close. If your vote is chosen, you get a portion of the rewards for closing the block. + +Argon requires a tax on all transactions, which it uses to stabilize its value. For transactions over 1 Argon, this is set to ~20 cents (0.2 argons). For any micropayments (under 1 Argon), it is only 20% of the transaction value. So as your users pay for queries, you're actually automatically collecting a tax that you can use to close Argon blocks. + +## Closing Blocks + +When you setup your Datastore for payments, you'll configure the Argon Localchain that your Datastore will use. You'll also choose a specific Argon Miner that you believe is an honest operator. + +Your Localchain collects payments and automatically claims the appropriate amount of tax when it settles with your chosen Argon Notary (`ARGON_NOTARY_ID`). + +The Localchain can be setup to automatically create block votes if you set an `ARGON_BLOCK_REWARDS_ADDRESS` in your env, or you set it using the `@ulixee/cloud` command line. The Localchain will create votes with a strategy of voting anytime your collected tax exceeds the minimum vote threshold. + +NOTE: you can code a more sophisticated strategy if you want to change this process. The code for creating votes can be currently traced in the Argon Mainchain codebase in [Localchain/src/balance_sync.rs](https://github.com/argonprotocol/mainchain/blob/416812ac0c905295dcf76472a68bee16d02e5f3c/localchain/src/balance_sync.rs#L508). A new strategy doesn't have to be in rust. There's a nodejs library to interact with the mainchain `@argonprotocol/mainchain`, as well as node interaction with the notary and localchain at `@argonprotocol/localchain`. You'll find uses of that library in this [project](. + +## Securing you Block Rewards Address + +Your block rewards account does not need to have a Localchain yet (you can always create one later that attaches to this private key). For that reason, you should generate your block rewards address, but only create a key. You should have a seed phrase and an account address once you're done. Do not publish the seed phrase to any CloudNode or anything public (like a code repository). This will separate your block rewards from your Localchain account - kind of like automatically depositing them into a more secure vault. + +NOTE: In a production environment, you would create these with a hardware wallet or a secure offline computer. + +For the Testnet, it's perfectly valid to follow the online Polkadot.js process to create an account [here](https://github.com/argonprotocol/mainchain/blob/3a4bfab8cb296b85da0543d577a2a33e85b83b54/docs/account-setup.md) or even reuse your Localchain account if you don't want to bother with this step. + +## Watching for Closed Blocks + +You can watch for closed blocks by listening to the Argon Mainchain. You can use the `@argonprotocol/mainchain` library to listen for new blocks and check if your vote was chosen. If your vote was chosen, you'll receive a reward in the form of Argons. + +To monitor for blocks using your address, you could do something like this: + +```typescript +import { getClient } from '@argonprotocol/mainchain'; + +const client = await getClient(`wss://rpc.testnet.argonprotocol.org`); +const eventMetadata = client.events.blockRewards.RewardCreated.meta; +const rewardsIndex = eventMetadata.fields.findIndex(x => x.name.toString() === 'rewards'); +const unsub = mainchainClient.rpc.chain.subscribeNewHeads(async lastHeader => { + const blockHash = lastHeader.hash.toHex(); + const blockNumber = lastHeader.number.toNumber(); + + const events = await mainchainClient.query.system.events.at(blockHash); + for (const { event } of events) { + if (event.section === 'blockRewards' && event.method === 'RewardCreated') { + const rewards = event.data[rewardsIndex].toJSON(); + for (const reward of rewards) { + if (reward.accountId == '5DtCHcwuh7Mhp8cZtvinxDzSa36rh7m3TG9LFo3Tgxuyx889') { + console.log(`You closed a block! (${blockNumber}: ${blockHash})`, JSON.stringify(reward)); + } + } + } + } +}); +``` + +Or you can use the Argon [developer console](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc.testnet.argonprotocol.org#/explorer) UI to watch for your rewards (this is a common utility built by Polkadot, which produces the Substrate framework that Argon is built on top of. You should see a BlockRewards event with your address if you closed a block. + +![Polkadot.js - Block Explorer](../images/pjs-block-explorer.png) diff --git a/datastore/docs/guides/querying-datastores.md b/datastore/docs/guides/querying-datastores.md new file mode 100644 index 00000000..1894e6aa --- /dev/null +++ b/datastore/docs/guides/querying-datastores.md @@ -0,0 +1,62 @@ +# Querying Datastores + +The primary way you'll interact with Datastores is the `@ulixee/client` library. The client library allows you to use an addressing system to lookup datastores by version and id, and then run queries against them using SQL. You can learn the details of the client library [here](../../client). + +## Using a Localchain + +Datastores that require payment use the [Argon currency](https://argonprotocol.org). You can learn more about the Argon [here](./using-localchain.md). You can directly use a Localchain account to pay for queries, as shown below. In this example, a Datastore with a `domain` of `Meals.Health` is queried for all recipes that are `paleo`. Version `0.0.1` of the Datastore is used. + +This payment services is using all default settings, which will use the `primary` Localchain on the machine installed into the default location. It will attempt to create [Channel Holds](../basics/payments.md#micropayment-channelholds) for 100 queries at a time - if the price is 1 milligon per query, this will load 100 milligons into the Channel Hold. You can choose different "Channel Hold" strategies by passing in a different `channelHoldAllocationStrategy` object. + +```typescript +import { Client, DefaultPaymentService } from '@ulixee/client'; + +(async () => { + const client = new Client(`ulx://Meals.Health/v0.0.1`, { + paymentService: await DefaultPaymentService.fromLocalchain(), + }); + const results = await client.query( + `SELECT * from recipes where diet = 'paleo`, + ); + + console.log(results); + + await client.disconnect(); +})().catch(console.error); +``` + +### Acquire Testnet Argons + +If you want to test out a Datastore using the Argon testnet, you can request them using the Discord Faucet. Find directions [here](https://github.com/argonprotocol/mainchain/blob/main/docs/account-setup.md#requesting-testnet-funds). + +## Using the Testnet Databroker + +> Under construction. + +An easier way to query Datastores is to use the Ulixee Foundation's Databroker. The Databroker allows you to run queries against any Datastore on the Testnet without needing to manage a Localchain account. The Databroker will automatically pay for your queries using the Ulixee Foundation's Localchain account. You'll need to register on the Databroker Admin Panel to get an API key (see next step). + +```typescript +import { Client, DefaultPaymentService } from '@ulixee/client'; + +(async () => { + const client = new Client(`ulx://Meals.Health/v0.0.1`, { + paymentService: await DefaultPaymentService.fromBroker( + 'wss://databroker.testnet.ulixee.org', + { + pemPath: 'path to your Identity pem file', + }, + ), + }); + const results = await client.query( + `SELECT * from recipes where diet = 'paleo`, + ); + + console.log(results); + + await client.disconnect(); +})().catch(console.error); +``` + +### Register on the Databroker Admin Panel + +Normally, the admin panel for a Databroker is locked down to the owner of the Datastore. However, the Ulixee Foundation has opened up the Databroker for the Testnet to allow anyone to register and run queries. You can register for the Databroker [here](https://databroker.testnet.ulixee.org:18171/admin). diff --git a/datastore/docs/guides/register-a-domain.md b/datastore/docs/guides/register-a-domain.md new file mode 100644 index 00000000..0c6c6eaa --- /dev/null +++ b/datastore/docs/guides/register-a-domain.md @@ -0,0 +1,143 @@ +# Register a Domain + +## Purpose + +Domains serve a few purposes in Ulixee. + +1. They allow users of your Datastore to run queries with a fixed domain name, instead of an ip address. This can change per version, so it operates like a DNS for versions of your Datastore. + + ```typescript + const client = new Client(`ulx://baseball.stats/v1.0.0`); + ``` + + Here you can see the domain `baseball.stats` is used to connect to the Datastore. When a query initiates, it will lookup the IP and Datastore ID in the Argon network and use that information to connect. If you need to move your hosting, you can update the IP and Datastore ID in the Argon network and all clients will automatically connect to the new location. You can also host different versions of your Datastore at different hosts. This gives you flexibility to move your Datastore around while keeping links to it resilient. + +2. As a Datastore operator, you define your payment information in the Domain, which allows you to safely use a Cloud Datastore hosting service because clients are configured to check for payment first. The Ulixee Foundation will launch a public cloud like this, but no launch date is currently available. + +3. In the future, you can also use a Domain to delegate payments to an external party. This will allow you not to worry about managing a Localchain directly. This feature is possible now, but there aren't any public services available yet. + +## Using a Domain + +To add a domain to your Datastore, you need to: + +1. [Lease](#lease-a-domain) it with your Localchain +2. [Register](#register-your-zone-record) the Domain with the Argon network +3. [Add](#communicating-your-domain) the Domain to your Datastore + +## Lease a Domain + +A Domain Lease costs 1 Argon per year in "tax", which you can use on [Block Votes](./close-argon-blocks.md). There are a limited set of top level domains in Argon that are currently available. You can see them below. + +NOTE: Please submit a PR for any new top level domains you'd like to see added. + +```rust +# https://github.com/argonprotocol/mainchain/blob/3a4bfab8cb296b85da0543d577a2a33e85b83b54/primitives/src/domain_top_level.rs +``` + +### Check for a Domain availability: + +```bash +npx @argonprotocol/localchain domains check Meals.Health \ + -m "wss://rpc.testnet.argonprotocol.org" +╭──────────────┬─────────────┬──────────────────────────────────────────────────────────────────╮ +│ Domain ┆ Registered? ┆ Hash │ +╞══════════════╪═════════════╪══════════════════════════════════════════════════════════════════╡ +│ Meals.Health ┆ No ┆ 05a0b6ebcaf7049b841073f4a34d2657e92ca7294fa4f8f2270cca37a7bebe12 │ +╰──────────────┴─────────────┴──────────────────────────────────────────────────────────────────╯ +``` + +### Lease a Domain: + +As stated above, you need to have at least 1 Argon in your wallet to lease a domain (instructions to transfer funds to localchain [here](./create-a-localchain.md#transfer-in-funds)). This will include your lease in the next Notebook submitted to Argon by your notary (it's using default of notaryId=1 below). You can renew a lease by submitting a new lease with the same domain name before the year is up. + +You can see an example below: + +```bash +npx @argonprotocol/localchain domains lease Meals.Health \ + -m "wss://rpc.testnet.argonprotocol.org" \ + --owner-address "5DtCHcwuh7Mhp8cZtvinxDzSa36rh7m3TG9LFo3Tgxuyx889" + +Meals.Health registered at tick 2351 in notebook 1357. Domain hash=0x05a0b6ebcaf7049b841073f4a34d2657e92ca7294fa4f8f2270cca37a7bebe12 (use this hash for zone record registration on mainchain). +``` + +This will output your Domain Hash, a hex encoded 32 byte string. You'll use this hash to track your domain in the mainchain, as well as to register your Zone Record. The "owner-address" is the account you will manage the domain with on the Argon network. + +### Confirming your Lease + +You can confirm your lease by checking the Argon network for your domain. You'll want to navigate to Storage -> domain -> registeredDomains, and inputting your domain hash. You can see an example below: + +> ![Argon Registered Domain](../images/pjs-registered-domain.png) + +You can also check on the status of your domain using the Localchain cli: + +```bash +$ npx @argonprotocol/localchain domains list \ + -m "wss://rpc.testnet.argonprotocol.org" +╭───────────┬──────────────┬──────────────────────────────────────────────────┬───────────────────┬─────────────╮ +│ Top Level ┆ Second Level ┆ Owner ┆ Registration Tick ┆ Hash │ +╞═══════════╪══════════════╪══════════════════════════════════════════════════╪═══════════════════╪═════════════╡ +│ health ┆ meals ┆ 5DtCHcwuh7Mhp8cZtvinxDzSa36rh7m3TG9LFo3Tgxuyx889 ┆ 2351 ┆ 0x05a0…be12 │ +╰───────────┴──────────────┴──────────────────────────────────────────────────┴───────────────────┴─────────────╯ +``` +## Register your Zone Record + +A Zone Record has your payment information, as well as a host mapping for your versions. You can see an example below: + +```json +{ + "paymentAccount": "5DtCHcwuh7Mhp8cZtvinxDzSa36rh7m3TG9LFo3Tgxuyx889", + "notaryId": 1, + "versions": { + "1.0.0": { + "host": "wss://206.189.181.34:1818", + "datastoreId": "vegetarian-recipes" + } + } +} +``` + +You can register your Zone Record using the Polkadot.js UI. You can see a partially filled out form [here](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc.testnet.argonprotocol.org#/extrinsics/decode/0x0d000000000000000000000000000000000000000000000000000000000000000000d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d0100000004010000000000000000000000487665676574617269616e2d6c617361676e612c3139322e3136382e312e30). + +You'll want to replace the paymentAccount with your Datastore Payments account, and obviously update the versions and datastoreIds: + +![Polkadot.js - register zone record](../images/pjs-register-zone-record.png) + +## Communicating your Domain + +Once you've added a domain to your Datastore, your docs will be automatically updated with the new domain. Here's where you would add it to your Datastore: + +```typescript +import Datastore, { Extractor } from '@ulixee/datastore'; +import { array } from '@ulixee/schema'; + +export default new Datastore({ + domain: 'Meals.Health', + version: '1.0.0', + id: 'vegetarian-recipes', + extractors: { + test: new Extractor({ + run(ctx) { + ctx.Output.emit({ + meal: 'Vegetarian Lasagna', + }); + }, + schema: { + input: { + ingredients: array(string()), + }, + output: { + meal: string(), + }, + }, + }), + }, +}); +``` + +Client examples in the autogenerated docs will now show the domain to your prospective users: + +```typescript +import Client from '@ulixee/client'; + +const client = new Client('ulx://Meals.Health/v1.0.0'); +``` diff --git a/datastore/docs/guides/setup-datastore-payments.md b/datastore/docs/guides/setup-datastore-payments.md new file mode 100644 index 00000000..aafff94b --- /dev/null +++ b/datastore/docs/guides/setup-datastore-payments.md @@ -0,0 +1,97 @@ +# Setup Datastore Payments + +> Datastore payments can be as easy as setting a price for your Datastore and adding a flag to your CloudNode configuration. This guide will walk you through the steps to setup your Datastore with payments. + +### Background + +There are a lot of concepts around payments described [here](../basics/payments.md#concepts). These might be helpful to understand before you setup your Datastore with payments. + +## 1. Install CloudNode + +When you install Ulixee Cloud, it will install a few dependencies: + +- `@ulixee/cloud` - The main package for running a CloudNode +- `@argonprotocol/localchain` - A package for interacting with the Argon Localchain, which is used for payments. +- `@ulixee/datastore` - The main package for running a Datastore +- `@ulixee/datastore-plugins-hero` - A package for running Hero-wrapped Datastore + +```bash +npm install @ulixee/cloud +``` + +## 2. Setup your Localchain + +The easy way to start your Localchain is to simple add the following command to your ulixee start script: + +```bash +npx @ulixee/cloud start --argon-localchain-create-if-missing +``` + +When you do this, a Localchain named "primary" will be created for you in the default [location](https://github.com/argonprotocol/mainchain/tree/main/docs/localchain.md#command-line-interface). + +> A Localchain can only be used by a single process at a time. If you're running multiple Datastores, you might find it useful to use the `--argon-localchain-path` parameter to place the Localchain in a folder local to your project. + +You can choose to add more features to your Localchain setup like a password, a type of public key signing, or a different location. Details are available in the [Cloud configuration settings](../overview/configuration.md#argon-payment-configuration). + +## 3. Optional: Determine your Argon Block Rewards Address + +Datastores can earn up to 25% of the block rewards from the Argon mainchain. This includes Argons that you can buy data with, as well as ownership tokens that grant you rights to mine in the network. You can set this address in your CloudNode configuration (env var: `ARGON_BLOCK_REWARDS_ADDRESS`), or with a cli command to your `@ulixee/cloud` start command. + +```bash +npx @ulixee/cloud start \ + --argon-block-rewards-address=5DfXFKuCXHuyzdpo1ih3yizabyAC47frbbCidKjFsw3ucs8C +``` + +You can learn more about creating an account [here](./using-localchain.md#create-an-account). + +## 4. Configure your Datastore + +### Add a Price + +Datastore pricing is set in "Microgons", which are one millionth of an Argon (or said differently, 1/1000 of a penny). You can set a price for your Datastore by adding a `basePrice` to the entities in your Datastore. + +```typescript +import Datastore, { Extractor } from '@ulixee/datastore'; + +const datastore = new Datastore({ + name: 'HelloWorld', + extractors: { + // Add a price of 1 milligon (a penny) + basic: new Extractor({ + basePrice: 1_000, + async run({ Output }) { + for (const line of ['hello', 'world']) { + Output.emit({ + line, + }); + } + }, + }), + }, +}); + +export default datastore; +``` + +### Optional: Add a Domain + +You can add a domain to your Localchain to help users lookup the ip hosting AND payment information in the same place. This can be very useful for using public hosting (there will be public Datastore hosting services in the near future). You can learn more about adding a domain [here](./register-a-domain.md). + +## 5. Deploy your Datastore + +Now you can deploy your Datastore to the CloudNode from step 1/2. You will need port 1818 available publicly by default on that server. + +To deploy your datastore, from the location of your Datastore project, run (assuming your file is called `datastore.ts`): +```bash +npx @ulixee/cloud deploy ./datastore.ts \ + --cloud-host :1818 +``` +
+ +> Note: you can secure admin activities like these by supplying [`Admin Identities`](../overview/configuration.md#admin) to your CloudNode. + +## 6. Test Querying with Payments + +Ok, you're all set! + +Now you can test out using the `@ulixee/client` with payments enabled. Read how to do that [here](./querying-datastores.md). diff --git a/datastore/docs/guides/using-localchain.md b/datastore/docs/guides/using-localchain.md new file mode 100644 index 00000000..9dddca87 --- /dev/null +++ b/datastore/docs/guides/using-localchain.md @@ -0,0 +1,141 @@ +# Using the Argon Localchain + +When you create an [Argon](https://github.com/argonprotocol/mainchain) account, you'll be able to use that account to access both a "Mainchain", which is like a traditional decentralized blockchain, but you'll also get a "Localchain" account. A Localchain is like a personal blockchain where your computer will hold the balance of the account and you can send "Notes" to other users, which represent fund transfers. + +NOTE: the commands in the Argon github page are about the rust based localchain, but you can interact with the Localchain in nodejs directly using the `@argonprotocol/localchain` module, as well as the [Ulixee Desktop](https://github.com/ulixee/desktop) app, which has a more visual interface. + +### Argon is in Testnet + +The Argon is currently in a Testnet phase. The tokens are purely for experimental purposes and hold no value. The network is not secure and is not meant for production use. The Testnet will be reset periodically, and all tokens will be lost. + +Follow along with the Argon project [here](https://github.com/argonprotocol/mainchain). + +## Core Concepts + +You can read about the core concepts of a Localchain [here](https://github.com/argonprotocol/mainchain/tree/main/docs/localchain.md). + +Within Ulixee, the "Localchain" represents the primary way that you'll interact with the Argon network. + +### Dual Chain Accounts + +A single address or "account" can be used on both the Mainchain and Localchain. The Mainchain is a public blockchain that is used to store the state of the network, and focuses on keeping the Argon price stable. The Localchain is a personal blockchain that allows for direct, fast p2p payments. + +### Notaries + +A notary is a rollup service that will verify and then aggregate your personal blockchain transactions into a single transaction on the Mainchain called a "Notebook". The Mainchain works with an agreed time-state called "ticks", and Notebooks must be submitted in order for each tick. Once your "balance change" is in a notebook, it's considered final - there's no fee or acceptance process for a notebook beyond it having all valid transactions per network rules. + +### Localchain Internals + +If you want to explore whats in your Localchain, you'll find a folder in the [`Data`](../overview/configuration.md#data) directory of your operating system where a SQLite database is created per Localchain account. In nodejs, you can use the `npx @argonprotocol/localchain` cli to interact with your Localchain. + +### Datastore Payments + +You can create a Localchain account for your CloudNodes. Your Localchain can be considered a "hot wallet" - the funds on that machine are locked by any passphrase you put on there, but they are not as secure as a "cold wallet" (like a hardware wallet). You likely want to periodically transfer funds from your Localchain to a Localchain not connected to your public cloud. See below for how to [transfer out funds](#export-localchain-funds). + +Localchains allow for micropayments down to one millionth of an Argon (which is ~ one millionth of a penny). This allows for a very fine-grained payment system that can be used to pay for things like queries, storage, and other services. Learn more [here](../basics/payments.md). + +### Datastore Query Payments + +A consumer of a Datastore will use [Localchain Argons](../basics/payments.md) or a [Databroker](../advanced/databrokers.md) to pay for queries. Queries can be paid for one by one without any form of long-term account, and your funds will be valid on every Datastore. That allows us to do things like [Cloning](../basics/cloning.md), which ensures everyone receives payment for their work. + +### Databrokers + +Databrokers allow a user to take advantage of the simplicity of Argon payments without worrying about keeping a Localchain account funded. They're like an API service you might put your credit card into. In the Testnet, you can use the Ulixee Foundation's Databroker for free. In the future, you'll be able to use other Databrokers for a fee. Find out more [here](../advanced/databrokers.md). + +### Domains + +Localchains are able to lease "Domains" in Argon, which allow a Datastore author to communicate version hosting and payment information to users. More [here](./register-a-domain.md). + +## Create an Account + +You need to first create an Argon Account. Follow the directions [here](https://github.com/argonprotocol/mainchain/blob/main/docs/account-setup.md). You should have a seed phrase and an account address once you're done. + +The Argon docs show you how to create a Localchain account with the cli: + +```bash +$ npx @argonprotocol/localchain accounts create --name="alice" \ + --suri="bottom drive obey lake curtain smoke basket hold race lonely fit walk//Alice" \ + --key-password="password" + --mainchain-url="wss://rpc.testnet.argonprotocol.org" +``` + +You can also create an account in Ulixee Desktop in the "Wallet tab": + +> ![Ulixee Desktop Wallet](../images/desktop-wallet.png) + +## Create a Seed Phrase + +You can create a seed phrase using a lot of tooling online that generate 12-24 word mnemonic phrases. You can also use the [Polkadot.js Developer Console](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc.testnet.argonprotocol.org#/accounts) to create an account and get a seed phrase if you have enabled `Allow local in-browser account storage` in the Settings tab. Or you can create mnemonic phrases manually using the [@polkadot/util-crypto](https://polkadot.js.org/docs/util-crypto/examples/create-mnemonic/) library. + +```typescript +const { + mnemonicGenerate, +} = require('@polkadot/util-crypto'); + +async function main () { + const mnemonicAlice = mnemonicGenerate(); + + console.log(`Generated mnemonic: ${mnemonicAlice}`); +} + +main().catch(console.error).finally(() => process.exit()); +``` + +## Transfer In Funds + +Once you've created an account on the Mainchain and a corresponding Localchain account, you can transfer funds into your Localchain account. You can do this using the `@argonprotocol/localchain` cli: + +> NOTE: This will take a few minutes to process, as it requires the block that includes your transaction to be finalized on the Mainchain. Mainchain blocks close once a minute, but finalization occurs as a consensus outside of that process. It can vary from a few seconds to several minutes. + +```bash +$ npx @argonprotocol/localchain accounts from-mainchain 5.0 \ + --name="Blake" \ + --mainchain-url="wss://rpc.testnet.argonprotocol.org" +``` + +Or using the Ulixee Desktop Wallet: +![desktop-transfer-from-mainchain.png](../images/desktop-transfer-from-mainchain.png) +![desktop-transfer-from-mainchain2.png](../images/desktop-transfer-from-mainchain2.png) + +## Person-to-Person Payments + +If you need to reload your Localchain (named `BlakeDatastore`) because it's paying for Data queries, or embedded in a Datastore to pay for data from Cloned Datastores, you can move funds to your account from a Localchain on another machine. + +Assuming you have a Localchain account on your local machine called `Funds`, you'll create a signed half of a balance change to send to `BlakeDatastore` (whose address is `5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY`): + +```bash +$ npx @argonprotocol/localchain transactions send \ + 50 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY \ + --name="Funds" \ + --mainchain-url="wss://rpc.testnet.argonprotocol.org" +``` + +This will output a file that you'll upload to the `BlakeDatastore` machine to complete the transaction. On your remote machine, you'll run: + +```bash +$ npx @argonprotocol/localchain transactions receive \ + --name="BlakeDatastore" \ + --mainchain-url="wss://rpc.testnet.argonprotocol.org" +``` + +## Export Localchain Funds + +If you want to export funds from your the `BlakeDatastore` Localchain to a more secure Localchain, you can inverse the process above, generating a signed balance change request on `BlakeDatastore` and then sending it to your more secure machine. + +From `BlakeDatastore`: + +```bash +$ npx @argonprotocol/localchain transactions send 50 \ + 5Gsecureaddress... \ + --name="BlakeDatastore" \ + --mainchain-url="wss://rpc.testnet.argonprotocol.org" +``` + +Then you would FTP this file back to your secure machine, and run: + +```bash +$ npx @argonprotocol/localchain transactions receive \ + \ + --name="SecureMachine" \ + --mainchain-url="wss://rpc.testnet.argonprotocol.org" +``` diff --git a/datastore/docs/images/desktop-transfer-from-mainchain.png b/datastore/docs/images/desktop-transfer-from-mainchain.png new file mode 100644 index 00000000..e14ba7da Binary files /dev/null and b/datastore/docs/images/desktop-transfer-from-mainchain.png differ diff --git a/datastore/docs/images/desktop-transfer-from-mainchain2.png b/datastore/docs/images/desktop-transfer-from-mainchain2.png new file mode 100644 index 00000000..76574ad1 Binary files /dev/null and b/datastore/docs/images/desktop-transfer-from-mainchain2.png differ diff --git a/datastore/docs/images/desktop-wallet.png b/datastore/docs/images/desktop-wallet.png new file mode 100644 index 00000000..b72e732a Binary files /dev/null and b/datastore/docs/images/desktop-wallet.png differ diff --git a/datastore/docs/images/pjs-block-explorer.png b/datastore/docs/images/pjs-block-explorer.png new file mode 100644 index 00000000..c869ede8 Binary files /dev/null and b/datastore/docs/images/pjs-block-explorer.png differ diff --git a/datastore/docs/links.yaml b/datastore/docs/links.yaml index cce43240..bfa46647 100644 --- a/datastore/docs/links.yaml +++ b/datastore/docs/links.yaml @@ -14,10 +14,19 @@ - Crawler - Passthrough Extractor - Cloning + - Payments - Extractor Context - Input - Output +- title: Guides + items: + - Using Localchain + - Register a Domain + - Setup Datastore Payments + - Close Argon Blocks + - Querying Datastores + - title: Advanced items: - Extractor Schemas diff --git a/datastore/docs/overview/concepts.md b/datastore/docs/overview/concepts.md index 8339ad91..33ad779b 100644 --- a/datastore/docs/overview/concepts.md +++ b/datastore/docs/overview/concepts.md @@ -14,10 +14,14 @@ Datastore Extractors contain all their core logic in a single unit. This unit ca When you write a scraper script, you often need to run the same logic across a series of pages. Sometimes you know them ahead of time, and sometimes you spawn new tasks based on the data found in the current task. -Datastore extractors and cralwers are expected to use input to drive these dynamic starting points. They're also expected to return the same data format consistently. +Datastore extractors and crawlers are expected to use input to drive these dynamic starting points. They're also expected to return the same data format consistently. When you combine these two ideas, it means your "Scraper scripts" can become simple functions that provide parameters and expect a result in a certain format. And it makes them infinitely composable. ## Deployable Datastores can be packaged up and deployed as a "unit" to a Ulixee CloudNode. You can query using the [Ulixee Client](https://ulixee.org/docs/client) library. By deploying to a CloudNode, Datastores are able to run very efficiently - the code is cached, all individual commands for the Datastore Extractors are run on the CloudNode and do not need serialization/transport to a Client. The only necessary communications are an "input" and the resulting "output" as a response. + +## Wage Protection + +Datastores use the Argon to charge per-query with a base price. A base-price in Argon will hold the same value over time, so you should have no need to adjust it in 5 years to a new reality of the current value of money. The Argon will always be worth an inflation-adjusted amount. However, it works by adjusting the supply of Argons to match demand, and the price can fluctuate in the short term to match that demand. To stabilize your earnings, Datastores automatically employ a wage-protector, which will adjust your base query price based on the current value of the Argon vs what it should be. So if Argon should be worth $1.00, and it's currently worth $0.99, your Datastore will automatically adjust the price * 1.01 to keep your earnings stable. This doesn't actually mean your customers will pay more either, because they'll be able to buy Argons to pay for queries at that same rate. diff --git a/datastore/docs/overview/configuration.md b/datastore/docs/overview/configuration.md index 41731444..d52adc07 100644 --- a/datastore/docs/overview/configuration.md +++ b/datastore/docs/overview/configuration.md @@ -31,7 +31,7 @@ ULX_DATASTORE_LOOKUP_SERVICE_HOST=# Datastore Lookup Service Host ### Argon Payment Configuration -Your Argon Localchain configuration should be setup only on a Payment Services Host. Received payments will be sent to this Localchain. In order to properly claim them, you should ensure to follow the [payment setup guide](../advanced/payments.md). +Your Argon Localchain configuration should be setup only on a Payment Services Host. Received payments will be sent to this Localchain. In order to properly claim them, you should ensure to follow the [payment setup guide](../basics/payments.md). #### Using the Cloud Node Cli @@ -39,10 +39,11 @@ Your Argon Localchain configuration should be setup only on a Payment Services H $ npx @ulixee/cloud start \ --argon-localchain-path /path/to/localchain \ --argon-mainchain-url wss://rpc.testnet.argonprotocol.org - --argon-localchain-password-interactive \ - + --argon-localchain-password-interactive ``` +> Try the command `npx @ulixee/cloud start --help` for more options. + OR you can tell the Cloud Node to boot up a Localchain for you: ```bash diff --git a/datastore/docs/overview/introduction.md b/datastore/docs/overview/introduction.md index 3b433b35..817a0c5e 100644 --- a/datastore/docs/overview/introduction.md +++ b/datastore/docs/overview/introduction.md @@ -1,27 +1,38 @@ # Introduction -> Datastores are deployable "databases" that have extractors and tables, support native payment, and can be cloned and expanded as you see fit. Datastore Extractors contain data retrieval functions, like Hero scraper scripts, with structured input and output. Deploying a Datastore provides you with a structured Data API that can be privately consumed, or sold "out of the box". +> Datastores are structured, deployable "micro-databases" that combine private data sources, live-extracted web data and static metadata. Out of the box, they support charging per query using the [Argon](https://argonprotocol.org) currency. Best of all, any published Datastores can be stacked on top of each other using our "cloning" process. -## What is a Datastore? +## Core Features -Datastores create databases of specialized data structures - for instance, a Travel database, or a Jobs database. They're a combination of static metadata tables, dynamically retrieved web data and cached aggregated data that make up a data category. They support payment out of the box, so a client can pay per query without any setup or contracts. Datastores also support PostgreSQL natively (including payment), so can be tested out and integrated across programming languages. +- **Structured APIs:** Datastores create "constantly refreshing" databases of specialized data structures - for instance, a Travel database, or a Jobs database. They're a combination of static metadata tables, filtered views of internal data-sources, and dynamically retrieved web data. +- **Out of Box Payments:** Datastore costs can be completely customized for each published data entity: whether per Kilobyte of information, per extraction, or per query. You can get started very simply though - no need to setup a bank account or create customer contracts. Payments are made on a single query basis using the [Argon](https://argonprotocol.org), a fiat-independent stablecoin that you can export to your native currency whenever you choose. +- **Smart Caching:** Datastores automatically cache extracted data so you can reuse more complex data extraction across multiple client queries. You can also choose to allow the query-er to define how fresh the data should be. +- **Postgres Compatibility:** Datastores also support PostgreSQL natively (including payment), so can be tested out and integrated across programming languages. -## Datastore Extractors +## Payments -Datastore Extractors create structure -- boundaries -- around a single "scrape", which make your scripts are far easier to test, re-try, scale and compose. It allows us to do things like: +Datastores are designed to be monetized, but it's your choice. If you set a price per query, the user will be prompted to pay using Argon. Learn more about setting up payments in the [Datastore Payments](../basics/payments.md) guide. -- Restart a script during development as you change it. -- Rotate inputs to try out a variety of IPs, parameters, and more to make sure you can handle edge cases. -- Test the extraction of 100s of different potential results pages and ensure your Output follows the same structure. -- Spawn new Extractors from the current one if you need to parallelize following links. +## Structured Data + +Datastores are designed to be structured, so you can easily query them using the Ulixee Desktop or the [@ulixee/client](https://ulixee.org/docs/client) library. Users will get a type-checked response using SQL and Typescript definitions. -## Datastore Crawlers +They are built around these core data entities: -Datastore Crawlers allow you to write specialized Extractors that only output a "cached" scrape. It comes with built-in caching, so you can automatically re-use results that have been recently recorded. +1. **Extractors:** Functions that take input and return output [more](../basics/extractor.md). +2. **Crawlers:** Web crawlers that load pages in a format that can be re-used and cached [more](../basics/crawler.md). +3. **Tables:** Database tables that can be queried and joined with your extractors [more](../basics/table.md). +4. **Cron Jobs:** Scheduled tasks that can run trigger Extractors or Crawlers to refresh data (**FUTURE FEATURE**). -## Datastore Tables +## Enhanced Developer Experience -Datastore Tables allow you to manage and deploy database tables as part of your "api". This can be useful to enhance your functions with metadata or cached data. +The structure of Datastores allows us to enhance the developer experience of building and testing scraping scripts using the [Hero](https://ulixee.org/docs/hero) scraping browser. It allows us to do things like: + +- Automatically rerun a script during development as you change it. +- Timetravel through the script to see the exact state of the browser at any point. +- Rotate inputs to try out a variety of IPs, parameters, and more to make sure you can handle edge cases. +- Test the extraction of 100s of different potential results pages and ensure your Output follows the same structure. +- Spawn new Extractors from the current one if you need to parallelize following links. ## Installation diff --git a/datastore/docs/overview/relationship-with-argon.md b/datastore/docs/overview/relationship-with-argon.md new file mode 100644 index 00000000..36c85717 --- /dev/null +++ b/datastore/docs/overview/relationship-with-argon.md @@ -0,0 +1,35 @@ +# Relationship with Argon + +> Argon is an inflation-proof stablecoin that is designed to handle micropayments as small as 1 millionth of a dollar. Ulixee comes built-in with support for Argon payments per-query. +> +> It's used for payment in Ulixee Datastores, but the tax revenue generated from Datastore Micropayments can actually be turned around to close Argon blocks. + +## Argon Overview + +Argon is a sister-project to Ulixee, built by the same initial developers. It was built to solve the lack of payment options that satisfied the following needs: + +### Cross border payments + +The Ulixee developers live around the world, and consumers of Datastores are often in different countries. This makes it difficult to use traditional payment methods like bank transfers or credit cards. Argon is globally available and does not have to worry about conversion between currencies. + +### Micropayments in high volume + +A datastore might serve up millions of queries an hour. We wanted to build Ulixee so it could charge per-query, but still handle large volumes. + +### No contracts or agreements + +Part of a per-query model is a response to the existing data silos that often require huge contracts with very restrictive data sharing terms. But this goes against the very reason scraping even exists - data is out in the open already. It shouldn't come with onerous restrictions and you should be able to pay for only what you use. + +### Simple and Immediate + +We wanted developers and data owners to be able to monetize their data immediately. It should be as easy as writing an adapter or scraper, wiring it up to a Datastore, and slapping a payment address on it. + +### Safe to hold + +Cryptocurrencies are highly volatile, so while we looked for an existing solution, we aimed for one that would support stable value (eg, a stablecoin). We didn't want developers to be anxious to unload their earnings, and we didn't want query-ers to worry about timing their currency acquisition. The Argon is a stablecoin, but actually goes one step further - it's not tied to a nation fiat currency, but to a basket of goods and services (eg, it's inflation proof). + +## Datastores Closing Argon Blocks + +Argon requires a tax on all transactions, which it uses to stabilize its value. For transactions over 1 Argon, this is set to ~20 cents. For any micropayments (under 1 Argon), it is only 20% of the transaction value. Argon tax is a little different than what you typically think of as a tax - it's a fee that you personally collect, but then are able to convert into "votes" on which block Argon should follow. If your vote is chosen, you get a portion of the rewards for closing the block. + +Argon rewards start at 5 Argons per block, but also 5 Ownership tokens. There are 21 million overall ownership shares that will be created, and it follows a halving formula identical to Bitcoin. Argon Ownership tokens allow you to run Mining Nodes, which are able to mint new Argons as more demand for Argons outstrips supply. diff --git a/website/src/pages/Documentation.vue b/website/src/pages/Documentation.vue index f5ba2e09..f4895803 100644 --- a/website/src/pages/Documentation.vue +++ b/website/src/pages/Documentation.vue @@ -14,8 +14,7 @@ + :to="itm.link"> {{ itm.title }} @@ -24,8 +23,7 @@ + :to="item.link"> {{ item.title }} @@ -33,9 +31,9 @@ -
+
-
+