[indexer-alt] - add rpc api ingestion to indexer-alt #20787

patrickkuo · 2025-01-06T16:18:23Z

Description

add rpc api ingestion to indexer-alt, this is to unblock a private-net app indexer.

Test plan

How did you test the new or updated feature?

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2025-01-06T16:18:44Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 5, 2025 5:47pm

3 Skipped Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	Mar 5, 2025 5:47pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	Mar 5, 2025 5:47pm
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	Mar 5, 2025 5:47pm

bmwill · 2025-01-07T14:39:17Z

crates/sui-indexer-alt-framework/src/ingestion/mod.rs

+    #[clap(long)]
+    pub basic_auth: Option<String>,


Can you include some documentation of the expected format of this CLI arg, below you can see that its expected to be <username>:<password>.

+1 on the docs here, and some other comments:

Let's also clarify that it is only relevant if --rpc-api-url is set.

Consider accepting this value via an environment variable as well.

bmwill · 2025-01-07T14:41:22Z

crates/sui-rpc-api/src/client/mod.rs

+        let value: MetadataValue<Ascii> = format!("Basic {}", auth)
+            .parse()
+            .map_err(Into::into)
+            .map_err(Status::from_error)?;


Given this is a username/password lets make sure to set the sensitive flag on the MetadataValue:

https://docs.rs/tonic/latest/tonic/metadata/struct.MetadataValue.html#method.set_sensitive

bmwill · 2025-01-07T14:42:40Z

crates/sui-rpc-api/src/client/mod.rs

+                .tls_config(
+                    ClientTlsConfig::new()
+                        .with_enabled_roots()
+                        .assume_http2(true),


Is this needed? any TLS negotiation should include h2 in ALPN

this is needed for the full node we setup in the private net

amnn

Some high level thoughts but the change on the indexing framework side looks good here, I see that @bmwill has some unresolved comments on the RPC API side, so I will leave it with him for the final stamp.

amnn · 2025-01-13T16:10:56Z

crates/sui-indexer-alt-framework/src/ingestion/rpc_client.rs

+        Ok(Bytes::from(
+            Blob::encode(&data, BlobEncoding::Bcs)?.to_bytes(),
+        ))


I'm fairly sure we're going to decode this straight away, which is a little unfortunate -- is it worth changing the IngestionClientTrait so that fetch returns a CheckpointData?

Added FetchData, which can be either raw bytes or CheckpointData

amnn · 2025-01-13T16:12:01Z

crates/sui-indexer-alt-framework/src/ingestion/rpc_client.rs

+use sui_storage::blob::{Blob, BlobEncoding};
+use url::Url;
+
+pub(crate) struct RpcIngestionClient {


I'm in two minds about this myself, but do we need this type? (As opposed to implementing IngestionClientTrait directly on sui_rpc_api::Client?

crates/sui-indexer-alt-framework/src/ingestion/mod.rs

amnn · 2025-01-13T16:14:37Z

crates/sui-indexer-alt-framework/src/ingestion/mod.rs

+    #[clap(long)]
+    pub basic_auth: Option<String>,


+1 on the docs here, and some other comments:

Let's also clarify that it is only relevant if --rpc-api-url is set.

Consider accepting this value via an environment variable as well.

amnn · 2025-01-13T16:15:15Z

crates/sui-indexer-alt-framework/src/ingestion/mod.rs

+            let basic_auth = args.basic_auth.map(|s| {
+                let split = s.split(":").collect::<Vec<_>>();
+                assert_eq!(2, split.len());
+                (split[0].to_string(), split[1].to_string())
+            });


Why not accept this as two different flags to avoid hand-rolling the parser?

I am now using the username and password encoded in the URL instead of having separated args

crates/sui-indexer-alt-framework/src/ingestion/client.rs

bmwill · 2025-01-27T17:33:11Z

Note #20986 added support for configuring auth

patrickkuo · 2025-01-30T14:49:47Z

Note #20986 added support for configuring auth

thanks! that's great! I will rebase and update this PR

amnn

Indexing framework side looks good modulo that comment about how we expose auth for RPC, @bmwill, how are we on the RPC API side of things?

crates/sui-indexer-alt-framework/src/ingestion/client.rs

…via ALPN.

…eserilised CheckpointData * use username and password in URL instead of separated field * implement IngestionClientTrait for sui_rpc_api::Client instead of RpcIngestionClient

use grpc status code instead of scanning error message.

bmwill · 2025-03-05T17:30:55Z

crates/sui-rpc-api/src/client/mod.rs

+                .tls_config(
+                    ClientTlsConfig::new()
+                        .with_enabled_roots()
+                        .assume_http2(true),


Is this necessary?

this was needed previously when connecting to a private full node via a basic auth proxy, ALPN negotiation wasn't working. But now we are connecting to the full node directly inside the VPC so it's not needed anymore, I will revert this change.

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 6, 2025 16:18 — with GitHub Actions Inactive

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 10:03 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2025 10:05 View deployment

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 11:00 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2025 11:01 View deployment

patrickkuo force-pushed the pat/indexer-rpc-api branch from 8b65939 to 5d6f78d Compare January 7, 2025 11:28

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 11:28 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2025 11:32 View deployment

patrickkuo force-pushed the pat/indexer-rpc-api branch from 5d6f78d to bf7ed0c Compare January 7, 2025 11:45

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 11:45 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2025 11:47 View deployment

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 12:00 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2025 12:02 View deployment

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 12:04 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2025 12:05 View deployment

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 12:40 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 7, 2025 12:41 View deployment

patrickkuo marked this pull request as ready for review January 7, 2025 14:01

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env January 7, 2025 14:01 — with GitHub Actions Inactive

patrickkuo requested review from bmwill, amnn and wlmyng January 7, 2025 14:01

bmwill reviewed Jan 7, 2025

View reviewed changes

amnn reviewed Jan 13, 2025

View reviewed changes

vercel bot deployed to Preview – sui-docs January 29, 2025 17:36 View deployment

patrickkuo force-pushed the pat/indexer-rpc-api branch from 18cd4de to 82b8fe2 Compare February 3, 2025 12:18

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env February 3, 2025 12:18 — with GitHub Actions Inactive

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env February 11, 2025 12:11 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 11, 2025 12:13 View deployment

patrickkuo force-pushed the pat/indexer-rpc-api branch from 1a47c37 to c21b790 Compare February 19, 2025 18:40

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env February 19, 2025 18:40 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 19, 2025 18:42 View deployment

patrickkuo requested a review from amnn February 20, 2025 11:45

amnn reviewed Feb 21, 2025

View reviewed changes

crates/sui-indexer-alt-framework/src/ingestion/client.rs Outdated Show resolved Hide resolved

crates/sui-indexer-alt-framework/src/ingestion/client.rs Show resolved Hide resolved

crates/sui-indexer-alt-framework/src/ingestion/client.rs Outdated Show resolved Hide resolved

patrickkuo added 13 commits March 4, 2025 10:02

add rpc api ingestion to indexer-alt

d102148

group required for client args

eafd2b9

set assume_http2 true to support server without protocol negotiation …

1805fdc

…via ALPN.

update base64 dep

c9808a3

fix test

d03c413

increase protobuf encoding size limit

550e10e

fixup after rebase

7863aa2

* return FetchData instead of Bytes, Fetch data can be raw bytes or d…

5881857

…eserilised CheckpointData * use username and password in URL instead of separated field * implement IngestionClientTrait for sui_rpc_api::Client instead of RpcIngestionClient

minor fix

db373d1

address PR comment

e62d17f

fix up after rebase

3030f0d

use grpc status code instead of scanning error message.

fix up after rebase

19c6c56

address PR comments

b9dc42c

patrickkuo force-pushed the pat/indexer-rpc-api branch from c21b790 to b9dc42c Compare March 4, 2025 23:24

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env March 4, 2025 23:24 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 4, 2025 23:30 View deployment

minor fix

66c610a

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env March 4, 2025 23:37 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 4, 2025 23:38 View deployment

bmwill reviewed Mar 5, 2025

View reviewed changes

revert assume http2

f02266b

patrickkuo temporarily deployed to sui-typescript-aws-kms-test-env March 5, 2025 17:46 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs March 5, 2025 17:47 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[indexer-alt] - add rpc api ingestion to indexer-alt #20787

[indexer-alt] - add rpc api ingestion to indexer-alt #20787

patrickkuo commented Jan 6, 2025

vercel bot commented Jan 6, 2025 •

edited

Loading

bmwill Jan 7, 2025

amnn Jan 13, 2025

bmwill Jan 7, 2025

bmwill Jan 7, 2025

patrickkuo Feb 3, 2025

amnn left a comment

amnn Jan 13, 2025

patrickkuo Feb 3, 2025

amnn Jan 13, 2025

amnn Jan 13, 2025

amnn Jan 13, 2025

patrickkuo Feb 3, 2025

bmwill commented Jan 27, 2025

patrickkuo commented Jan 30, 2025 •

edited

Loading

amnn left a comment

bmwill Mar 5, 2025

patrickkuo Mar 5, 2025

[indexer-alt] - add rpc api ingestion to indexer-alt #20787

Are you sure you want to change the base?

[indexer-alt] - add rpc api ingestion to indexer-alt #20787

Conversation

patrickkuo commented Jan 6, 2025

Description

Test plan

Release notes

vercel bot commented Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bmwill commented Jan 27, 2025

patrickkuo commented Jan 30, 2025 • edited Loading

amnn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vercel bot commented Jan 6, 2025 •

edited

Loading

patrickkuo commented Jan 30, 2025 •

edited

Loading