Releases · streamingfast/substreams · GitHub

04 Aug 20:43

Eduard-Voiculescu

v0.0.16

Changelog

8cda6d9 Prepare release v0.0.16

Assets 7

04 Aug 16:42

Eduard-Voiculescu

v0.0.15

Changelog

4559084 bumping substreams, substreams-macro and test cargo toml versions to 0.0.15 for release

Assets 7

03 Jun 23:39

abourget

v0.0.13

CLI

Changed substreams manifest info -> substreams info
Changed substreams manifest graph -> substreams graph
Updated usage

Changelog

43e9bd7 Added a test with modInit starting at 0.
b89c9c8 Added comment to --start-block. Fix ranges splitter tests.
e5d3aaf Don't early kill the scheduler if it didn't fail.
db8780d Don't send BlockScopedData when we're doing backprocessing.
13084a4 Fix tests.
b0dfd11 Fixed issue with ranges.
cc6c420 Major rework of the scheduling plan, called SplitWork, called WorkUnit, tobe renamed.
cdce6e5 Merge remote-tracking branch 'origin/develop' into splitter
fb9c955 Revise usage and CLI docs.
21e5772 Simplify the scheduler exit loop.
e346177 SplitWork implementation, responsible for driving the show in Squasher and Strategy.
152de88 Squasher / Waiter / Queue / RequestPool / Strategy Simplified. Hanging bug fixed, which was caused by the Squasher never preparing the store when no work was required.
0a00c03 Unlock, but what does that test do anyway?
b42926d Updated change-log.
8c4805f Updated some element of the docs
e203658 Updated substreams info
8d82cb2 Whoops, don't print "" on 'enter'.. misfeature :)
2520a0b substreams info and substreams graph now.
f752f3c block range: split occurs on chunk size boundary
069d028 orchestrator test: make adding to queue concurrent
c75a8ba queue: add more testing
054021e splitter wip
e761111 squasher: fix test

Assets 7

03 Jun 18:32

maoueh

v0.0.12

Summary

Fix null pointer exception at the end of CLI run in some cases.
Do log last error when the CLI exit with an error has the error is already printed to the user and it creates a weird behavior.

Changelog

b10966f Improved error reporting in CLI
7b30088 Preparing for release 0.0.12
13b2945 Updated releaase notes for v0.0.11
bcebc40 remove cancel() if prog is nil
65b3740 returning ctx.Err() if ctx cancelled caught in the scheduler. caused issue that an error would be missed to be caught at top level
721aeb1 worker pool: streamline contexts

Assets 7

03 Jun 06:13

maoueh

v0.0.11

Changelog

27872dc Added mean on top level README (all linking to direct docs links) and reduce emphasis on Developer Preview
e631024 Added updates rate in CLI.
105768c Better boundaries for job scheduling.
20bd9e4 Better failure capture when a module fails.
c67dbe1 Exclude GitBook: commits from showing up in the changelog
d615bad Fixed all doctests, remove all ignores, move docs in substreams-macro directly to substreams::handlers
936023c Merge remote-tracking branch 'origin/develop' into develop
32d4183 Permit arguments to be passed to docker image on run directly
916fc14 Preparing for 0.0.11 release
43f0c59 Prioritize first ranges
682c864 Send the writing of the output cache into a goroutine
b98ca77 better log
363fa03 crate: substreams bumped version
95dec45 fix squashing partial range
1204cc1 fix squashing partial range again
dc65d53 pad block nums in tui
d16f0ae release terminal on ctrl+c
bd9d4d6 state: add retry for writing state to dstore, normalize derr retry to 3 everywhere

Assets 7

02 Jun 19:56

abourget

v0.0.10-beta

Changelog

25a5b5d "state: keep delete of merge data for compatibility of files"
20bd9e4 Better failure capture when a module fails.
86e1db1 Fix log propagation between instances
009a6b7 GitBook: [#62] No subject
c9dd9a2 GitBook: [#63] No subject
b4266f1 GitBook: [#64] No subject
d583d50 GitBook: [#65] No subject
0b2036c GitBook: [#67] No subject
fd56dd5 GitBook: [#68] No subject
8fab00b GitBook: [#69] more-additions
aa3827c GitBook: [#70] No subject
7d84c92 GitBook: [#71] No subject
5f13b4f GitBook: [#73] No subject
cf516ca GitBook: [#74] No subject
5e293b1 Ok, now solved the issue. By simplifying!
10fc034 Reset the instances between blocks.
a5c6797 small fix for subsequent executions of the same substreams which caused edge case issue and not detecting the stores where ready
74c7410 state: output value types all normalized to lowercase
8277aa4 state: remove merge data special key. no longer used

Assets 7

02 Jun 06:05

abourget

v0.0.9-beta

Changelog

`substreams` CLI

Added substreams run -o ui|json|jsonl|module-json|module-jsonl.

Commits

5a4a193 Add a clause to write a temporary KV when not on boundaries.
eb53701 Added a bit of time between crate publishing
5cd7e55 Added hex_literal macro re-export
6fc452a Added more documentation
c282412 Align bounds properly.
66c27c6 Always clean-up terminal
3887379 An attempt to fix hanging, improved
893b9b4 Bar graph! Hit 'm' to see :)
1ba6af7 Better TUI
eb83089 Better clean-up and work on progress.
322aa2a Better mermaid graph.
6b38234 Better range calculations.
79d46e7 Better ranges again.
ee0665e Better var name
40ff1a0 COmment
a517c8d Cosmetics
2a287f3 Default to BarMode. A few refactors.
14e5358 Do error checking on incoming manifests, client and server.
64616b3 First draft at bubbletea integration. Takes control of keyboard mon enfouaré.
296f8e3 Fix change log
60e2148 Fix documentation setup
6c09bf2 Fix more range management issues.
fc1f855 Fixed small typo in docs
79cf059 Fixup latest tests.
8f32e9f Fixup progress.
c12c7ef Fixup some race between channels.
2d8ba92 Fixup the squashing world
ff0b8a8 Fixup which ranges we're using
6c99b8e Found a subtle bug that impeched the merging :)
d98928e Four new output modes
92ef8fc Improve ranges again
a9cd3ff Incredible progress :)
8569066 Merge commit 'ee0665ef71b94a1d10f72805fffd42f9b0c61956' into develop
1f0962d Merge remote-tracking branch 'origin/develop' into develop
37970e3 Properly replace the stores after a backprocessing shot.
5d263fe Quick rename.
c95e01e Remove README by default
1e228ce Remove the BlockRange on the Store
731dc47 Removed extraneous import
0b8f5e5 Removed metadata section in README.md
dff1fdd Removed progress package, now lives in the TUI. Repacked decode and print into a more formal and expansive TUI.
55332d8 Removed unused code
05d34b4 Rename
bce17d6 Revert "small fix on kv getting reinitialized on each new squasher for each store"
fd2c258 Simplify output
8434b97 Small update to docs
eb6e21b SynchronizeStore to output the stores when they are built.
a631908 Updated to use tagged release of dependencies in docs
b8ba02c When we know there aren't requests from the STRATEGY we early exit.
ad68e06 Working version for SF Ethereum: * Need to compute the merge ranges better.
a1d7a32 Works!!! Oh well :)
109259b adding tests for merge block ranges
d019739 doc: testing layout and file structure
2a545dd fix back process not returning when no back process request need to be run
8d8ef94 fixing waiter tests
a61c1ff getting out of the go func when we get and error from the worker of a scheduler
ff0041a orchestrator: add stringer methods for debugging, waiter does not wait for stores if its initial block is gt desired block num
d6f776e orchestrator: add tests
f1f7352 orchestrator: concurrnt squasher test
fcd2a88 orchestrator: fix squash test
e0df979 orchestrator: refactoring waiter condition, update strategy test
4f5da66 pool: add log for notify
3a1a8ca renaming stuff and adding logs
96bf4e9 replacing some println statements for some zlog and fixing partial/kv boundary condition
17c685d small fix for run cmd
941dfe5 small fix on kv getting reinitialized on each new squasher for each store

Assets 7

30 May 19:59

maoueh

v0.0.8-beta

Changelog

a9384e3 Adjusted release script to be more robust and correclty run publish commands in dry mode
409f4c1 Prepare for 0.0.8-beta release
6a986f2 Replaced rust/substreams/build.rs with explicit generation using protoc-gen-prost plugin
4f68088 Updated to for next release, removed statik again
8159f8d fix builder kv not initialize
dc9a3ca now failed when a store is reference as input with a wrong name

Assets 7

30 May 16:54

maoueh

v0.0.7-beta

Summary

Changed startBlock to initialBlock in substreams.yaml manifests.
code: is now defined in the binaries section of the manifest, instead of in each module. A module can select which binary with the binary: field on the Module definition.
Added substreams inspect ./substreams.yaml or inspect some.spkg to see what's inside. Requires protoc to be installed (which you should have anyway).
Added command substreams protogen that writes a temporary buf.gen.yaml and generates Rust structs based on the contents of the provided manifest or package.
Added substreams::handlers macros to reduce boilerplate when create substream modules.

substreams::handlers::map is used for the handlers corresponding to modules of type map. Modules of type map should return a Result where the error is of type Error
```
/// Map module example
#[substreams::handlers::map]
pub fn map_module_func(blk: eth::Block) -> Result<erc721::Transfers, Error> {
     ...
}
```
substreams::handlers::store is used for the handlers corresponding to modules of type store. Modules of type store should have no return value.
```
/// Map module example
#[substreams::handlers::store]
pub fn store_module(transfers: erc721::Transfers, s: store::StoreAddInt64, pairs: store::StoreGet, tokens: store::StoreGet) {
      ...
}
```

Changelog

eceec63 Added --exclude-paths flag on substream protocgen
4945217 Added Substreams crate documentation
82ae6b7 Added protogen
c02a5b4 Added some missing well known proto descriptors in the system.pb, removed statik replaced by
621ba88 Added support for the sf.susbtreams.v1.Clock in inputs.
1a779ca Added top-level doc.. really thin though :)
42517ea Avoid panic in state builder when missing deltas is encountered
81a642c Better print
c9742a0 Bunch of renames and refactors. BREAK BACKWARDS COMPATIBILITY. Needs to be redeployed everywhere. Packages need to be rebuilt and repackaged. New substreams needs to be used to interact with server.
db26ea7 Change all those store methods helpers to be u64 instead of i64, as that's the type we use in all ordinals all over the place (Eth, and Solana).
153b2cf Cleaned up some unecessary deps
ce857aa Cleaned up substream lib and exported macros
c7debad Correctly handle error on call to grpcClientFactory
d380853 Doc fixes.
29bdd3f Ensure that tests passes in rust code
f9a32d9 First draft at documentaiton.
9dca46b Fix manifest info formatting
280b9e1 Fix pack output message
5545177 Fix proto import dependancy
134a5f7 Fix typo
ec40b6c Fixed up docs. Added set_many for a few.
59215b0 Fixed wrong version in Cargo on substreams crate
c2633a4 Fixup protogen
c260fca Fixup range.
0286a7f Fixup some retries.
0dc3c0a GitBook: [#10] No subject
68efa88 GitBook: [#12] No subject
f4ab67a GitBook: [#13] Modules and packages
cd2ae90 GitBook: [#20] No subject
43158ee GitBook: [#23] getting-started
ee01b07 GitBook: [#24] Better docs, merge strategies, improvements to modules.
bf2ee07 GitBook: [#25] getting-started
709ee88 GitBook: [#26] No subject
07382c8 GitBook: [#27] No subject
26cddf4 GitBook: [#28] getting-started
931c9de GitBook: [#29] No subject
dae3ffc GitBook: [#2] title modification
97e2c49 GitBook: [#4] Substreams doc revamp
ef4cf64 GitBook: [#5] No subject
408ed30 Improved hex encoder and added decoding from hex library
9aec010 Merge branch 'develop' into wasm-refactor
59f299e Merge branch 'develop' of github.com:streamingfast/substreams into develop
4ddec88 Merge branch 'develop' of github.com:streamingfast/substreams into develop
c78ffc2 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
eac4b1b Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
d9d39d9 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
e9da6a8 Merge branch 'develop' of github.com:streamingfast/substreams into feature/progress_wip
b5730c9 Merge branch 'feature/parallel_partial_sub_req' of github.com:streamingfast/substreams into feature/parallel_partial_sub_req
5766c32 Merge branch 'feature/publish' into develop
deabc1f Merge branch 'manifest-reader' into develop
e7412d3 Merge branch 'rust-macros' into develop
77fe753 Merge branch 'wasm-refactor' into develop
c174526 Merge pull request #13 from streamingfast/feature/parallel_partial_sub_req
3682925 Merge pull request #14 from streamingfast/feature/broken_cache
af7288b One response func instead of many.
f023ca0 Preparing for release of CLI + crate
654b58e Refactor substream macros
c885e3e Refactored instrensiv names
c3fcf05 Remove --partial-mode and --no-return-handler .. we can put --partial-mode back if necessary to ease debugging of backend services. We could use SUBSTREAMS_PARTIAL_MODE=true, as to not bloat the end user's experience.
89c8a0f Remove ETH reference
cd4a4f6 Remove a Grrr
67dafd8 Remove the Ethereum stuff
f3f52dc Removed blank lines in errors
dd2dd59 Renamed loadDetla to loadDeltas, replaced some panic with errors, updated some zlog statements
450855f Review the Progress signaling.
c0e477d Some renames
4ed5ae8 Take out Partial from the proto.
45c5647 Take out non working install instructions.
836ef82 Try again
1b2672c Try to pad things. Fix the proto import priority.
ff67ce5 Trying to fix cargo publish command
70d042e Tweaked docs.
cd1bd78 Updated CHANGELOG
33abcc2 Updated CHANGELOG, with substream macros
06f8f35 Updated README.md
9309b53 Updated Summary
eb8171a Updated explicit traits
b74178a Updated latest rendered substreams version
ca3c4b0 Updated packages metadata
3883630 WIP
beb843e WIP: cleaned up args
8ffb631 Working
3b3e008 Wrap up protogen. Fix tests.
76b46f4 added an early exit when no catchup request are needed
893e016 added couple of MarshalLogObject func
5b8cf2d added lock to squasher
1cecae2 added logs to debug context stuff
381202c added missing return
8edee92 added missing return worker to pool
1da7057 added retry around dstore access
6c92140 added retry on sub request
3c4fd6a added some log
2261fa9 added unlock to squasher ;-)
022b2b7 adding config for scheduler's subrequests and block range size
228b828 adding information on the progress bar
fe447ba adding log for request created and adding nil check on if the partial file doesn't exist
7b47ce6 adding missing pointer for squasher in scheduler struct
47e11a7 adding pipeline and scheduler wip
e3ff44c adding progress print for catch up of parallel requests
54b8c54 adding return for scheduler workers
70e1c39 better logging
6345d61 better logging and grpc stream handling
a1e1517 better logging and remove sched config
9c1e030 broken wip
9629eab builder, etc: replace builder start block and end block with block range object
ed57b8a change block range split algo
77001af clean up save stores snapshots
e654b3c cleaned up fix macros
3eb410e client: fix package name overwrite
339fed9 delete partial file after squash
2767c43 do not reuse grpc client create a new for each sub request
30b4096 fix "substreams-partial-mode" request header better logging fix early exit of workers
1387ff2 fix NO_RETURN_HANDLER
996c163 fix all merge tests
d5768da fix delta output
1076c00 fix error message for max subrequest range size
cdbbb22 fix init builder block range
1b5bf42 fix init builder block range
09883e1 fix initialize store so only module outputs one can be initialized has partial fix builder initialization with deltas
62790e9 fix load deltas when cache file is empty
24869bb fix load state info
420c037 fix logs
df92975 fix output caching
b76b0de fix partial file naming
327d094 fix range split
2d4c509 fix reader
c01e1bd fix request pop order from list
78daa28 fix retry code scope
cd64eeb fix scheduler error handling
4682ce6 fix some retry code
83ecdc6 fix squasher not finding the right file
3393f9b fix squashing
41ee189 fix summary layout
8270603 fix test
54f2fa5 fix test
89f8a2a fix test and remove Squash func from builder
e8898d5 fix worker threading issue
63fa967 fixing mermaid
b28b287 flip builder merger func
a8778d5 handle grpc client getter error
a11432c merge develop
d8adcb3 merge values into struct
587ce1b more logs
783b45e move scheduler and squasher to orchestrator package
082aa25 move some things around
5f49840 moving stuff around and creating own concept for progress, instead of having 2 different files doing similarly the same thing
d6865d7 new global worker pool
8252c5b orchestrator refactoring
fd9c3de orchestrator: add basic first test for squasher
8fdc673 orchestrator: add priority queue to pool to re-order requests in the wait buffer
057389c orchestrator: better context handling
1915fa7 orchestrator: change lock to rw mutex
6ebca76 orchestrator: first draft of ordered strategy
6793cdc orchestrator: fix pool tests
bd2c674 orchestrator: fix race conditions in pool
8c55852 orchestrator: fixed close of closed channel on waiter item, switch pipeline to ordered strategy
f237c49 orchestrator: more testing
7abd562 orchestrator: more tests for block waiter
28abc3a orchestrator: pass map of already processed blocks to Waiter for early exit condition
791efcd orchestrator: priority queue now uses csp instead of polling, with some additional renamings and cleanup
193ba09 orchestrator: renames
e65e8fa orchestrator: request stream handles context errors
d2b4d1a orchestrator: smarter locking fo...

Read more

Assets 7

17 May 05:34

abourget

v0.0.6-beta

Changelog

b4df491 Add strings to beginning of signature.
5112914 Added Semver check for package version.
d986c5d Added --stop-block count support
f3b0293 Added log::debug! and log::info! and improved log::println
20a3646 Added substreams::Hex wrapper type to more easily deal with printing and encoding bytes to hexadecimal string
8797355 Added reading of SF_API_TOKEN also if notjing else finding
627ab02 Added validation for names. Merge for proto, packages, rename modules and implement package imports.
c4d2d41 Fix delta start block
8cb54cf Fixed Docker image and improved its building speed
a799cfd Fixed Dockerfile
2502a0b Fixed Slack notification
fb17a73 Fixup required/optional flags.
86af019 Handle error, and better error reporting for packing.
16812f9 Improved around logs handling and reporting
beb3616 Improved image around rust installation
552ed05 Improved readability of substreams run
b6f3aaf Merge branch 'feature/new_store_sync' into develop
1ef72a8 Merge branch 'feature/packaging' into develop
b8c3b6e Package vs Manifest vs Modules.
9169b1b Remove RPC for Eth, not the concern of this repo. Make import paths absolute.
3954a1b Remove left over debug log
b51ed28 Removed wasm-pack from substreams
ea114bd Removed wasm-pack mentions
2b7b28c Support http and https for package dependencies.
b120ea8 The -c (or --compact-output) can be used to print JSON as a single compact line
3148278 The whole thing builds now!
87030db Trap context canceled error when WASMExtension are executed
568e2d4 Try to dedupe some protobuf files based on Sum256 of its serialized payload. Ideally, we'd use the latest, so some reverse sort of some kind, so newer stuff takes precedence on older stuff.
c3273b7 Updated Dockerfile to have a better image
b848c7a Works! .spkg acts as a substreams.yaml :)
4f3a55e add 'tools package' commands: get,list,info,create
43e788d add protobuf descriptor.proto to statik fileDescriptorSet
55ec114 added a grrr log
621dac0 eee wut, fix typos
f9650b0 fix caching issues
39008a5 fix init builder block range
ad8d17d fix output cache by encoding it into json
3413675 fix start-block missing in command line to pick up from the manifest and streaming till live if no stop-block is set
9e78226 fix temp removed code
cd29802 pipeline: blocksFunc split into chunks of 10k block ranges
7d80854 remove 'cli', put all in cmd
91b8549 remove mode name from all stored file remove baseCacheStore params remove all hardcode 10_000 value
bb43944 remove my packaging tools
0b30f78 removing mermaid print on run of substreams
4ce57ac replacing wasmer-go from wasmerio with streamingfast wasmer-go
e2f31d5 service: worker pool moved to struct-level
0804fe6 start block validation
6372493 sub requested for store synchronization load state at any block from kv and deltas data folder re-org
2a83fe5 tools: fix store checker
449cec2 wip for bundling proto files

Assets 7