Add support for Snowflake Stages and SNOWFLAKE_FULL encryption #24
Conversation
andrebsguedes
force-pushed
the
snowflake-tss
branch
from
c9ebf80
to
172e4bb
Compare
…e behavior between Julia and Rust users of the library
andrebsguedes
force-pushed
the
snowflake-tss
branch
from
172e4bb
to
e18beee
Compare
andrebsguedes
force-pushed
the
snowflake-tss
branch
from
b402188
to
84128cc
Compare
| let n_files = 20_000; | ||
| let concurrency = 512; | ||
|
|
||
| // let t0 = std::time::Instant::now(); |
There was a problem hiding this comment.
just commenting in case this was left unintentionally
| }; | ||
| } | ||
|
|
||
| metric_const!(get_attempt_duration); |
There was a problem hiding this comment.
how can we read these metrics?
There was a problem hiding this comment.
We can't read them currently as they are not getting plumbed all the way to Julia, I am afraid the plumbing will have to come in a later PR.
|
|
||
| struct Buffer { | ||
| buf: Vec<u8>, | ||
| pos: usize, |
There was a problem hiding this comment.
From call: both of pos and filled are monotonically increasing and chase each other.
| } | ||
| } | ||
| State::Filling => { | ||
| if this.inbuf.filled().len() <= extract_tag_len { |
There was a problem hiding this comment.
from call: to be able to find out whether the buffered input is tag or sth else
|
A ToDo for one day is to test the encrypted put/get using randomized input/chunk size as a stress test to exercise all state machine transitions inside CrypterReader and CrypterWriter |
|
I managed to write the randomized test for CrypterWriter and CrypterReader and add the requested clarifications. |
This PR consolidates all changes on the Rust side for supporting Snowflake internal stages as a first class store.
It also restructures much of the code (with many misc changes) to ensure the encryption functionality could be introduced in a generic way.