feat: implement azure blob storage toolkit #53668
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| implementation("com.azure:azure-storage-blob:12.29.0") | ||
|
|
||
| testImplementation("io.mockk:mockk:1.13.16") | ||
| testImplementation("org.junit.jupiter:junit-jupiter-api:5.11.3") |
There was a problem hiding this comment.
Nit: consider making the version a constant just to make it easy to update. See
as an example. 5.11.4 is the latest.| // If you need to filter by prefix: | ||
| // val pager = containerClient.listBlobs(ListBlobsOptions().setPrefix(prefix), null) | ||
|
|
||
| for (blobItem in pager) { |
There was a problem hiding this comment.
Nit: more Kotin way would be to do something like: pager.map { it.name }.forEach { name -> ... }
| } | ||
| } else { | ||
| // Sort by the “index” if you want strictly in ascending order: | ||
| // but we stored them in arrival order, which may be good enough |
There was a problem hiding this comment.
We should sort them by ascending index.
The index should correspond to the order the part occurs in the file.
Order of arrival might vary. Specifically, if you're using the current object load paradigm from ObjectStorageStreamLoader that we went over yesterday, order of arrival is not guaranteed to match file order. (This will change in the future, but the client should still guard against mis-ordered calls.)
There was a problem hiding this comment.
*Assuming that the order here determines how Azure assembles them.
There was a problem hiding this comment.
Oh the comment was old, I change the structure to use ConcurrentSkipListMap which sorts the entries on the basis of keys which is the index. So this is sorted. removed the comment
| api project(':airbyte-cdk:bulk:toolkits:bulk-cdk-toolkit-load-object-storage') | ||
|
|
||
| testImplementation("io.mockk:mockk:1.13.16") | ||
| testImplementation("org.junit.jupiter:junit-jupiter-api:5.11.4") |
There was a problem hiding this comment.
Nit: consider making the version a constant just to make it easy to update. See
as an example.
Issue : https://github.com/airbytehq/airbyte-internal-issues/issues/11550