Code

Better Azure blob storage support #1842

Code

Merged

wild-endeavor merged 24 commits intoflyteorg:masterflyteorg/flytekit:masterfrom Tom-Newton:tomnewton/add_azure_data_configTom-Newton/flytekit:tomnewton/add_azure_data_configCopy head branch name to clipboardSep 28, 2023

+256-37

Commits

Commits on Sep 19, 2023

• Add Azure blob storage configs

  Show description for ff9c57e
  Tom-Newton

  committed
  ff9c57e

  Copy full SHA for ff9c57e

• Use Azure args

  Show description for 2719f6a
  Tom-Newton

  committed
  2719f6a

  Copy full SHA for 2719f6a

• Add tenant id and correct some typos

  Show description for 1547f88
  Tom-Newton

  committed
  1547f88

  Copy full SHA for 1547f88

• Fix typos

  Show description for 8ed4a7d
  Tom-Newton

  committed
  8ed4a7d

  Copy full SHA for 8ed4a7d

• Extract get_storage_options_for_filesystem

  Show description for 172ee25
  Tom-Newton

  committed
  172ee25

  Copy full SHA for 172ee25

• Use get_storage_options when encoding structured datasets

  Show description for c94fb7d
  Tom-Newton

  committed
  c94fb7d

  Copy full SHA for c94fb7d

• Typo

  Show description for f8c8c6d
  Tom-Newton

  committed
  f8c8c6d

  Copy full SHA for f8c8c6d

• Remove unused import

  Show description for 83cfe5b
  Tom-Newton

  committed
  83cfe5b

  Copy full SHA for 83cfe5b

Commits on Sep 23, 2023

• Handle unrecognised protocol

  Show description for dc0f424
  Tom-Newton

  committed
  dc0f424

  Copy full SHA for dc0f424

• Handle difference between pandas and fsspec storage_options

  Show description for 418689f
  Tom-Newton

  committed
  418689f

  Copy full SHA for 418689f

• Add test for azure_setup_args

  Show description for f9da829
  Tom-Newton

  committed
  f9da829

  Copy full SHA for f9da829

• Add tests for initialising Azure filesystem and fix anonymous

  Show description for 2d4f54b
  Tom-Newton

  committed
  2d4f54b

  Copy full SHA for 2d4f54b

Commits on Sep 25, 2023

• Fix anon assertion

  Show description for f846352
  Tom-Newton

  committed
  f846352

  Copy full SHA for f846352

• Autoformat

  Show description for 7ffca40
  Tom-Newton

  committed
  7ffca40

  Copy full SHA for 7ffca40

• Improve test coverage for getting fsspec storage options

  Show description for 087021d
  Tom-Newton

  committed
  087021d

  Copy full SHA for 087021d

• Re-name env variable names + autoformat

  Show description for b5df7c1
  Tom-Newton

  committed
  b5df7c1

  Copy full SHA for b5df7c1

• Update tests for new env var names

  Show description for c3ee81f
  Tom-Newton

  committed
  c3ee81f

  Copy full SHA for c3ee81f

• Better mocking of os.environ

  Show description for b689b3c
  Tom-Newton

  committed
  b689b3c

  Copy full SHA for b689b3c

• Mostly working test for structured dataset filesystems use

  Show description for a279b47
  Tom-Newton

  committed
  a279b47

  Copy full SHA for a279b47

• Working test for structured dataset azure encode decode

  Show description for 80f34d8
  Tom-Newton

  committed
  80f34d8

  Copy full SHA for 80f34d8

• Fix get_pandas_storage_options

  Show description for 71d2486
  Tom-Newton

  committed
  71d2486

  Copy full SHA for 71d2486

• Tidy

  Show description for 5dd00ed
  Tom-Newton

  committed
  5dd00ed

  Copy full SHA for 5dd00ed

Commits on Sep 26, 2023

• Only assert on storage_options in structured dataset handler tests

  Show description for 572725b
  Tom-Newton

  committed
  572725b

  Copy full SHA for 572725b

Commits on Sep 27, 2023

• Type hints

  Show description for 72a440e
  Tom-Newton

  committed
  72a440e

  Copy full SHA for 72a440e