diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index ead9fa466..f88c21953 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -293,7 +293,7 @@ jobs: pip install uv uv venv source .venv/bin/activate - uv pip install "flytekit>=1.12.2" flytekitplugins-deck-standard torch tabulate pyarrow + uv pip install "flytekit>=1.12.2" flytekitplugins-deck-standard torch tabulate pyarrow pandas pip freeze - name: Checkout flytesnacks uses: actions/checkout@v3 diff --git a/.readthedocs.yml b/.readthedocs.yml index b1825c215..796b75d7d 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -7,7 +7,7 @@ version: 2 build: os: "ubuntu-22.04" tools: - python: "3.11" + python: "3.12" apt_packages: - pandoc diff --git a/README.md b/README.md index bd2e73154..f6ad212d4 100644 --- a/README.md +++ b/README.md @@ -3,10 +3,10 @@ Flyte Logo

- Flyte User Guide & Tutorials + Flyte tutorials and integrations examples

- Flytesnacks encompasses code examples showcasing Flytekit Python + Flytesnacks encompasses tutorials and integrations examples showcasing Flytekit Python

@@ -30,11 +30,9 @@ > To get the hang of Python SDK, refer to the [Getting Started](https://docs.flyte.org/en/latest/getting_started.html) tutorial before exploring the examples. -The [User Guide](https://docs.flyte.org/projects/cookbook/en/latest/index.html) section has code examples, tips, and tricks that showcase the usage of Flyte features and integrations. +The [Tutorials](https://docs.flyte.org/en/latest/flytesnacks/tutorials/index.html) section has real-world examples, ranging from machine learning training, data processing to feature engineering. -The [Tutorials](https://docs.flyte.org/projects/cookbook/en/latest/tutorials.html) section has real-world examples, ranging from machine learning training, data processing to feature engineering. - -the [Integrations](https://docs.flyte.org/projects/cookbook/en/latest/integrations.html) section demonstrates how to use Flyte with other tools and frameworks. +the [Integrations](https://docs.flyte.org/en/latest/flytesnacks/integrations/index.html) section demonstrates how to use Flyte with other tools and frameworks. > Flytesnacks currently has all examples in Python (Flytekit Python SDK). @@ -44,7 +42,7 @@ the [Integrations](https://docs.flyte.org/projects/cookbook/en/latest/integratio -You can find the detailed contribution guide [here](docs/contribute.md). +You can find the detailed contribution guide [here](https://docs.flyte.org/en/latest/community/contribute_docs.html#contributing-tutorials-and-integrations-examples).

@@ -52,4 +50,4 @@ You can find the detailed contribution guide [here](docs/contribute.md).

-Refer to the [issues](https://docs.flyte.org/en/latest/community/contribute.html##file-an-issue) section in the contribution guide if you'd like to file an issue. +Refer to the [issues](https://docs.flyte.org/en/latest/community/contribute.html#file-an-issue) section in the contribution guide if you'd like to file an issue. diff --git a/dev-requirements.txt b/dev-requirements.txt index 39a1cdb2d..6716a9987 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -1,188 +1,185 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.12 # by the following command: # -# pip-compile ./dev-requirements.in +# pip-compile dev-requirements.in # -adlfs==2023.10.0 +adlfs==2024.7.0 # via flytekit -aiobotocore==2.5.4 +aiobotocore==2.15.1 # via s3fs -aiohttp==3.9.1 +aiohappyeyeballs==2.4.3 + # via aiohttp +aiohttp==3.10.8 # via # adlfs # aiobotocore # gcsfs # s3fs -aioitertools==0.11.0 +aioitertools==0.12.0 # via aiobotocore aiosignal==1.3.1 # via aiohttp -arrow==1.3.0 - # via cookiecutter -async-timeout==4.0.3 - # via aiohttp -attrs==23.1.0 - # via aiohttp -autoflake==2.2.1 +attrs==24.2.0 + # via + # aiohttp + # jsonlines +autoflake==2.3.1 # via -r dev-requirements.in -azure-core==1.29.5 +azure-core==1.31.0 # via # adlfs # azure-identity # azure-storage-blob azure-datalake-store==0.0.53 # via adlfs -azure-identity==1.15.0 +azure-identity==1.18.0 # via adlfs -azure-storage-blob==12.19.0 +azure-storage-blob==12.23.1 # via adlfs -binaryornot==0.4.4 - # via cookiecutter -botocore==1.31.17 +botocore==1.35.23 # via aiobotocore -cachetools==5.3.2 +cachetools==5.5.0 # via google-auth -certifi==2023.11.17 - # via - # kubernetes - # requests -cffi==1.16.0 +certifi==2024.8.30 + # via requests +cffi==1.17.1 # via # azure-datalake-store # cryptography cfgv==3.4.0 # via pre-commit -chardet==5.2.0 - # via binaryornot charset-normalizer==3.3.2 # via requests click==8.1.7 # via - # cookiecutter + # dask # flytekit # rich-click cloudpickle==3.0.0 - # via flytekit -codespell==2.2.6 + # via + # dask + # flytekit +codespell==2.3.0 # via -r dev-requirements.in -cookiecutter==2.5.0 - # via flytekit -coverage==7.3.2 +coverage==7.6.1 # via -r dev-requirements.in -croniter==2.0.1 +croniter==3.0.3 # via flytekit -cryptography==41.0.7 +cryptography==43.0.1 # via # azure-identity # azure-storage-blob # msal # pyjwt - # secretstorage +dask[array,dataframe]==2024.9.1 + # via + # -r dev-requirements.in + # dask-expr +dask-expr==1.1.15 + # via dask dataclasses-json==0.5.9 # via flytekit decorator==5.1.1 # via gcsfs diskcache==5.6.3 # via flytekit -distlib==0.3.7 +distlib==0.3.8 # via virtualenv -docker==6.1.3 +docker==7.1.0 # via flytekit -docstring-parser==0.15 +docstring-parser==0.16 # via flytekit -exceptiongroup==1.2.0 - # via pytest -filelock==3.13.1 +filelock==3.16.1 # via virtualenv -flyteidl==1.10.6 +flyteidl==1.13.4 # via flytekit -flytekit==1.10.2 +flytekit==1.13.7 # via -r dev-requirements.in -frozenlist==1.4.0 +frozenlist==1.4.1 # via # aiohttp # aiosignal -fsspec==2023.9.2 +fsspec==2024.9.0 + # via # adlfs + # dask # flytekit # gcsfs # s3fs -gcsfs==2023.9.2 +gcsfs==2024.9.0.post1 # via flytekit -google-api-core==2.15.0 +google-api-core==2.20.0 # via # google-cloud-core # google-cloud-storage -google-auth==2.25.1 +google-auth==2.35.0 # via # gcsfs # google-api-core # google-auth-oauthlib # google-cloud-core # google-cloud-storage - # kubernetes -google-auth-oauthlib==1.1.0 +google-auth-oauthlib==1.2.1 # via gcsfs google-cloud-core==2.4.1 # via google-cloud-storage -google-cloud-storage==2.13.0 +google-cloud-storage==2.18.2 # via gcsfs -google-crc32c==1.5.0 +google-crc32c==1.6.0 # via # google-cloud-storage # google-resumable-media -google-resumable-media==2.6.0 +google-resumable-media==2.7.2 # via google-cloud-storage -googleapis-common-protos==1.62.0 +googleapis-common-protos==1.65.0 # via # flyteidl # flytekit # google-api-core # grpcio-status -grpcio==1.60.0 + # protoc-gen-openapiv2 +grpcio==1.66.2 # via # flytekit # grpcio-status -grpcio-status==1.60.0 +grpcio-status==1.66.2 # via flytekit -identify==2.5.33 +identify==2.6.1 # via pre-commit -idna==3.6 +idna==3.10 # via # requests # yarl -importlib-metadata==7.0.0 - # via - # flytekit - # keyring +importlib-metadata==8.5.0 + # via flytekit iniconfig==2.0.0 # via pytest isodate==0.6.1 # via azure-storage-blob -jaraco-classes==3.3.0 +jaraco-classes==3.4.0 + # via keyring +jaraco-context==6.0.1 + # via keyring +jaraco-functools==4.1.0 # via keyring -jeepney==0.8.0 - # via - # keyring - # secretstorage -jinja2==3.1.2 - # via cookiecutter jmespath==1.0.1 # via botocore -joblib==1.3.2 +joblib==1.4.2 # via flytekit -jsonpickle==3.0.2 +jsonlines==4.0.0 # via flytekit -keyring==24.3.0 +jsonpickle==3.3.0 # via flytekit -kubernetes==28.1.0 +keyring==25.4.1 # via flytekit +locket==1.0.0 + # via partd markdown-it-py==3.0.0 - # via rich -markupsafe==2.1.3 - # via jinja2 -marshmallow==3.20.1 + # via + # flytekit + # rich +marshmallow==3.22.0 # via # dataclasses-json # marshmallow-enum @@ -193,7 +190,7 @@ marshmallow-enum==1.5.1 # flytekit marshmallow-jsonschema==0.13.0 # via flytekit -mashumaro==3.11 +mashumaro==3.13.1 # via # -r dev-requirements.in # flytekit @@ -201,157 +198,151 @@ mdurl==0.1.2 # via markdown-it-py mock==5.1.0 # via -r dev-requirements.in -more-itertools==10.1.0 - # via jaraco-classes -msal==1.26.0 +more-itertools==10.5.0 + # via + # jaraco-classes + # jaraco-functools +msal==1.31.0 # via # azure-datalake-store # azure-identity # msal-extensions -msal-extensions==1.0.0 +msal-extensions==1.2.0 # via azure-identity -multidict==6.0.4 +multidict==6.1.0 # via # aiohttp # yarl -mypy==1.7.1 +mypy==1.11.2 # via -r dev-requirements.in mypy-extensions==1.0.0 # via # mypy # typing-inspect -nodeenv==1.8.0 +nodeenv==1.9.1 # via pre-commit -numpy==1.26.2 +numpy==2.1.1 # via - # flytekit + # dask # pandas # pyarrow + # scipy + # xgboost oauthlib==3.2.2 + # via requests-oauthlib +packaging==24.1 # via - # kubernetes - # requests-oauthlib -packaging==23.2 - # via - # docker + # dask # marshmallow # pytest -pandas==1.5.3 - # via flytekit -platformdirs==4.1.0 +pandas==2.2.3 # via - # virtualenv -pluggy==1.3.0 + # dask + # dask-expr +partd==1.4.2 + # via dask +platformdirs==4.3.6 + # via virtualenv +pluggy==1.5.0 # via pytest -portalocker==2.8.2 +portalocker==2.10.1 # via msal-extensions -pre-commit==3.5.0 +pre-commit==3.8.0 # via -r dev-requirements.in -protobuf==4.24.4 +proto-plus==1.24.0 + # via google-api-core +protobuf==5.28.2 # via # flyteidl # flytekit # google-api-core # googleapis-common-protos # grpcio-status - # protoc-gen-swagger -protoc-gen-swagger==0.1.0 + # proto-plus + # protoc-gen-openapiv2 +protoc-gen-openapiv2==0.0.1 # via flyteidl -pyarrow==14.0.1 - # via flytekit -pyasn1==0.5.1 +pyarrow==17.0.0 + # via dask-expr +pyasn1==0.6.1 # via # pyasn1-modules # rsa -pyasn1-modules==0.3.0 +pyasn1-modules==0.4.1 # via google-auth -pycparser==2.21 +pycparser==2.22 # via cffi -pygments==2.17.2 - # via rich -pyjwt[crypto]==2.8.0 +pyflakes==3.2.0 + # via autoflake +pygments==2.18.0 + # via + # flytekit + # rich +pyjwt[crypto]==2.9.0 # via # msal # pyjwt -pytest==7.4.3 +pytest==8.3.3 # via -r dev-requirements.in -python-dateutil==2.8.2 +python-dateutil==2.9.0.post0 # via - # arrow # botocore # croniter - # kubernetes # pandas python-json-logger==2.0.7 # via flytekit -python-slugify==8.0.1 - # via cookiecutter pytimeparse==1.1.8 # via flytekit -pytz==2023.3.post1 +pytz==2024.2 # via # croniter # pandas -pyyaml==6.0.1 +pyyaml==6.0.2 # via - # cookiecutter + # dask # flytekit - # kubernetes # pre-commit -requests==2.31.0 +requests==2.32.3 # via # azure-core # azure-datalake-store - # cookiecutter # docker # flytekit # gcsfs # google-api-core # google-cloud-storage - # kubernetes # msal # requests-oauthlib -requests-oauthlib==1.3.1 - # via - # google-auth-oauthlib - # kubernetes -rich==13.7.0 +requests-oauthlib==2.0.0 + # via google-auth-oauthlib +rich==13.9.1 # via - # cookiecutter # flytekit # rich-click -rich-click==1.7.2 +rich-click==1.8.3 # via flytekit rsa==4.9 # via google-auth -s3fs==2023.9.2 +s3fs==2024.9.0 # via flytekit -secretstorage==3.3.3 - # via keyring +scipy==1.14.1 + # via xgboost six==1.16.0 # via # azure-core # isodate - # kubernetes # python-dateutil -statsd==3.3.0 +statsd==4.0.1 # via flytekit -text-unidecode==1.3 - # via python-slugify -tomli==2.0.1 +toolz==0.12.1 # via - # autoflake - # black - # flake8-black - # mypy - # pytest -types-python-dateutil==2.8.19.14 - # via arrow -typing-extensions==4.8.0 + # dask + # partd +typing-extensions==4.12.2 # via # azure-core + # azure-identity # azure-storage-blob - # black # flytekit # mashumaro # mypy @@ -359,25 +350,21 @@ typing-extensions==4.8.0 # typing-inspect typing-inspect==0.9.0 # via dataclasses-json -urllib3==1.26.18 +tzdata==2024.2 + # via pandas +urllib3==2.2.3 # via # botocore # docker # flytekit - # kubernetes # requests -virtualenv==20.25.0 +virtualenv==20.26.6 # via pre-commit -websocket-client==1.7.0 - # via - # docker - # kubernetes wrapt==1.16.0 # via aiobotocore -yarl==1.9.4 +xgboost==2.1.1 + # via -r dev-requirements.in +yarl==1.13.1 # via aiohttp -zipp==3.17.0 +zipp==3.20.2 # via importlib-metadata - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/docs-requirements.in b/docs-requirements.in index 04c8ceb24..fb2cbba01 100644 --- a/docs-requirements.in +++ b/docs-requirements.in @@ -1,7 +1,7 @@ flytekit -fsspec<2023.10.0 +pydata-sphinx-theme +fsspec mashumaro -git+https://github.com/flyteorg/furo@main pillow psycopg2-binary flytekitplugins-deck-standard @@ -9,7 +9,6 @@ flytekitplugins-kfpytorch flytekitplugins-sqlalchemy jupytext sphinx -sphinx_rtd_theme sphinx-gallery sphinx-prompt sphinx-code-include @@ -27,6 +26,6 @@ grpcio grpcio-status myst-nb ipython!=8.7.0 -sphinx-tags==0.2.1 +sphinx-tags scikit-learn torch diff --git a/docs-requirements.txt b/docs-requirements.txt index 517adf4fd..8df2d7a24 100644 --- a/docs-requirements.txt +++ b/docs-requirements.txt @@ -1,88 +1,80 @@ # -# This file is autogenerated by pip-compile with Python 3.10 +# This file is autogenerated by pip-compile with Python 3.12 # by the following command: # -# /Library/Developer/CommandLineTools/usr/bin/make docs-requirements.txt +# pip-compile docs-requirements.in # -adlfs==2023.10.0 +accessible-pygments==0.0.5 + # via pydata-sphinx-theme +adlfs==2024.7.0 # via flytekit -aiobotocore==2.5.4 +aiobotocore==2.15.1 # via s3fs -aiohttp==3.9.1 +aiohappyeyeballs==2.4.3 + # via aiohttp +aiohttp==3.10.8 # via # adlfs # aiobotocore # gcsfs # s3fs -aioitertools==0.11.0 +aioitertools==0.12.0 # via aiobotocore aiosignal==1.3.1 # via aiohttp -alabaster==0.7.15 +alabaster==1.0.0 # via sphinx -annotated-types==0.6.0 - # via pydantic -anyio==4.2.0 - # via azure-core -appnope==0.1.3 +appnope==0.1.4 # via ipykernel -arrow==1.3.0 - # via cookiecutter -astroid==3.0.2 +astroid==3.3.4 # via # -r docs-requirements.in # sphinx-autoapi asttokens==2.4.1 # via stack-data -async-timeout==4.0.3 - # via aiohttp -attrs==23.2.0 +attrs==24.2.0 # via # aiohttp + # jsonlines # jsonschema # jupyter-cache # referencing - # visions -azure-core==1.29.6 +azure-core==1.31.0 # via # adlfs # azure-identity # azure-storage-blob azure-datalake-store==0.0.53 # via adlfs -azure-identity==1.15.0 +azure-identity==1.18.0 # via adlfs -azure-storage-blob==12.19.0 +azure-storage-blob==12.23.1 # via adlfs -babel==2.14.0 - # via sphinx -beautifulsoup4==4.12.2 +babel==2.16.0 # via - # furo + # pydata-sphinx-theme + # sphinx +beautifulsoup4==4.12.3 + # via + # pydata-sphinx-theme # sphinx-code-include -binaryornot==0.4.4 - # via cookiecutter -botocore==1.31.17 +botocore==1.35.23 # via aiobotocore -cachetools==5.3.2 +cachetools==5.5.0 # via google-auth -certifi==2023.11.17 +certifi==2024.8.30 # via # kubernetes # requests -cffi==1.16.0 + # sphinx-prompt +cffi==1.17.1 # via # azure-datalake-store # cryptography -cfgv==3.4.0 - # via pre-commit -chardet==5.2.0 - # via binaryornot charset-normalizer==3.3.2 # via requests click==8.1.7 # via - # cookiecutter # flytekit # jupyter-cache # rich-click @@ -90,29 +82,19 @@ cloudpickle==3.0.0 # via # flytekit # flytekitplugins-kfpytorch -comm==0.2.1 - # via - # ipykernel - # ipywidgets -contourpy==1.2.0 - # via matplotlib -cookiecutter==2.5.0 - # via flytekit -croniter==2.0.1 +comm==0.2.2 + # via ipykernel +croniter==3.0.3 # via flytekit -cryptography==41.0.7 +cryptography==43.0.1 # via # azure-identity # azure-storage-blob # msal # pyjwt -cycler==0.12.1 - # via matplotlib -dacite==1.8.1 - # via ydata-profiling dataclasses-json==0.5.9 # via flytekit -debugpy==1.8.0 +debugpy==1.8.6 # via ipykernel decorator==5.1.1 # via @@ -120,54 +102,47 @@ decorator==5.1.1 # ipython diskcache==5.6.3 # via flytekit -distlib==0.3.8 - # via virtualenv -docker==6.1.3 +docker==7.1.0 # via flytekit -docstring-parser==0.15 +docstring-parser==0.16 # via flytekit -docutils==0.17.1 +docutils==0.21.2 # via # myst-parser + # pydata-sphinx-theme # sphinx # sphinx-panels - # sphinx-rtd-theme + # sphinx-prompt # sphinx-tabs -exceptiongroup==1.2.0 - # via - # anyio - # ipython -executing==2.0.1 +durationpy==0.9 + # via kubernetes +executing==2.1.0 # via stack-data -fastjsonschema==2.19.1 +fastjsonschema==2.20.0 # via nbformat -filelock==3.13.1 - # via - # torch - # virtualenv -flyteidl==1.10.6 +filelock==3.16.1 + # via torch +flyteidl==1.13.4 # via # flytekit # flytekitplugins-kfpytorch -flytekit==1.10.2 +flytekit==1.13.7 # via # -r docs-requirements.in # flytekitplugins-deck-standard # flytekitplugins-kfpytorch # flytekitplugins-sqlalchemy -flytekitplugins-deck-standard==1.10.2 +flytekitplugins-deck-standard==1.13.7 # via -r docs-requirements.in -flytekitplugins-kfpytorch==1.10.2 +flytekitplugins-kfpytorch==1.13.7 # via -r docs-requirements.in -flytekitplugins-sqlalchemy==1.10.2 +flytekitplugins-sqlalchemy==1.13.7 # via -r docs-requirements.in -fonttools==4.47.0 - # via matplotlib frozenlist==1.4.1 # via # aiohttp # aiosignal -fsspec==2023.9.2 +fsspec==2024.9.0 # via # -r docs-requirements.in # adlfs @@ -175,15 +150,13 @@ fsspec==2023.9.2 # gcsfs # s3fs # torch -furo @ git+https://github.com/flyteorg/furo@main - # via -r docs-requirements.in -gcsfs==2023.9.2 +gcsfs==2024.9.0.post1 # via flytekit -google-api-core==2.15.0 +google-api-core==2.20.0 # via # google-cloud-core # google-cloud-storage -google-auth==2.26.1 +google-auth==2.35.0 # via # gcsfs # google-api-core @@ -191,127 +164,111 @@ google-auth==2.26.1 # google-cloud-core # google-cloud-storage # kubernetes -google-auth-oauthlib==1.2.0 +google-auth-oauthlib==1.2.1 # via gcsfs google-cloud-core==2.4.1 # via google-cloud-storage -google-cloud-storage==2.14.0 +google-cloud-storage==2.18.2 # via gcsfs -google-crc32c==1.5.0 +google-crc32c==1.6.0 # via # google-cloud-storage # google-resumable-media -google-resumable-media==2.7.0 +google-resumable-media==2.7.2 # via google-cloud-storage -googleapis-common-protos==1.62.0 +googleapis-common-protos==1.65.0 # via # flyteidl # flytekit # google-api-core # grpcio-status -grpcio==1.60.0 + # protoc-gen-openapiv2 +grpcio==1.66.2 # via # -r docs-requirements.in # flytekit # grpcio-status -grpcio-status==1.60.0 +grpcio-status==1.66.2 # via # -r docs-requirements.in # flytekit -htmlmin==0.1.12 - # via ydata-profiling -identify==2.5.33 - # via pre-commit -idna==3.6 +idna==3.10 # via - # anyio # requests + # sphinx-prompt # yarl -imagehash==4.3.1 - # via - # visions - # ydata-profiling imagesize==1.4.1 # via sphinx -importlib-metadata==7.0.1 +importlib-metadata==8.5.0 # via # flytekit # jupyter-cache - # keyring # myst-nb -ipykernel==6.28.0 +ipykernel==6.29.5 # via myst-nb -ipython==8.20.0 +ipython==8.28.0 # via # -r docs-requirements.in # ipykernel - # ipywidgets # myst-nb -ipywidgets==8.1.1 - # via flytekitplugins-deck-standard isodate==0.6.1 # via azure-storage-blob -jaraco-classes==3.3.0 +jaraco-classes==3.4.0 + # via keyring +jaraco-context==6.0.1 + # via keyring +jaraco-functools==4.1.0 # via keyring jedi==0.19.1 # via ipython -jinja2==3.0.3 +jinja2==3.1.4 # via - # cookiecutter # myst-parser # sphinx # sphinx-autoapi - # sphinx-tabs # torch - # ydata-profiling jmespath==1.0.1 # via botocore -joblib==1.3.2 +joblib==1.4.2 # via # flytekit - # phik # scikit-learn -jsonpickle==3.0.2 +jsonlines==4.0.0 + # via flytekit +jsonpickle==3.3.0 # via flytekit -jsonschema==4.20.0 +jsonschema==4.23.0 # via nbformat jsonschema-specifications==2023.12.1 # via jsonschema -jupyter-cache==0.6.1 +jupyter-cache==1.0.0 # via myst-nb -jupyter-client==8.6.0 +jupyter-client==8.6.3 # via # ipykernel # nbclient -jupyter-core==5.7.1 +jupyter-core==5.7.2 # via # ipykernel # jupyter-client # nbclient # nbformat -jupyterlab-widgets==3.0.9 - # via ipywidgets -jupytext==1.16.0 +jupytext==1.16.4 # via -r docs-requirements.in -keyring==24.3.0 +keyring==25.4.1 # via flytekit -kiwisolver==1.4.5 - # via matplotlib -kubernetes==29.0.0 - # via flytekit -llvmlite==0.41.1 - # via numba -markdown==3.5.1 - # via flytekitplugins-deck-standard -markdown-it-py==2.2.0 +kubernetes==31.0.0 + # via flytekitplugins-kfpytorch +markdown-it-py==3.0.0 # via + # flytekit # jupytext # mdit-py-plugins # myst-parser # rich -markupsafe==2.1.3 +markupsafe==2.1.5 # via jinja2 -marshmallow==3.20.1 +marshmallow==3.22.0 # via # dataclasses-json # marshmallow-enum @@ -322,233 +279,170 @@ marshmallow-enum==1.5.1 # flytekit marshmallow-jsonschema==0.13.0 # via flytekit -mashumaro==3.11 +mashumaro==3.13.1 # via # -r docs-requirements.in # flytekit -matplotlib==3.8.2 - # via - # phik - # seaborn - # wordcloud - # ydata-profiling -matplotlib-inline==0.1.6 +matplotlib-inline==0.1.7 # via # ipykernel # ipython -mdit-py-plugins==0.3.5 +mdit-py-plugins==0.4.2 # via # jupytext # myst-parser mdurl==0.1.2 # via markdown-it-py -more-itertools==10.2.0 - # via jaraco-classes +more-itertools==10.5.0 + # via + # jaraco-classes + # jaraco-functools mpmath==1.3.0 # via sympy -msal==1.26.0 +msal==1.31.0 # via # azure-datalake-store # azure-identity # msal-extensions -msal-extensions==1.1.0 +msal-extensions==1.2.0 # via azure-identity -multidict==6.0.4 +multidict==6.1.0 # via # aiohttp # yarl -multimethod==1.10 - # via - # visions - # ydata-profiling mypy-extensions==1.0.0 # via typing-inspect -myst-nb==0.17.2 +myst-nb==1.1.2 # via -r docs-requirements.in -myst-parser==0.18.1 +myst-parser==4.0.0 # via myst-nb -nbclient==0.7.4 +nbclient==0.10.0 # via # jupyter-cache # myst-nb -nbformat==5.9.2 +nbformat==5.10.4 # via # jupyter-cache # jupytext # myst-nb # nbclient -nest-asyncio==1.5.8 +nest-asyncio==1.6.0 # via ipykernel -networkx==3.2.1 +networkx==3.3 + # via torch +numpy==2.1.1 # via - # torch - # visions -nodeenv==1.8.0 - # via pre-commit -numba==0.58.1 - # via ydata-profiling -numpy==1.25.2 - # via - # contourpy - # flytekit - # imagehash - # matplotlib - # numba # pandas - # patsy - # phik - # pyarrow - # pywavelets # scikit-learn # scipy - # seaborn - # statsmodels - # visions - # wordcloud - # ydata-profiling oauthlib==3.2.2 # via # kubernetes # requests-oauthlib -packaging==23.2 +packaging==24.1 # via - # docker # ipykernel # jupytext # marshmallow - # matplotlib - # msal-extensions - # plotly + # pydata-sphinx-theme # sphinx - # statsmodels -pandas==1.5.3 - # via - # flytekit - # phik - # seaborn - # statsmodels - # visions - # ydata-profiling -parso==0.8.3 +pandas==2.2.3 + # via flytekitplugins-sqlalchemy +parso==0.8.4 # via jedi -patsy==0.5.6 - # via statsmodels pexpect==4.9.0 # via ipython -phik==0.12.3 - # via ydata-profiling -pillow==10.2.0 +pillow==10.4.0 # via # -r docs-requirements.in - # imagehash - # matplotlib # sphinx-gallery - # visions - # wordcloud -platformdirs==4.1.0 - # via - # jupyter-core - # virtualenv -plotly==5.18.0 - # via flytekitplugins-deck-standard -portalocker==2.8.2 +platformdirs==4.3.6 + # via jupyter-core +portalocker==2.10.1 # via msal-extensions -pre-commit==3.6.0 - # via sphinx-tags -prompt-toolkit==3.0.43 +prompt-toolkit==3.0.48 # via ipython -protobuf==4.24.4 +proto-plus==1.24.0 + # via google-api-core +protobuf==5.28.2 # via # flyteidl # flytekit # google-api-core # googleapis-common-protos # grpcio-status - # protoc-gen-swagger -protoc-gen-swagger==0.1.0 + # proto-plus + # protoc-gen-openapiv2 +protoc-gen-openapiv2==0.0.1 # via flyteidl -psutil==5.9.7 +psutil==6.0.0 # via ipykernel psycopg2-binary==2.9.9 # via -r docs-requirements.in ptyprocess==0.7.0 # via pexpect -pure-eval==0.2.2 +pure-eval==0.2.3 # via stack-data -pyarrow==14.0.2 - # via flytekit -pyasn1==0.5.1 +pyasn1==0.6.1 # via # pyasn1-modules # rsa -pyasn1-modules==0.3.0 +pyasn1-modules==0.4.1 # via google-auth -pycparser==2.21 +pycparser==2.22 # via cffi -pydantic==2.5.3 - # via ydata-profiling -pydantic-core==2.14.6 - # via pydantic -pygments==2.17.2 +pydata-sphinx-theme==0.15.4 + # via -r docs-requirements.in +pygments==2.18.0 # via - # furo + # accessible-pygments + # flytekit # ipython + # pydata-sphinx-theme # rich # sphinx # sphinx-prompt # sphinx-tabs -pyjwt[crypto]==2.8.0 +pyjwt[crypto]==2.9.0 # via # msal # pyjwt -pyparsing==3.1.1 - # via matplotlib -python-dateutil==2.8.2 +python-dateutil==2.9.0.post0 # via - # arrow # botocore # croniter # jupyter-client # kubernetes - # matplotlib # pandas python-json-logger==2.0.7 # via flytekit -python-slugify==8.0.1 - # via cookiecutter pytimeparse==1.1.8 # via flytekit -pytz==2023.3.post1 +pytz==2024.2 # via # croniter # pandas -pywavelets==1.5.0 - # via imagehash -pyyaml==6.0.1 +pyyaml==6.0.2 # via - # cookiecutter # flytekit # jupyter-cache # jupytext # kubernetes # myst-nb # myst-parser - # pre-commit # sphinx-autoapi - # ydata-profiling -pyzmq==25.1.2 +pyzmq==26.2.0 # via # ipykernel # jupyter-client -referencing==0.32.1 +referencing==0.35.1 # via # jsonschema # jsonschema-specifications -requests==2.31.0 +requests==2.32.3 # via # azure-core # azure-datalake-store - # cookiecutter # docker # flytekit # gcsfs @@ -559,61 +453,48 @@ requests==2.31.0 # requests-oauthlib # sphinx # sphinxcontrib-youtube - # ydata-profiling -requests-oauthlib==1.3.1 +requests-oauthlib==2.0.0 # via # google-auth-oauthlib # kubernetes -rich==13.7.0 +rich==13.9.1 # via - # cookiecutter # flytekit # rich-click -rich-click==1.7.3 +rich-click==1.8.3 # via flytekit -rpds-py==0.16.2 +rpds-py==0.20.0 # via # jsonschema # referencing rsa==4.9 # via google-auth -s3fs==2023.9.2 +s3fs==2024.9.0 # via flytekit -scikit-learn==1.3.2 +scikit-learn==1.5.2 # via -r docs-requirements.in -scipy==1.11.4 - # via - # imagehash - # phik - # scikit-learn - # statsmodels - # ydata-profiling -seaborn==0.12.2 - # via ydata-profiling +scipy==1.14.1 + # via scikit-learn six==1.16.0 # via # asttokens # azure-core # isodate # kubernetes - # patsy # python-dateutil # sphinx-code-include # sphinxext-remoteliteralinclude -sniffio==1.3.0 - # via anyio snowballstemmer==2.2.0 # via sphinx -soupsieve==2.5 +soupsieve==2.6 # via beautifulsoup4 -sphinx==4.5.0 +sphinx==8.0.2 # via # -r docs-requirements.in - # furo # myst-nb # myst-parser + # pydata-sphinx-theme # sphinx-autoapi - # sphinx-basic-ng # sphinx-code-include # sphinx-copybutton # sphinx-fontawesome @@ -621,154 +502,112 @@ sphinx==4.5.0 # sphinx-panels # sphinx-prompt # sphinx-reredirects - # sphinx-rtd-theme # sphinx-tabs # sphinx-tags - # sphinxcontrib-jquery # sphinxcontrib-youtube # sphinxext-remoteliteralinclude -sphinx-autoapi==2.0.1 +sphinx-autoapi==3.3.2 # via -r docs-requirements.in -sphinx-basic-ng==1.0.0b2 - # via furo -sphinx-code-include==1.1.1 +sphinx-code-include==1.4.0 # via -r docs-requirements.in sphinx-copybutton==0.5.2 # via -r docs-requirements.in sphinx-fontawesome==0.0.6 # via -r docs-requirements.in -sphinx-gallery==0.15.0 +sphinx-gallery==0.17.1 # via -r docs-requirements.in -sphinx-panels==0.6.0 +sphinx-panels==0.4.1 # via -r docs-requirements.in -sphinx-prompt==1.5.0 +sphinx-prompt==1.9.0 # via -r docs-requirements.in -sphinx-reredirects==0.1.3 +sphinx-reredirects==0.1.5 # via -r docs-requirements.in -sphinx-rtd-theme==1.3.0 +sphinx-tabs==3.4.5 # via -r docs-requirements.in -sphinx-tabs==3.4.0 +sphinx-tags==0.4 # via -r docs-requirements.in -sphinx-tags==0.2.1 - # via -r docs-requirements.in -sphinxcontrib-applehelp==1.0.4 +sphinxcontrib-applehelp==2.0.0 # via sphinx -sphinxcontrib-devhelp==1.0.2 +sphinxcontrib-devhelp==2.0.0 # via sphinx -sphinxcontrib-htmlhelp==2.0.1 +sphinxcontrib-htmlhelp==2.1.0 # via sphinx -sphinxcontrib-jquery==4.1 - # via sphinx-rtd-theme sphinxcontrib-jsmath==1.0.1 # via sphinx sphinxcontrib-mermaid==0.9.2 # via -r docs-requirements.in -sphinxcontrib-qthelp==1.0.3 +sphinxcontrib-qthelp==2.0.0 # via sphinx -sphinxcontrib-serializinghtml==1.1.5 +sphinxcontrib-serializinghtml==2.0.0 # via sphinx -sphinxcontrib-youtube==1.3.0 +sphinxcontrib-youtube==1.4.1 # via -r docs-requirements.in sphinxext-remoteliteralinclude==0.4.0 # via -r docs-requirements.in -sqlalchemy==2.0.25 +sqlalchemy==2.0.35 # via # flytekitplugins-sqlalchemy # jupyter-cache stack-data==0.6.3 # via ipython -statsd==3.3.0 +statsd==4.0.1 # via flytekit -statsmodels==0.14.1 - # via ydata-profiling -sympy==1.12 +sympy==1.13.3 # via torch tabulate==0.9.0 # via jupyter-cache -tangled-up-in-unicode==0.2.0 - # via visions -tenacity==8.2.3 - # via plotly -text-unidecode==1.3 - # via python-slugify -threadpoolctl==3.2.0 +threadpoolctl==3.5.0 # via scikit-learn -toml==0.10.2 - # via jupytext -torch==2.1.2 +torch==2.4.1 # via -r docs-requirements.in -tornado==6.4 +tornado==6.4.1 # via # ipykernel # jupyter-client -tqdm==4.66.1 - # via ydata-profiling -traitlets==5.14.1 +traitlets==5.14.3 # via # comm # ipykernel # ipython - # ipywidgets # jupyter-client # jupyter-core # matplotlib-inline # nbclient # nbformat -typeguard==4.1.5 - # via ydata-profiling -types-python-dateutil==2.8.19.20240106 - # via arrow -typing-extensions==4.9.0 - # via - # anyio - # astroid +typing-extensions==4.12.2 + # via # azure-core + # azure-identity # azure-storage-blob # flytekit # mashumaro # myst-nb - # myst-parser - # pydantic - # pydantic-core + # pydata-sphinx-theme # rich-click # sqlalchemy # torch - # typeguard # typing-inspect typing-inspect==0.9.0 # via dataclasses-json -unidecode==1.3.7 - # via sphinx-autoapi -urllib3==1.26.18 +tzdata==2024.2 + # via pandas +urllib3==2.2.3 # via # botocore # docker # flytekit # kubernetes # requests -virtualenv==20.25.0 - # via pre-commit -visions[type-image-path]==0.7.5 - # via - # visions - # ydata-profiling + # sphinx-prompt wcwidth==0.2.13 # via prompt-toolkit -websocket-client==1.7.0 - # via - # docker - # kubernetes -widgetsnbextension==4.0.9 - # via ipywidgets -wordcloud==1.9.3 - # via ydata-profiling +websocket-client==1.8.0 + # via kubernetes wrapt==1.16.0 # via aiobotocore -yarl==1.9.4 +yarl==1.13.1 # via aiohttp -ydata-profiling==4.6.4 - # via flytekitplugins-deck-standard -zipp==3.17.0 +zipp==3.20.2 # via importlib-metadata # The following packages are considered to be unsafe in a requirements file: diff --git a/docs/conf.py b/docs/conf.py index d8ce10fca..9853316bc 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -97,7 +97,12 @@ ] include_patterns = [ + "index.md", + "_tags/tagsindex", + "*.md", + "**/*.md", "auto_examples/**/index.md", + "auto_examples/**/*.md", ] # The master toctree document. @@ -121,7 +126,7 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = "furo" +html_theme = "pydata_sphinx_theme" html_title = "Flyte" announcement = """ @@ -129,6 +134,7 @@ Please visit the new documentation here. """ +""" html_theme_options = { "light_css_variables": { "color-brand-primary": "#4300c9", @@ -148,6 +154,7 @@ "docs_path": "docs", # path to documentation source "announcement": announcement, } +""" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, @@ -183,9 +190,6 @@ "extending", "productionizing", "testing", - "bigquery_plugin", - "databricks_plugin", - "snowflake_plugin", ] # intersphinx configuration diff --git a/docs/contribute.md b/docs/contribute.md deleted file mode 100644 index 2bd45bff3..000000000 --- a/docs/contribute.md +++ /dev/null @@ -1,359 +0,0 @@ -# Contributing to User Guide, Tutorials and Integrations - -```{eval-rst} -.. tags:: Contribute, Basic -``` - -The examples documentation provides an easy way for the community to learn about the rich set of -features that Flyte offers, and we are constantly improving them with your help! - -Whether you're a novice or experienced software engineer, data scientist, or machine learning -practitioner, all contributions are welcome! - -## How to contribute - -The Flyte documentation examples guides are broken up into three types: - -1. {ref}`User Guides `: These are short, simple guides that demonstrate how to use a particular Flyte feature. The Markdown files live in the [flyte repository](https://github.com/flyteorg/flyte/tree/master/docs/user_guide), while the example code lives in flytesnacks. - These examples should be runnable locally. **Note:** The comments in the user guide Python files **will not** be rendered as user-facing docs. To update the user-facing documentation, open a pull request in the flyte repo. -2. {ref}`Tutorials `: These are longer, more advanced guides that use multiple Flyte features to solve - real-world problems. Tutorials are generally more complex examples that may require extra setup or that can only run - on larger clusters. -3. {ref}`Integrations `: These examples showcase how to use the Flyte plugins that integrate with the - broader data and ML ecosystem. - -The first step to contributing an example is to open up a -[documentation issue](https://github.com/flyteorg/flyte/issues/new?assignees=&labels=documentation%2Cuntriaged&template=docs_issue.yaml&title=%5BDocs%5D+) -to articulate the kind of example you want to write. The Flyte maintainers will guide and help you figure out where your example would fit best. - -## Creating an example - -:::{admonition} Prerequisites -Follow the {ref}`env_setup` guide to get your development environment ready. -::: - -The `flytesnacks` repo examples live in the `examples` directory, where each -subdirectory contains a self-contained example project that covers a particular -feature, integration, or use case. - -```{code-block} bash -examples -├── README.md -├── airflow_plugin -├── athena_plugin -├── aws_batch_plugin -├── basics -├── bigquery_plugin -... -``` - -### Adding an example script to an existing project - -If you're adding a new example to an existing project, you can simply create a -new `.py` file in the appropriate directory. For example, if you want to add a new -example in the `examples/basics` project, simply do: - -```{prompt} bash -touch examples/basics/my_new_example.py -``` - -If you are creating a new user guide example, you can reference the code in the user guide documentation using the `rli` (remoteliteralinclude) directive. - -If you are creating a new integration or tutorial example, add the example to the `README.md` file of the -example project as an entry in the `auto-examples-toc` directive: - -````{code-block} -```{auto-examples-toc} -... -my_new_example -``` -```` - -### Creating a new example project - -````{important} -If you're creating a new example in the User Guide, Tutorials, or Integrations -that doesn't fit into any of the existing subdirectories, you'll need to setup a -new example project. - -In the `flytesnacks` root directory, create one with: - -```{prompt} bash -./scripts/create-example-project new_example_project -``` - -This will create a new directory under `examples`: - -```{code-block} bash -examples/new_example_project -├── Dockerfile -├── README.md -├── new_example_project -│   ├── __init__.py -│   └── example.py -├── requirements.in -└── requirements.txt -``` - -```` - -### Creating python examples - -#### User guide examples - -If you are writing a user guide example, write your example Python script in regular Python, with regular comments. These comments **will not** be extracted from the Python file and turned into user-facing documentation. To update user-facing user guide documentation, edit the user guide files in the [flyte repository](https://github.com/flyteorg/flyte/tree/master/docs/user_guide). You can use the `rli` ([remoteliteralinclude](https://github.com/wpilibsuite/sphinxext-remoteliteralinclude/blob/main/README.md)) directive to include snippets of code from your example Python file. - -#### Tutorial or integration examples - -If you are writing a tutorial or integration example, write your example Python script in [percent format](https://jupytext.readthedocs.io/en/latest/formats.html#the-percent-format), -which allows you to interleave python code and markdown in the same file. Each -code cell should be delimited by `# %%`, and each markdown cell should be -delimited with `# %% [markdown]`. - -```{code-block} python -# %% -print("Hello World!") - -# %% [markdown] -# This is a markdown cell - -# %% -print("This is another code cell") -``` - -Markdown cells have access to sphinx directives through the -[myst markdown](https://myst-parser.readthedocs.io/en/latest/) format, -which is a flavor of markdown that makes it easier to write documentation while -giving you the utilities of sphinx. `flytesnacks` uses the -[myst-nb](https://myst-nb.readthedocs.io/en/latest/) and -[jupytext](https://github.com/mwouts/jupytext) packages to interpret the -python files as rst-compatible files. - -### Writing examples: explain what the code does - -Following the [literate programming](https://en.wikipedia.org/wiki/Literate_programming) paradigm, make sure to -interleave explanations in the `*.py` files containing the code example. - -:::{admonition} A Simple Example -:class: tip - -Here's a code snippet that defines a function that takes two positional arguments and one keyword argument: - -```python -def function(x, y, z=3): - return x + y * z -``` - -As you can see, `function` adds the two first arguments and multiplies the sum with the third keyword -argument. Can you think of a better name for this `function`? -::: - -Explanations don't have to be this detailed for such a simple example, but you can imagine how this makes for a better -reading experience for more complicated examples. - -### Creating examples in other formats - -Writing examples in `.py` files is preferred since they are easily tested and -packaged, but `flytesnacks` also supports examples written in `.ipynb` and -`.md` files in myst markdown format. This is useful in the following cases: - -- `.ipynb`: When a `.py` example needs a companion jupyter notebook as a task, e.g. - to illustrate the use of {py:class}`~flytekitplugins.papermill.NotebookTask`s, - or when an example is intended to be run from a notebook. -- `.md`: When a piece of documentation doesn't require testable or packaged - flyte tasks/workflows, an example page can be written as a myst markdown file. - -**Note:** If you want to add Markdown files to a user guide example project, add them to the [flyte repository](https://github.com/flyteorg/flyte/tree/master/docs/user_guide) instead. - -## Writing a README - -The `README.md` file needs to capture the _what_, _why_, and _how_ of the example. - -- What is the integration about? Its features, etc. -- Why do we need this integration? How is it going to benefit the Flyte users? -- Showcase the uniqueness of the integration -- How to install the plugin? - -Finally, **for tutorials and integrations only**, write a `auto-examples-toc` directive at the bottom of the file: - -````{code-block} -```{auto-examples-toc} -example_01 -example_02 -example_03 -``` -```` - -Where `example_01`, `example_02`, and `example_03` are the python module -names of the examples under the `new_example_project` directory. These can also -be the names of the `.ipynb` or `.md` files (but without the file extension). - -:::{tip} -Refer to any subdirectory in the `examples` directory -::: - -## Test your code - -If the example code can be run locally, just use `python .py` to run it. - -### Testing on a cluster - -Install {doc}`flytectl `, the commandline interface for flyte. - -:::{note} -Learn more about installation and configuration of Flytectl [here](https://docs.flyte.org/en/latest/flytectl/docs_index.html). -::: - -Start a Flyte demo cluster with: - -``` -flytectl demo start -``` - -### Testing the `basics` project examples on a local demo cluster - -In this example, we'll build the `basics` project: - -```{prompt} bash -# from flytesnacks root directory -cd examples/basics -``` - -Build the container: - -```{prompt} bash -docker build . --tag "basics:v1" -f Dockerfile -``` - -Package the examples by running: - -```{prompt} bash -pyflyte --pkgs basics package --image basics:v1 -f -``` - -Register the examples by running - -```{prompt} bash -flytectl register files \ - -p flytesnacks \ - -d development \ - --archive flyte-package.tgz \ - --version v1 -``` - -Visit `https://localhost:30081/console` to view the Flyte console, which consists -of the examples present in the `flytesnacks/core` directory. - -### Updating dependencies - -:::{admonition} Prerequisites -Install [pip-tools](https://pypi.org/project/pip-tools/) in your development -environment with: - -```{prompt} bash -pip install pip-tools -``` - -::: - -If you've updated the dependencies of the project, update the `requirements.txt` -file by running: - -```{prompt} bash -pip-compile requirements.in --upgrade --verbose --resolver=backtracking -``` - -### Rebuild the image - -If you've updated the source code or dependencies of the project, and rebuild -the image with: - -```{prompt} bash -docker build . --tag "basics:v2" -f core/Dockerfile -pyflyte --pkgs basics package --image basics:v2 -f -flytectl register files \ - -p flytesnacks \ - -d development \ - --archive flyte-package.tgz \ - --version v2 -``` - -Refer to {ref}`this guide ` -if the code in itself is updated and requirements.txt is the same. - -## Pre-commit hooks - -We use [pre-commit](https://pre-commit.com/) to automate linting and code formatting on every commit. -Configured hooks include [ruff](https://github.com/astral-sh/ruff) to ensure newlines are added to the end of files, and there is proper spacing in files. - -We run all those hooks in CI, but if you want to run them locally on every commit, run `pre-commit install` after -installing the dev environment requirements. In case you want to disable `pre-commit` hooks locally, run -`pre-commit uninstall`. More info [here](https://pre-commit.com/). - -### Formatting - -We use [ruff](https://github.com/astral-sh/ruff) to autoformat code. They -are configured as git hooks in `pre-commit`. Run `make fmt` to format your code. - -### Spell-checking - -We use [codespell](https://github.com/codespell-project/codespell) to catch common misspellings. Run -`make spellcheck` to spell-check the changes. - -## Update Documentation Pages - -The `docs/conf.py` contains the sphinx configuration for building the -`flytesnacks` documentation. - -At build-time, the `flytesnacks` sphinx build system will convert all of the -projects in the `examples` directory into `docs/auto_examples`, and will be -available in the documentation. - -::::{important} - -The docs build system will convert the `README.md` files in each example -project into a `index.md` file, so you can reference the root page of each -example project, e.g., in myst markdown format, you can write a table-of-content -directive like so: - -:::{code-block} - -```{toc} -auto_examples/basics/index -``` - -::: - -:::: - -If you've created a new example project, you'll need to add the `index` page -in the table of contents in `docs/index.md` to make sure the project -shows up in the documentation. Additonally, you'll need to update the appropriate -`list-table` directive in `docs/userguide.md`, `docs/tutorials.md`, or -`docs/integrations.md` so that it shows up in the respective section of the -documentation. - -## Build the documentation locally - -Verify that the code and documentation look as expected: - -- Learn about the documentation tools [here](https://docs.flyte.org/en/latest/community/contribute.html#documentation) -- Install the requirements by running `pip install -r docs-requirements.txt`. -- Run `make -C docs html` - - ```{tip} - To run a fresh build, run `make -C docs clean html`. - ``` - -- Open the HTML pages present in the `docs/_build` directory in the browser with - `open docs/_build/index.html` - -## Create a pull request - -Create the pull request, then ensure that the docs are rendered correctly by clicking on the documentation check. - -```{image} https://raw.githubusercontent.com/flyteorg/static-resources/main/common/test_docs_link.png -:alt: Docs link in a PR -``` - -You can refer to [this PR](https://github.com/flyteorg/flytesnacks/pull/332) for the exact changes required. diff --git a/docs/index.md b/docs/index.md index 027058e5c..335236f8f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -12,145 +12,14 @@ next-page-title: Quickstart guide (getting_started_index)= -# Introduction to Flyte +# Flytesnacks -Flyte is a workflow orchestrator that unifies machine learning, data engineering, and data analytics stacks for building robust and reliable applications. Flyte features: - -- Reproducible, repeatable workflows -- Strongly typed interfaces -- Structured datasets to enable easy conversion of dataframes between types, and column-level type checking -- Easy movement of data between local and cloud storage -- Easy tracking of data lineages -- Built-in data and artifact visualization - -For a full list of feature, see the [Flyte features page](https://flyte.org/features). - -## Basic Flyte components - -Flyte is made up of a user plane, control plane, and data plane. - -- The **user plane** contains the elements you need to develop the code that will implement your application's directed acyclic graph (DAG). These elements are FlyteKit and Flytectl. Data scientists and machine learning engineers primarily work in the user plane. -- The **control plane** is part of the Flyte backend that is configured by platform engineers or others tasked with setting up computing infrastructure. It consists of FlyteConsole and FlyteAdmin, which serves as the main Flyte API to process requests from clients in the user plane. The control plane sends workflow execution requests to the data plane for execution, and stores information such as current and past running workflows, and provides that information upon request. -- The **data plane** is another part of the Flyte backend that contains FlytePropeller, the core engine of Flyte that executes workflows. FlytePropeller is designed as a [Kubernetes Controller](https://kubernetes.io/docs/concepts/architecture/controller/). The data plane sends status events back to the control plane so that information can be stored and surfaced to end users. - -## Next steps - -- To quickly try out Flyte on your machine, follow the {ref}`Quickstart guide `. -- To create a Flyte project that can be used to package workflow code for deployment to a Flyte cluster, see {ref}`"Getting started with workflow development" `. -- To set up a Flyte cluster, see the [Deployment documentation](https://docs.flyte.org/en/latest/deployment/index.html). - -```{toctree} -:maxdepth: 1 -:hidden: - -Getting Started -User Guide -Tutorials -Concepts -Deployment -API Reference -Community -``` - -```{toctree} -:maxdepth: -1 -:caption: Getting Started -:hidden: - -Introduction to Flyte -Quickstart guide -Getting started with workflow development -Flyte fundamentals -Core use cases -``` - -```{toctree} -:maxdepth: -1 -:caption: User Guide -:hidden: - -📖 User Guide -🌳 Environment Setup -🔤 Basics -⌨️ Data Types and IO -🔮 Advanced Composition -🧩 Customizing Dependencies -🏡 Development Lifecycle -⚗️ Testing -🚢 Productionizing -🏗 Extending -📝 Contributing -``` - -```{toctree} -:maxdepth: -1 -:caption: Tutorials -:hidden: - -Tutorials -Model Training -feature_engineering -bioinformatics_examples -flyte_lab -``` - -```{toctree} -:maxdepth: -1 -:caption: Integrations -:hidden: - -Integrations -auto_examples/airflow_agent/index -auto_examples/airflow_plugin/index -auto_examples/athena_plugin/index -auto_examples/aws_batch_plugin/index -auto_examples/bigquery_agent/index -auto_examples/chatgpt_agent/index -auto_examples/k8s_dask_plugin/index -auto_examples/databricks_agent/index -auto_examples/dbt_plugin/index -auto_examples/dolt_plugin/index -auto_examples/duckdb_plugin/index -auto_examples/flyteinteractive_plugin/index -auto_examples/greatexpectations_plugin/index -auto_examples/hive_plugin/index -auto_examples/k8s_pod_plugin/index -auto_examples/mlflow_plugin/index -auto_examples/mmcloud_agent/index -auto_examples/modin_plugin/index -auto_examples/kfmpi_plugin/index -auto_examples/nim_plugin/index -auto_examples/ollama_plugin/index -auto_examples/onnx_plugin/index -auto_examples/openai_batch_agent/index -auto_examples/papermill_plugin/index -auto_examples/pandera_plugin/index -auto_examples/kfpytorch_plugin/index -auto_examples/ray_plugin/index -auto_examples/sagemaker_inference_agent/index -auto_examples/sensor/index -auto_examples/snowflake_agent/index -auto_examples/k8s_spark_plugin/index -auto_examples/sql_plugin/index -auto_examples/kftensorflow_plugin/index -auto_examples/whylogs_plugin/index -``` - -```{toctree} -:maxdepth: -1 -:caption: Deprecated integrations -:hidden: - -Deprecated integrations -BigQuery plugin -Databricks plugin -Snowflake plugin -``` +Welcome to Flytesnacks. ```{toctree} :maxdepth: -1 -:caption: Tags :hidden: -_tags/tagsindex +Tutorials +Integrations ``` diff --git a/docs/integrations/deprecated_integrations/index.md b/docs/integrations/deprecated_integrations/index.md new file mode 100644 index 000000000..ff5c12e2a --- /dev/null +++ b/docs/integrations/deprecated_integrations/index.md @@ -0,0 +1,10 @@ +# Deprecated integrations + +```{toctree} +:maxdepth: -1 +:hidden: + +BigQuery plugin +Databricks plugin +Snowflake plugin +``` diff --git a/docs/integrations.md b/docs/integrations/index.md similarity index 63% rename from docs/integrations.md rename to docs/integrations/index.md index b763017ad..b1cd315ac 100644 --- a/docs/integrations.md +++ b/docs/integrations/index.md @@ -5,7 +5,7 @@ Flyte is designed to be highly extensible and can be customized in multiple ways. ```{note} -Want to contribute an example? Check out the {doc}`Example Contribution Guide `. +Want to contribute an example? Check out the {ref}`Documentation contribution guide `. ``` ## Flytekit Plugins @@ -18,33 +18,33 @@ Flytekit functionality. These plugins can be anything and for comparison can be :header-rows: 0 :widths: 20 30 -* - {doc}`SQL ` +* - {doc}`SQL ` - Execute SQL queries as tasks. -* - {doc}`Great Expectations ` +* - {doc}`Great Expectations ` - Validate data with `great_expectations`. -* - {doc}`Papermill ` +* - {doc}`Papermill ` - Execute Jupyter Notebooks with `papermill`. -* - {doc}`Pandera ` +* - {doc}`Pandera ` - Validate pandas dataframes with `pandera`. -* - {doc}`Modin ` +* - {doc}`Modin ` - Scale pandas workflows with `modin`. -* - {doc}`Dolt ` +* - {doc}`Dolt ` - Version your SQL database with `dolt`. -* - {doc}`DBT ` +* - {doc}`DBT ` - Run and test your `dbt` pipelines in Flyte. -* - {doc}`WhyLogs ` +* - {doc}`WhyLogs ` - `whylogs`: the open standard for data logging. -* - {doc}`MLFlow ` +* - {doc}`MLFlow ` - `mlflow`: the open standard for model tracking. -* - {doc}`ONNX ` +* - {doc}`ONNX ` - Convert ML models to ONNX models seamlessly. -* - {doc}`DuckDB ` +* - {doc}`DuckDB ` - Run analytical queries using DuckDB. -* - {doc}`Weights and Biases ` +* - {doc}`Weights and Biases ` - `wandb`: Machine learning platform to build better models faster. -* - {doc}`NIM ` +* - {doc}`NIM ` - Serve optimized model containers with NIM. -* - {doc}`Ollama ` +* - {doc}`Ollama ` - Serve fine-tuned LLMs with Ollama in a Flyte workflow. ``` @@ -70,19 +70,19 @@ orchestrated by Flyte itself, within its provisioned Kubernetes clusters. :header-rows: 0 :widths: 20 30 -* - {doc}`K8s Pods ` +* - {doc}`K8s Pods ` - Execute K8s pods for arbitrary workloads. -* - {doc}`K8s Cluster Dask Jobs ` +* - {doc}`K8s Cluster Dask Jobs ` - Run Dask jobs on a K8s Cluster. -* - {doc}`K8s Cluster Spark Jobs ` +* - {doc}`K8s Cluster Spark Jobs ` - Run Spark jobs on a K8s Cluster. -* - {doc}`Kubeflow PyTorch ` +* - {doc}`Kubeflow PyTorch ` - Run distributed PyTorch training jobs using `Kubeflow`. -* - {doc}`Kubeflow TensorFlow ` +* - {doc}`Kubeflow TensorFlow ` - Run distributed TensorFlow training jobs using `Kubeflow`. -* - {doc}`MPI Operator ` +* - {doc}`MPI Operator ` - Run distributed deep learning training jobs using Horovod and MPI. -* - {doc}`Ray Task ` +* - {doc}`Ray Task ` - Run Ray jobs on a K8s Cluster. ``` @@ -96,23 +96,23 @@ orchestrated by Flyte itself, within its provisioned Kubernetes clusters. :header-rows: 0 :widths: 20 30 -* - {doc}`Airflow agent ` +* - {doc}`Airflow agent ` - Run Airflow jobs in your workflows with the Airflow agent. -* - {doc}`BigQuery agent ` +* - {doc}`BigQuery agent ` - Run BigQuery jobs in your workflows with the BigQuery agent. -* - {doc}`ChatGPT agent ` +* - {doc}`ChatGPT agent ` - Run ChatGPT jobs in your workflows with the ChatGPT agent. -* - {doc}`Databricks ` +* - {doc}`Databricks ` - Run Databricks jobs in your workflows with the Databricks agent. -* - {doc}`Memory Machine Cloud ` +* - {doc}`Memory Machine Cloud ` - Execute tasks using the MemVerge Memory Machine Cloud agent. -* - {doc}`OpenAI Batch ` +* - {doc}`OpenAI Batch ` - Submit requests for asynchronous batch processing on OpenAI. -* - {doc}`SageMaker Inference ` +* - {doc}`SageMaker Inference ` - Deploy models and create, as well as trigger inference endpoints on SageMaker. -* - {doc}`Sensor ` +* - {doc}`Sensor ` - Run sensor jobs in your workflows with the sensor agent. -* - {doc}`Snowflake ` +* - {doc}`Snowflake ` - Run Snowflake jobs in your workflows with the Snowflake agent. ``` @@ -127,13 +127,13 @@ As the term suggests, external service backend plugins rely on external services :header-rows: 0 :widths: 20 30 -* - {doc}`AWS Athena plugin ` +* - {doc}`AWS Athena plugin ` - Execute queries using AWS Athena -* - {doc}`AWS Batch plugin ` +* - {doc}`AWS Batch plugin ` - Running tasks and workflows on AWS batch service -* - {doc}`Flyte Interactive ` +* - {doc}`Flyte Interactive ` - Execute tasks using Flyte Interactive to debug. -* - {doc}`Hive plugin ` +* - {doc}`Hive plugin ` - Run Hive jobs in your workflows. ``` @@ -187,6 +187,48 @@ constructs natively within other orchestration tools. :header-rows: 0 :widths: 20 30 -* - {doc}`Airflow ` +* - {doc}`Airflow ` - Trigger Flyte executions from Airflow. ``` + +```{toctree} +:maxdepth: -1 +:hidden: + +/auto_examples/airflow_agent/index +/auto_examples/airflow_plugin/index +/auto_examples/athena_plugin/index +/auto_examples/aws_batch_plugin/index +/auto_examples/bigquery_agent/index +/auto_examples/chatgpt_agent/index +/auto_examples/k8s_dask_plugin/index +/auto_examples/databricks_agent/index +/auto_examples/dbt_plugin/index +/auto_examples/dolt_plugin/index +/auto_examples/duckdb_plugin/index +/auto_examples/flyteinteractive_plugin/index +/auto_examples/greatexpectations_plugin/index +/auto_examples/hive_plugin/index +/auto_examples/k8s_pod_plugin/index +/auto_examples/mlflow_plugin/index +/auto_examples/mmcloud_agent/index +/auto_examples/modin_plugin/index +/auto_examples/kfmpi_plugin/index +/auto_examples/nim_plugin/index +/auto_examples/ollama_plugin/index +/auto_examples/onnx_plugin/index +/auto_examples/openai_batch_agent/index +/auto_examples/papermill_plugin/index +/auto_examples/pandera_plugin/index +/auto_examples/kfpytorch_plugin/index +/auto_examples/ray_plugin/index +/auto_examples/sagemaker_inference_agent/index +/auto_examples/sensor/index +/auto_examples/snowflake_agent/index +/auto_examples/k8s_spark_plugin/index +/auto_examples/sql_plugin/index +/auto_examples/kftensorflow_plugin/index +/auto_examples/wandb_plugin/index +/auto_examples/whylogs_plugin/index +Deprecated integrations +``` diff --git a/docs/bioinformatics_examples.md b/docs/tutorials/bioinformatics/index.md similarity index 81% rename from docs/bioinformatics_examples.md rename to docs/tutorials/bioinformatics/index.md index 0e3bd5508..9207682c4 100644 --- a/docs/bioinformatics_examples.md +++ b/docs/tutorials/bioinformatics/index.md @@ -10,7 +10,7 @@ Flyte very much supports running your bioinformatics applications. Dive deeper i :header-rows: 0 :widths: 20 30 -* - {doc}`Nucleotide Sequence Querying with BLASTX ` +* - {doc}`Nucleotide Sequence Querying with BLASTX ` - Use BLASTX to Query a Nucleotide Sequence Against a Local Protein Database ``` @@ -19,5 +19,5 @@ Flyte very much supports running your bioinformatics applications. Dive deeper i :caption: Contents :hidden: -auto_examples/blast/index +/auto_examples/blast/index ``` diff --git a/docs/feature_engineering.md b/docs/tutorials/feature_engineering/index.md similarity index 62% rename from docs/feature_engineering.md rename to docs/tutorials/feature_engineering/index.md index b34a73891..99ac9f39b 100644 --- a/docs/feature_engineering.md +++ b/docs/tutorials/feature_engineering/index.md @@ -1,4 +1,4 @@ -# Feature Engineering +# Feature engineering **Feature Engineering** is an essential part of Machine Learning. Itss the process of transforming raw data into features that better represent the underlying problem @@ -10,9 +10,9 @@ Explore how features can be engineered with the power of Flyte. :header-rows: 0 :widths: 20 30 -* - {doc}`EDA and Feature Engineering With Papermill ` +* - {doc}`EDA and Feature Engineering With Papermill ` - How to use Jupyter notebook within Flyte -* - {doc}`Data Cleaning and Feature Serving With Feast ` +* - {doc}`Data Cleaning and Feature Serving With Feast ` - How to use Feast to serve data in Flyte ``` @@ -22,6 +22,6 @@ Explore how features can be engineered with the power of Flyte. :caption: Contents :hidden: -auto_examples/exploratory_data_analysis/index -auto_examples/feast_integration/index +/auto_examples/exploratory_data_analysis/index +/auto_examples/feast_integration/index ``` diff --git a/docs/flyte_lab.md b/docs/tutorials/flytelab/index.md similarity index 100% rename from docs/flyte_lab.md rename to docs/tutorials/flytelab/index.md diff --git a/docs/weather_forecasting.md b/docs/tutorials/flytelab/weather_forecasting.md similarity index 100% rename from docs/weather_forecasting.md rename to docs/tutorials/flytelab/weather_forecasting.md diff --git a/docs/tutorials.md b/docs/tutorials/index.md similarity index 62% rename from docs/tutorials.md rename to docs/tutorials/index.md index 559a71554..38aaa5b91 100644 --- a/docs/tutorials.md +++ b/docs/tutorials/index.md @@ -1,8 +1,3 @@ ---- -next-page: ml_training -next-page-title: Model Training ---- - (tutorials)= # Tutorials @@ -17,7 +12,7 @@ contributing samples easy. If this is your first time running these examples, fo {ref}`setup guide ` to get started. ```{note} -Want to contribute an example? Check out the {doc}`Example Contribution Guide `. +Want to contribute an example? Check out the [Documentation contribution guide](https://docs.flyte.org/en/latest/flytesnacks/contribute.html). ``` ## 🤖 Model Training @@ -28,15 +23,15 @@ Train machine learning models from using your framework of choice. :header-rows: 0 :widths: 20 30 -* - {doc}`Diabetes Classification ` +* - {doc}`Diabetes Classification ` - Train an XGBoost model on the Pima Indians Diabetes Dataset. -* - {doc}`House Price Regression ` +* - {doc}`House Price Regression ` - Use dynamic workflows to train a multiregion house price prediction model using XGBoost. -* - {doc}`MNIST Classification ` +* - {doc}`MNIST Classification ` - Train a neural network on MNIST with PyTorch and W&B -* - {doc}`NLP Processing with Gensim ` +* - {doc}`NLP Processing with Gensim ` - Word embedding and topic modelling on lee background corpus with Gensim -* - {doc}`Sales Forecasting ` +* - {doc}`Sales Forecasting ` - Use the Rossmann Store data to forecast sales with distributed training using Horovod on Spark. ``` @@ -48,9 +43,9 @@ Engineer the data features to improve your model accuracy. :header-rows: 0 :widths: 20 30 -* - {doc}`EDA and Feature Engineering With Papermill ` +* - {doc}`EDA and Feature Engineering With Papermill ` - How to use Jupyter notebook within Flyte -* - {doc}`Data Cleaning and Feature Serving With Feast ` +* - {doc}`Data Cleaning and Feature Serving With Feast ` - How to use Feast to serve data in Flyte ``` @@ -62,7 +57,7 @@ Perform computational biology with Flyte. :header-rows: 0 :widths: 20 30 -* - {doc}`Nucleotide Sequence Querying with BLASTX ` +* - {doc}`Nucleotide Sequence Querying with BLASTX ` - Use BLASTX to Query a Nucleotide Sequence Against a Local Protein Database ``` @@ -74,6 +69,16 @@ The open-source repository of machine learning projects using Flyte. :header-rows: 0 :widths: 20 30 -* - {doc}`Weather Forecasting ` +* - {doc}`Weather Forecasting ` - Build an online weather forecasting application. ``` + +```{toctree} +:maxdepth: -1 +:hidden: + +Model Training +Feature engineering +Bioinformatics +Flytelab +``` diff --git a/docs/ml_training.md b/docs/tutorials/model_training/index.md similarity index 50% rename from docs/ml_training.md rename to docs/tutorials/model_training/index.md index cd909b2ec..f46ad18c3 100644 --- a/docs/ml_training.md +++ b/docs/tutorials/model_training/index.md @@ -1,4 +1,4 @@ -# Model Training +# Model training Understand how machine learning models can be trained from within Flyte, with an added advantage of orchestration benefits. @@ -6,15 +6,15 @@ Understand how machine learning models can be trained from within Flyte, with an :header-rows: 0 :widths: 20 30 -* - {doc}`Diabetes Classification ` +* - {doc}`Diabetes Classification ` - Train an XGBoost model on the Pima Indians Diabetes Dataset. -* - {doc}`House Price Regression ` +* - {doc}`House Price Regression ` - Use dynamic workflows to train a multiregion house price prediction model using XGBoost. -* - {doc}`MNIST Classification ` +* - {doc}`MNIST Classification ` - Train a neural network on MNIST with PyTorch and W&B -* - {doc}`NLP Processing with Gensim ` +* - {doc}`NLP Processing with Gensim ` - Word embedding and topic modelling on lee background corpus with Gensim -* - {doc}`Forecast Sales Using Rossmann Store Sales ` +* - {doc}`Forecast Sales Using Rossmann Store Sales ` - Forecast sales data with data-parallel distributed training using Horovod on Spark. ``` @@ -23,9 +23,9 @@ Understand how machine learning models can be trained from within Flyte, with an :caption: Contents :hidden: -auto_examples/pima_diabetes/index -auto_examples/house_price_prediction/index -auto_examples/mnist_classifier/index -auto_examples/nlp_processing/index -auto_examples/forecasting_sales/index +/auto_examples/pima_diabetes/index +/auto_examples/house_price_prediction/index +/auto_examples/mnist_classifier/index +/auto_examples/nlp_processing/index +/auto_examples/forecasting_sales/index ``` diff --git a/examples/airflow_plugin/airflow_plugin/airflow.py b/examples/airflow_plugin/airflow_plugin/airflow.py index 35a296261..d47c75277 100644 --- a/examples/airflow_plugin/airflow_plugin/airflow.py +++ b/examples/airflow_plugin/airflow_plugin/airflow.py @@ -66,7 +66,7 @@ # Click `Save` in the end. # # :::{note} -# Use external IP as the Flyte `Host`. You can {std:ref}`deploy ` Flyte on an on-prem machine or on cloud. +# Use external IP as the Flyte `Host`. You can {ref}`deploy ` Flyte on an on-prem machine or on cloud. # ::: # # ## Register Flyte Code diff --git a/examples/bigquery_agent/requirements.in b/examples/bigquery_agent/requirements.in index a987746f1..e496aea34 100644 --- a/examples/bigquery_agent/requirements.in +++ b/examples/bigquery_agent/requirements.in @@ -1,5 +1,6 @@ flytekit wheel matplotlib +pandas flytekitplugins-deck-standard flytekitplugins-bigquery diff --git a/examples/bigquery_plugin/README.md b/examples/bigquery_plugin/README.md index 5027330e5..46e379751 100644 --- a/examples/bigquery_plugin/README.md +++ b/examples/bigquery_plugin/README.md @@ -1,5 +1,14 @@ -# BigQuery plugin example +# BigQuery plugin -**Note:** This example code uses the legacy implementation of the BigQuery integration. We recommend using the [BigQuery agent](https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_agent/index.html) instead. +```{warning} +This example code uses the legacy implementation of the BigQuery integration. We recommend using the [BigQuery agent](https://docs.flyte.org/en/latest/flytesnacks/examples/bigquery_agent/index.html) instead. +``` This directory contains example code for the deprecated BigQuery plugin. For documentation on installing and using the plugin, see the [BigQuery plugin documentation](https://docs.flyte.org/en/latest/deprecated_integrations/bigquery_plugin/index.html) + +```{toctree} +:maxdepth: -1 +:hidden: + +bigquery_plugin_example +``` diff --git a/examples/bigquery_plugin/bigquery_plugin/bigquery_plugin_example.py b/examples/bigquery_plugin/bigquery_plugin/bigquery_plugin_example.py index 000882653..3aa740edc 100644 --- a/examples/bigquery_plugin/bigquery_plugin/bigquery_plugin_example.py +++ b/examples/bigquery_plugin/bigquery_plugin/bigquery_plugin_example.py @@ -1,3 +1,9 @@ +# %% [markdown] +# (bigquery_plugin_example)= +# # BigQuery plugin example +# +# %% + try: from typing import Annotated except ImportError: diff --git a/examples/bigquery_plugin/requirements.in b/examples/bigquery_plugin/requirements.in index a987746f1..e496aea34 100644 --- a/examples/bigquery_plugin/requirements.in +++ b/examples/bigquery_plugin/requirements.in @@ -1,5 +1,6 @@ flytekit wheel matplotlib +pandas flytekitplugins-deck-standard flytekitplugins-bigquery diff --git a/examples/blast/requirements.in b/examples/blast/requirements.in index bec48ef8c..30f834337 100644 --- a/examples/blast/requirements.in +++ b/examples/blast/requirements.in @@ -1,4 +1,5 @@ flytekit>=0.32.3 wheel matplotlib +pandas flytekitplugins-deck-standard diff --git a/examples/databricks_plugin/README.md b/examples/databricks_plugin/README.md index ca7b3a2b6..636eb7f94 100644 --- a/examples/databricks_plugin/README.md +++ b/examples/databricks_plugin/README.md @@ -1,5 +1,14 @@ -# Databricks plugin example +# Databricks plugin -**Note:** This example code uses a legacy implementation of the Databricks integration. We recommend using the [Databricks agent](https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_agent/index.html) instead. +```{warning} +This example code uses a legacy implementation of the Databricks integration. We recommend using the [Databricks agent](https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_agent/index.html) instead. +``` This directory contains example code for the deprecated Databricks plugin. For documentation on installing and using the plugin, see the [Databricks plugin documentation](https://docs.flyte.org/en/latest/deprecated_integrations/databricks_plugin/index.html) + +```{toctree} +:maxdepth: -1 +:hidden: + +databricks_plugin_example +``` diff --git a/examples/databricks_plugin/databricks_plugin/databricks_plugin_example.py b/examples/databricks_plugin/databricks_plugin/databricks_plugin_example.py index 3137ca1b5..72f98c5ad 100644 --- a/examples/databricks_plugin/databricks_plugin/databricks_plugin_example.py +++ b/examples/databricks_plugin/databricks_plugin/databricks_plugin_example.py @@ -1,3 +1,9 @@ +# %% [markdown] +# (spark_on_databricks_plugin)= +# # Databricks plugin example +# +# %% + import datetime import random from operator import add diff --git a/examples/development_lifecycle/requirements.in b/examples/development_lifecycle/requirements.in index 8e50db9c6..89af2129e 100644 --- a/examples/development_lifecycle/requirements.in +++ b/examples/development_lifecycle/requirements.in @@ -3,4 +3,5 @@ flytekitplugins-deck-standard plotly scikit-learn tabulate +pandas pyarrow diff --git a/examples/house_price_prediction/README.md b/examples/house_price_prediction/README.md index 49f5c55c5..70341e6c2 100644 --- a/examples/house_price_prediction/README.md +++ b/examples/house_price_prediction/README.md @@ -12,7 +12,7 @@ In this example, we will train our data on the XGBoost model to predict house pr ## Where Does Flyte Fit In? - Orchestrates the machine learning pipeline. -- Helps cache the output state between {py:func}`tasks `. +- Helps cache the output state between {py:func}`tasks `. - Easier backtracking to the error source. - Provides a Rich UI to view and manage the pipeline. diff --git a/examples/house_price_prediction/requirements.in b/examples/house_price_prediction/requirements.in index b571e33d3..f09c8717b 100644 --- a/examples/house_price_prediction/requirements.in +++ b/examples/house_price_prediction/requirements.in @@ -7,3 +7,4 @@ joblib scikit-learn tabulate matplotlib +pandas diff --git a/examples/k8s_dask_plugin/README.md b/examples/k8s_dask_plugin/README.md index 713e810c2..c5f6a02ed 100644 --- a/examples/k8s_dask_plugin/README.md +++ b/examples/k8s_dask_plugin/README.md @@ -71,7 +71,7 @@ Flyte Dask utilizes the [Dask Kubernetes operator](https://kubernetes.dask.org/e in conjunction with a custom-built [Flyte Dask plugin](https://pkg.go.dev/github.com/flyteorg/flyteplugins@v1.0.28/go/tasks/plugins/k8s/dask). To leverage this functionality, you need to enable the backend plugin in your deployment. -You can follow the steps mentioned in the {ref}`flyte:deployment-plugin-setup-k8s` section +You can follow the steps mentioned in the {ref}`deployment-plugin-setup-k8s` section to enable the Flyte Dask plugin for your deployment. #### Step 2: Compute setup diff --git a/examples/k8s_pod_plugin/README.md b/examples/k8s_pod_plugin/README.md index e7b0e8dc8..4a1646be5 100644 --- a/examples/k8s_pod_plugin/README.md +++ b/examples/k8s_pod_plugin/README.md @@ -4,7 +4,7 @@ .. tags:: Integration, Kubernetes, Advanced ``` -Flyte tasks, represented by the {py:func}`@task ` decorator, are essentially single functions that run in one container. +Flyte tasks, represented by the {py:func}`@task ` decorator, are essentially single functions that run in one container. However, there may be situations where you need to run a job with more than one container or require additional capabilities, such as: - Running a hyper-parameter optimizer that stores state in a Redis database diff --git a/examples/k8s_spark_plugin/README.md b/examples/k8s_spark_plugin/README.md index 3bdbf721c..ad7a39cbe 100644 --- a/examples/k8s_spark_plugin/README.md +++ b/examples/k8s_spark_plugin/README.md @@ -27,7 +27,7 @@ However, please bear in mind that this functionality requires a significant Kube For optimal results, we highly recommend adopting the [multi-cluster mode](https://docs.flyte.org/en/latest/deployment/configuration/performance.html#multi-cluster-mode). -Additionally, consider enabling {std:ref}`resource quotas ` +Additionally, consider enabling {ref}`resource quotas ` for Spark Jobs that are both large in scale and executed frequently. Nonetheless, it is important to note that extremely short-duration jobs might not be the best fit for this setup. @@ -63,7 +63,7 @@ Flyte Spark employs the Spark on K8s operator in conjunction with a bespoke [Flyte Spark Plugin](https://pkg.go.dev/github.com/flyteorg/flyteplugins@v0.5.25/go/tasks/plugins/k8s/spark). This plugin serves as a backend component and necessitates activation within your deployment. -To enable it, follow the instructions outlined in the {ref}`flyte:deployment-plugin-setup-k8s` section. +To enable it, follow the instructions outlined in the {ref}`deployment-plugin-setup-k8s` section. :::{note} Refer to [this guide](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/gcp.md) to use GCP instead of AWS. diff --git a/examples/kfpytorch_plugin/README.md b/examples/kfpytorch_plugin/README.md index 95c67245e..e4f631780 100644 --- a/examples/kfpytorch_plugin/README.md +++ b/examples/kfpytorch_plugin/README.md @@ -17,7 +17,7 @@ To use the PyTorch plugin, run the following command: pip install flytekitplugins-kfpytorch ``` -To enable the plugin in the backend, follow instructions outlined in the {std:ref}`flyte:deployment-plugin-setup-k8s` guide. +To enable the plugin in the backend, follow instructions outlined in the {ref}`deployment-plugin-setup-k8s` guide. ## Run the example on the Flyte cluster diff --git a/examples/kftensorflow_plugin/README.md b/examples/kftensorflow_plugin/README.md index 636d0514d..d97b4b477 100644 --- a/examples/kftensorflow_plugin/README.md +++ b/examples/kftensorflow_plugin/README.md @@ -17,7 +17,7 @@ To install the Kubeflow TensorFlow plugin, run the following command: pip install flytekitplugins-kftensorflow ``` -To enable the plugin in the backend, follow instructions outlined in the {std:ref}`flyte:deployment-plugin-setup-k8s` guide. +To enable the plugin in the backend, follow instructions outlined in the {ref}`deployment-plugin-setup-k8s` guide. ## Run the example on the Flyte cluster diff --git a/examples/mmcloud_agent/mmcloud_agent/mmcloud_agent_example_usage.py b/examples/mmcloud_agent/mmcloud_agent/mmcloud_agent_example_usage.py index d045bbb80..eac12209d 100644 --- a/examples/mmcloud_agent/mmcloud_agent/mmcloud_agent_example_usage.py +++ b/examples/mmcloud_agent/mmcloud_agent/mmcloud_agent_example_usage.py @@ -24,7 +24,7 @@ def to_int(s: str) -> int: # %% [markdown] -# [Resource](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/customizing_resources.html) (cpu and mem) requests and limits, [container](https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/customizing_dependencies/multi_images.html) images, and [environment](https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html) variable specifications are supported. +# [Resource](https://docs.flyte.org/en/latest/user_guide/productionizing/customizing_task_resources.html) (cpu and mem) requests and limits, [container](https://docs.flyte.org/en/latest/user_guide/customizing_dependencies/index.html#customizing-dependencies) images, and [environment](https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html) variable specifications are supported. # %% diff --git a/examples/pima_diabetes/requirements.in b/examples/pima_diabetes/requirements.in index b571e33d3..f09c8717b 100644 --- a/examples/pima_diabetes/requirements.in +++ b/examples/pima_diabetes/requirements.in @@ -7,3 +7,4 @@ joblib scikit-learn tabulate matplotlib +pandas diff --git a/examples/ray_plugin/README.md b/examples/ray_plugin/README.md index 3ad2a65a8..575c97b50 100644 --- a/examples/ray_plugin/README.md +++ b/examples/ray_plugin/README.md @@ -25,7 +25,7 @@ To install the Ray plugin, run the following command: pip install flytekitplugins-ray ``` -To enable the plugin in the backend, refer to the instructions provided in the {std:ref}`flyte:deployment-plugin-setup-k8s` guide. +To enable the plugin in the backend, refer to the instructions provided in the {ref}`deployment-plugin-setup-k8s` guide. ## Implementation details diff --git a/examples/snowflake_plugin/README.md b/examples/snowflake_plugin/README.md index 6921e44ae..43a161c5c 100644 --- a/examples/snowflake_plugin/README.md +++ b/examples/snowflake_plugin/README.md @@ -1,3 +1,14 @@ -**Note:** This example code uses a legacy implementation of the Snowflake integration. We recommend using the [Snowflake agent](https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_agent/index.html) instead. +# Snowflake plugin + +```{warning} +This example code uses a legacy implementation of the Snowflake integration. We recommend using the [Snowflake agent](https://docs.flyte.org/en/latest/flytesnacks/examples/databricks_agent/index.html) instead. +``` This directory contains example code for the deprecated Snowflake plugin. For documentation on installing and using the plugin, see the [Snowflake plugin documentation](https://docs.flyte.org/en/latest/deprecated_integrations/snowflake_plugin/index.html) + +```{toctree} +:maxdepth: -1 +:hidden: + +snowflake_plugin_example +``` diff --git a/examples/snowflake_plugin/snowflake_plugin.md b/examples/snowflake_plugin/snowflake_plugin.md index 34f622387..e4313fb11 100644 --- a/examples/snowflake_plugin/snowflake_plugin.md +++ b/examples/snowflake_plugin/snowflake_plugin.md @@ -20,7 +20,7 @@ pip install flytekitplugins-snowflake If you intend to run the plugin on the Flyte cluster, you must first set it up on the backend. Please refer to the -{std:ref}`Snowflake plugin setup guide ` +{ref}`Snowflake plugin setup guide ` for detailed instructions. ## Run the example on the Flyte cluster diff --git a/examples/snowflake_plugin/snowflake_plugin/snowflake_plugin_example.py b/examples/snowflake_plugin/snowflake_plugin/snowflake_plugin_example.py index 7ee3b2858..a9903a900 100644 --- a/examples/snowflake_plugin/snowflake_plugin/snowflake_plugin_example.py +++ b/examples/snowflake_plugin/snowflake_plugin/snowflake_plugin_example.py @@ -1,3 +1,8 @@ +# %% [markdown] +# # Snowflake plugin example +# +# %% + from flytekit import kwtypes, workflow from flytekitplugins.snowflake import SnowflakeConfig, SnowflakeTask diff --git a/examples/whylogs_plugin/requirements.in b/examples/whylogs_plugin/requirements.in index 3f8bc07aa..3ecac63ec 100644 --- a/examples/whylogs_plugin/requirements.in +++ b/examples/whylogs_plugin/requirements.in @@ -7,3 +7,4 @@ scikit-learn whylogs[s3] whylogs[mlflow] whylogs[whylabs] +pandas