Add Python delta source and destination (#407)

* add python delta source and destination Signed-off-by: JamesKnBr <[email protected]> * Updates for destination Signed-off-by: rodalynbarce <[email protected]> * add delta source and destination Signed-off-by: JamesKnBr <[email protected]> * update delta destination Signed-off-by: JamesKnBr <[email protected]> * added tests to delta source Signed-off-by: JamesKnBr <[email protected]> * Added delta destination test Signed-off-by: rodalynbarce <[email protected]> * Added init Signed-off-by: rodalynbarce <[email protected]> * Doc updates Signed-off-by: rodalynbarce <[email protected]> * tidy delta source and destination Signed-off-by: JamesKnBr <[email protected]> * doc updates and paramater fix (#398) Signed-off-by: cching95 <[email protected]> * fix meta data tag name error and update unit test (#401) Signed-off-by: cching95 <[email protected]> * Feature/00320 - Weather Transformer (#387) * remove print statement in twa Signed-off-by: cching95 <[email protected]> * v0.2.0 (#121) * Add Releases Feed to Docs Release page (#69) * fixes Add Releases Feed to Docs page #68 Signed-off-by: GBBBAS <[email protected]> * Mkdocs test fixes Signed-off-by: GBBBAS <[email protected]> * Add unit tests Signed-off-by: GBBBAS <[email protected]> * Fix Code Smell Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Update Packages and Doc Logos (#76) * Upgrade Packages Signed-off-by: GBBBAS <[email protected]> * Upgrade Functions Packages Signed-off-by: GBBBAS <[email protected]> * Add logos Signed-off-by: GBBBAS <[email protected]> * Add logo to pypi readme Signed-off-by: GBBBAS <[email protected]> * Remove pydantic models Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Remove strawberry type Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Upgrade GraphQL Package Signed-off-by: GBBBAS <[email protected]> * Logo Size Changes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Fix Logo Alignment (#78) * Upgrade Packages Signed-off-by: GBBBAS <[email protected]> * Upgrade Functions Packages Signed-off-by: GBBBAS <[email protected]> * Add logos Signed-off-by: GBBBAS <[email protected]> * Add logo to pypi readme Signed-off-by: GBBBAS <[email protected]> * Remove pydantic models Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Remove strawberry type Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Upgrade GraphQL Package Signed-off-by: GBBBAS <[email protected]> * Logo Size Changes Signed-off-by: GBBBAS <[email protected]> * Logo fixes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * fixes #77 (#79) Signed-off-by: GBBBAS <[email protected]> * add spark session (#80) * set up spark sessions Signed-off-by: cching95 <[email protected]> * adding unit tests for spark session Signed-off-by: cching95 <[email protected]> * remove tests for spark session Signed-off-by: cching95 <[email protected]> * fix code smells Signed-off-by: cching95 <[email protected]> * add constants in pipeline utils Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Add Pipelines as Extras libraries (#82) Signed-off-by: GBBBAS <[email protected]> * Spark pipeline source Event hub and destination Event hub (#83) * add source eventhub to pipelines Signed-off-by: cching95 <[email protected]> * updates for documentation Signed-off-by: cching95 <[email protected]> * updates to mkdocs Signed-off-by: cching95 <[email protected]> * added unit tests Signed-off-by: cching95 <[email protected]> * documentation for eventhub source Signed-off-by: cching95 <[email protected]> * fix spelling error Signed-off-by: cching95 <[email protected]> * add source eventhub destination Signed-off-by: cching95 <[email protected]> * added destination source documentation Signed-off-by: cching95 <[email protected]> * remove unit tests to revisit later Signed-off-by: cching95 <[email protected]> * fix mkdoc errors Signed-off-by: cching95 <[email protected]> * fix code smells Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Spark pipeline source delta and destination delta (#84) * Spark pipeline source delta and destination delta Signed-off-by: rodalynbarce <[email protected]> * fix mkdocs errors Signed-off-by: rodalynbarce <[email protected]> * docstring amendments Signed-off-by: rodalynbarce <[email protected]> * Added to docstrings on delta source & destination Signed-off-by: JamesKnBr <[email protected]> * added query type for eventhubs Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: cching95 <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: cching95 <[email protected]> * Added delta sharing with docs (#85) Signed-off-by: JamesKnBr <[email protected]> * Unit Tests for sources (#88) * add pipeline utils unit tests Signed-off-by: cching95 <[email protected]> * add unit test for eventhub and delta Signed-off-by: cching95 <[email protected]> * Unit Test Fixes Signed-off-by: GBBBAS <[email protected]> * remove test spark Signed-off-by: cching95 <[email protected]> * add unit test for destination delta Signed-off-by: cching95 <[email protected]> * fix eventhub spell errors Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> Signed-off-by: GBBBAS <[email protected]> Co-authored-by: GBBBAS <[email protected]> * Add pipeline eventhub body transformer (#89) * add transform eventhub Signed-off-by: cching95 <[email protected]> * transformer docs set up Signed-off-by: cching95 <[email protected]> * documentation for transformer eventhub body Signed-off-by: cching95 <[email protected]> * add transformer eventhub unit tests Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Added pipeline source autoloader and unit test (#93) Signed-off-by: rodalynbarce <[email protected]> * Pipeline Executor (#95) * Dependency Injector Signed-off-by: GBBBAS <[email protected]> * Add Pipeline Executor Signed-off-by: GBBBAS <[email protected]> * Add Utilities Interface Signed-off-by: GBBBAS <[email protected]> * Add Pipeline Executor Signed-off-by: GBBBAS <[email protected]> * Add Pipeline Execute Test Signed-off-by: GBBBAS <[email protected]> * fixes #90 Signed-off-by: GBBBAS <[email protected]> * Fixes for tests Signed-off-by: GBBBAS <[email protected]> * Step Order fix Signed-off-by: GBBBAS <[email protected]> * Sonarqube fixes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Delta Table Create Utility (#97) * Add to docs site Signed-off-by: GBBBAS <[email protected]> * Update Delta Table Name Signed-off-by: GBBBAS <[email protected]> * Add Delta Table Create Signed-off-by: GBBBAS <[email protected]> * Add Delta Table Create Test Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * remove print statement in twa (#99) Signed-off-by: cching95 <[email protected]> Co-authored-by: GBBBAS <[email protected]> Co-authored-by: Amber-Rigg <[email protected]> Co-authored-by: NooraKubati <[email protected]> * Pipeline Deployment (#102) * Add DBX Pipeline Deployment Signed-off-by: GBBBAS <[email protected]> * Updates for Pipeline Deployment Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Pipeline Deployment Updates (#103) * Pipeline Deployment Updates Signed-off-by: GBBBAS <[email protected]> * Update to run logic of Pipeline Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Updates for Pipeline Deployment (#104) Signed-off-by: GBBBAS <[email protected]> * Unit test for spark pipeline eventhub destination (#105) * eventhub destination unit test Signed-off-by: cching95 <[email protected]> * add unit test for spark eventhub destination Signed-off-by: cching95 <[email protected]> * add spark eventhub write stream unit test Signed-off-by: cching95 <[email protected]> * update exception handling for eventhub and delta Signed-off-by: cching95 <[email protected]> * update exception in utilities Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Fix for Dependency Injection (#106) Signed-off-by: GBBBAS <[email protected]> * Unit tests for source and destination streams (#108) * streaming unit test for source eventhub Signed-off-by: cching95 <[email protected]> * add delta read stream unit test Signed-off-by: cching95 <[email protected]> * add unit test for delta destination streaming Signed-off-by: cching95 <[email protected]> * remove extra test Signed-off-by: cching95 <[email protected]> * remove code smell Signed-off-by: cching95 <[email protected]> * add previous line of code back Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Pipeline Deployments and Pipeline Secrets (#111) * Fix for Dependency Injection Signed-off-by: GBBBAS <[email protected]> * Pipeline Deployment And Secrets Signed-off-by: GBBBAS <[email protected]> * Add System Type Docs Signed-off-by: GBBBAS <[email protected]> * Updates for Secrets Signed-off-by: GBBBAS <[email protected]> * Code Smell fix Signed-off-by: GBBBAS <[email protected]> * Refactor Pipeline Secret Conversion Logic Signed-off-by: GBBBAS <[email protected]> * import required objects Signed-off-by: GBBBAS <[email protected]> * Code smell fix Signed-off-by: GBBBAS <[email protected]> * Code Smell Fix Signed-off-by: GBBBAS <[email protected]> * Fix Code Smells Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Added fails tests to pipeline sources (#112) * Added fails tests to pipeline sources Signed-off-by: JamesKnBr <[email protected]> * Added fails tests for pipeline destinations Signed-off-by: JamesKnBr <[email protected]> * Fixed code smell Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: JamesKnBr <[email protected]> * Fix for dockerbuild error (#114) Signed-off-by: GBBBAS <[email protected]> * Fix for docker build (#115) Signed-off-by: GBBBAS <[email protected]> * Cleanup Dockerfile (#116) Signed-off-by: GBBBAS <[email protected]> * Pipeline Secrets, Converters and Deployment Updates (#119) * Updates for Pipelines Signed-off-by: GBBBAS <[email protected]> * Update DBX Documentation Signed-off-by: GBBBAS <[email protected]> * Updates for Pipeline Deployment Signed-off-by: GBBBAS <[email protected]> * Remove package.json Signed-off-by: GBBBAS <[email protected]> * Fix code smells Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * time weighted average bug fix (#120) * fix twa group by dataframe return Signed-off-by: cching95 <[email protected]> * bug fixes for groupby and apply method Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> Signed-off-by: cching95 <[email protected]> Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: GBBBAS <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: Amber-Rigg <[email protected]> Co-authored-by: NooraKubati <[email protected]> * v0.4.0 (#304) * EdgeX Transformer (#271) * added transformer component Signed-off-by: rodalynbarce <[email protected]> * added edgeX test Signed-off-by: JamesKnBr <[email protected]> * update test Signed-off-by: JamesKnBr <[email protected]> * test updates Signed-off-by: rodalynbarce <[email protected]> * added kafka destination write format Signed-off-by: JamesKnBr <[email protected]> * component update Signed-off-by: rodalynbarce <[email protected]> * removed timestamp format Signed-off-by: JamesKnBr <[email protected]> * updated docs Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> * Feature/00264 (#272) * EdgeX Transformer (#271) * added transformer component Signed-off-by: rodalynbarce <[email protected]> * added edgeX test Signed-off-by: JamesKnBr <[email protected]> * update test Signed-off-by: JamesKnBr <[email protected]> * test updates Signed-off-by: rodalynbarce <[email protected]> * added kafka destination write format Signed-off-by: JamesKnBr <[email protected]> * component update Signed-off-by: rodalynbarce <[email protected]> * removed timestamp format Signed-off-by: JamesKnBr <[email protected]> * updated docs Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> * Adding support for MISO Sources Signed-off-by: Shivam Saxena <[email protected]> * Fixing sonar smells Signed-off-by: Shivam Saxena <[email protected]> * Fixing sonar smells Signed-off-by: Shivam Saxena <[email protected]> * Removing ISO Runner file Signed-off-by: Shivam Saxena <[email protected]> * Making ISO methods internal Signed-off-by: Shivam Saxena <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> * Add Partition Pruning to non broadcast join PCDM Merge (#274) * Add Partition Pruning on non broadcast join merge Signed-off-by: GBBBAS <[email protected]> * Update DropDuplicates Signed-off-by: GBBBAS <[email protected]> * Fix Drop Duplicates Logic Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * update docs (#286) * update docs Signed-off-by: JamesKnBr <[email protected]> * linked issues page Signed-off-by: JamesKnBr <[email protected]> * changed issues link Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: JamesKnBr <[email protected]> * Fix mkdocs colour bug (#291) * update Signed-off-by: cching95 <[email protected]> * fix colour issue on mkdocs Signed-off-by: cching95 <[email protected]> * fix colour bug on homepage Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Add Filter Parameter and TagName field mapping for OPC Publisher Transformer (#296) * Additional OPC Publisher Logic Signed-off-by: GBBBAS <[email protected]> * Add Device ID Filter Signed-off-by: GBBBAS <[email protected]> * Update Docs Signed-off-by: GBBBAS <[email protected]> * Update to Filter Logic Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Weather Domain and Data Model (#294) * added new overview and data model Signed-off-by: PaveeG <[email protected]> * adding signoff Signed-off-by: PaveeG <[email protected]> * added overview and data model Signed-off-by: PaveeG <[email protected]> * Adding signoff Signed-off-by: Author Name <[email protected]> Signed-off-by: PaveeG <[email protected]> --------- Signed-off-by: PaveeG <[email protected]> * Refactor for odbc and functions folders (#299) * Folder Refactor Signed-off-by: GBBBAS <[email protected]> * Code smell fixes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Pyspark 3.4.0 and delta-spark 2.4.0 support (#297) * Preview Support of Pyspark 3.4.0 Signed-off-by: GBBBAS <[email protected]> * Fix for running github actions tests Signed-off-by: GBBBAS <[email protected]> * Updates for delta-spark release candidate Signed-off-by: GBBBAS <[email protected]> * Update tests Signed-off-by: GBBBAS <[email protected]> * Downgrade Turbodbc to include 3.8 Signed-off-by: GBBBAS <[email protected]> * Add Spark Connect Signed-off-by: GBBBAS <[email protected]> * Updates to packages Signed-off-by: GBBBAS <[email protected]> * Updates for release of 2.4.0 Signed-off-by: GBBBAS <[email protected]> * Add hvac back Signed-off-by: GBBBAS <[email protected]> * Fixes for tests Signed-off-by: GBBBAS <[email protected]> * Remove Whitespace Signed-off-by: GBBBAS <[email protected]> * Build sdist and wheel Signed-off-by: GBBBAS <[email protected]> * Change micromamba Action Signed-off-by: GBBBAS <[email protected]> * Update Tests Workflow Signed-off-by: GBBBAS <[email protected]> * Code Smells Signed-off-by: GBBBAS <[email protected]> * Sonarqube Bug Fix Signed-off-by: GBBBAS <[email protected]> * Update Readme with logo Signed-off-by: GBBBAS <[email protected]> * Center table Signed-off-by: GBBBAS <[email protected]> * Update readme Signed-off-by: GBBBAS <[email protected]> * Folder Refactor Signed-off-by: GBBBAS <[email protected]> * Code smell fixes Signed-off-by: GBBBAS <[email protected]> * Updates for refactoriung spark connector Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Add OPC UA Transformer parameters (#301) Signed-off-by: GBBBAS <[email protected]> * Update for Micromamba Github Actions Parameters (#302) * Update micromamba parameters Signed-off-by: GBBBAS <[email protected]> * Update for Sonarqube Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Upgrade FastAPI package (#303) Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> Signed-off-by: GBBBAS <[email protected]> Signed-off-by: cching95 <[email protected]> Signed-off-by: PaveeG <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: IW-SS <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: cching95 <[email protected]> Co-authored-by: PaveeG <[email protected]> * Rebase main (#308) * remove print statement in twa Signed-off-by: cching95 <[email protected]> * v0.2.0 (#121) * Add Releases Feed to Docs Release page (#69) * fixes Add Releases Feed to Docs page #68 Signed-off-by: GBBBAS <[email protected]> * Mkdocs test fixes Signed-off-by: GBBBAS <[email protected]> * Add unit tests Signed-off-by: GBBBAS <[email protected]> * Fix Code Smell Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Update Packages and Doc Logos (#76) * Upgrade Packages Signed-off-by: GBBBAS <[email protected]> * Upgrade Functions Packages Signed-off-by: GBBBAS <[email protected]> * Add logos Signed-off-by: GBBBAS <[email protected]> * Add logo to pypi readme Signed-off-by: GBBBAS <[email protected]> * Remove pydantic models Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Remove strawberry type Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Upgrade GraphQL Package Signed-off-by: GBBBAS <[email protected]> * Logo Size Changes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Fix Logo Alignment (#78) * Upgrade Packages Signed-off-by: GBBBAS <[email protected]> * Upgrade Functions Packages Signed-off-by: GBBBAS <[email protected]> * Add logos Signed-off-by: GBBBAS <[email protected]> * Add logo to pypi readme Signed-off-by: GBBBAS <[email protected]> * Remove pydantic models Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Remove strawberry type Signed-off-by: GBBBAS <[email protected]> * Remove GraphQL Signed-off-by: GBBBAS <[email protected]> * Upgrade GraphQL Package Signed-off-by: GBBBAS <[email protected]> * Logo Size Changes Signed-off-by: GBBBAS <[email protected]> * Logo fixes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * fixes #77 (#79) Signed-off-by: GBBBAS <[email protected]> * add spark session (#80) * set up spark sessions Signed-off-by: cching95 <[email protected]> * adding unit tests for spark session Signed-off-by: cching95 <[email protected]> * remove tests for spark session Signed-off-by: cching95 <[email protected]> * fix code smells Signed-off-by: cching95 <[email protected]> * add constants in pipeline utils Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Add Pipelines as Extras libraries (#82) Signed-off-by: GBBBAS <[email protected]> * Spark pipeline source Event hub and destination Event hub (#83) * add source eventhub to pipelines Signed-off-by: cching95 <[email protected]> * updates for documentation Signed-off-by: cching95 <[email protected]> * updates to mkdocs Signed-off-by: cching95 <[email protected]> * added unit tests Signed-off-by: cching95 <[email protected]> * documentation for eventhub source Signed-off-by: cching95 <[email protected]> * fix spelling error Signed-off-by: cching95 <[email protected]> * add source eventhub destination Signed-off-by: cching95 <[email protected]> * added destination source documentation Signed-off-by: cching95 <[email protected]> * remove unit tests to revisit later Signed-off-by: cching95 <[email protected]> * fix mkdoc errors Signed-off-by: cching95 <[email protected]> * fix code smells Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Spark pipeline source delta and destination delta (#84) * Spark pipeline source delta and destination delta Signed-off-by: rodalynbarce <[email protected]> * fix mkdocs errors Signed-off-by: rodalynbarce <[email protected]> * docstring amendments Signed-off-by: rodalynbarce <[email protected]> * Added to docstrings on delta source & destination Signed-off-by: JamesKnBr <[email protected]> * added query type for eventhubs Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: cching95 <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: cching95 <[email protected]> * Added delta sharing with docs (#85) Signed-off-by: JamesKnBr <[email protected]> * Unit Tests for sources (#88) * add pipeline utils unit tests Signed-off-by: cching95 <[email protected]> * add unit test for eventhub and delta Signed-off-by: cching95 <[email protected]> * Unit Test Fixes Signed-off-by: GBBBAS <[email protected]> * remove test spark Signed-off-by: cching95 <[email protected]> * add unit test for destination delta Signed-off-by: cching95 <[email protected]> * fix eventhub spell errors Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> Signed-off-by: GBBBAS <[email protected]> Co-authored-by: GBBBAS <[email protected]> * Add pipeline eventhub body transformer (#89) * add transform eventhub Signed-off-by: cching95 <[email protected]> * transformer docs set up Signed-off-by: cching95 <[email protected]> * documentation for transformer eventhub body Signed-off-by: cching95 <[email protected]> * add transformer eventhub unit tests Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Added pipeline source autoloader and unit test (#93) Signed-off-by: rodalynbarce <[email protected]> * Pipeline Executor (#95) * Dependency Injector Signed-off-by: GBBBAS <[email protected]> * Add Pipeline Executor Signed-off-by: GBBBAS <[email protected]> * Add Utilities Interface Signed-off-by: GBBBAS <[email protected]> * Add Pipeline Executor Signed-off-by: GBBBAS <[email protected]> * Add Pipeline Execute Test Signed-off-by: GBBBAS <[email protected]> * fixes #90 Signed-off-by: GBBBAS <[email protected]> * Fixes for tests Signed-off-by: GBBBAS <[email protected]> * Step Order fix Signed-off-by: GBBBAS <[email protected]> * Sonarqube fixes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Delta Table Create Utility (#97) * Add to docs site Signed-off-by: GBBBAS <[email protected]> * Update Delta Table Name Signed-off-by: GBBBAS <[email protected]> * Add Delta Table Create Signed-off-by: GBBBAS <[email protected]> * Add Delta Table Create Test Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * remove print statement in twa (#99) Signed-off-by: cching95 <[email protected]> Co-authored-by: GBBBAS <[email protected]> Co-authored-by: Amber-Rigg <[email protected]> Co-authored-by: NooraKubati <[email protected]> * Pipeline Deployment (#102) * Add DBX Pipeline Deployment Signed-off-by: GBBBAS <[email protected]> * Updates for Pipeline Deployment Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Pipeline Deployment Updates (#103) * Pipeline Deployment Updates Signed-off-by: GBBBAS <[email protected]> * Update to run logic of Pipeline Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Updates for Pipeline Deployment (#104) Signed-off-by: GBBBAS <[email protected]> * Unit test for spark pipeline eventhub destination (#105) * eventhub destination unit test Signed-off-by: cching95 <[email protected]> * add unit test for spark eventhub destination Signed-off-by: cching95 <[email protected]> * add spark eventhub write stream unit test Signed-off-by: cching95 <[email protected]> * update exception handling for eventhub and delta Signed-off-by: cching95 <[email protected]> * update exception in utilities Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Fix for Dependency Injection (#106) Signed-off-by: GBBBAS <[email protected]> * Unit tests for source and destination streams (#108) * streaming unit test for source eventhub Signed-off-by: cching95 <[email protected]> * add delta read stream unit test Signed-off-by: cching95 <[email protected]> * add unit test for delta destination streaming Signed-off-by: cching95 <[email protected]> * remove extra test Signed-off-by: cching95 <[email protected]> * remove code smell Signed-off-by: cching95 <[email protected]> * add previous line of code back Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Pipeline Deployments and Pipeline Secrets (#111) * Fix for Dependency Injection Signed-off-by: GBBBAS <[email protected]> * Pipeline Deployment And Secrets Signed-off-by: GBBBAS <[email protected]> * Add System Type Docs Signed-off-by: GBBBAS <[email protected]> * Updates for Secrets Signed-off-by: GBBBAS <[email protected]> * Code Smell fix Signed-off-by: GBBBAS <[email protected]> * Refactor Pipeline Secret Conversion Logic Signed-off-by: GBBBAS <[email protected]> * import required objects Signed-off-by: GBBBAS <[email protected]> * Code smell fix Signed-off-by: GBBBAS <[email protected]> * Code Smell Fix Signed-off-by: GBBBAS <[email protected]> * Fix Code Smells Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Added fails tests to pipeline sources (#112) * Added fails tests to pipeline sources Signed-off-by: JamesKnBr <[email protected]> * Added fails tests for pipeline destinations Signed-off-by: JamesKnBr <[email protected]> * Fixed code smell Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: JamesKnBr <[email protected]> * Fix for dockerbuild error (#114) Signed-off-by: GBBBAS <[email protected]> * Fix for docker build (#115) Signed-off-by: GBBBAS <[email protected]> * Cleanup Dockerfile (#116) Signed-off-by: GBBBAS <[email protected]> * Pipeline Secrets, Converters and Deployment Updates (#119) * Updates for Pipelines Signed-off-by: GBBBAS <[email protected]> * Update DBX Documentation Signed-off-by: GBBBAS <[email protected]> * Updates for Pipeline Deployment Signed-off-by: GBBBAS <[email protected]> * Remove package.json Signed-off-by: GBBBAS <[email protected]> * Fix code smells Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * time weighted average bug fix (#120) * fix twa group by dataframe return Signed-off-by: cching95 <[email protected]> * bug fixes for groupby and apply method Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> Signed-off-by: cching95 <[email protected]> Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: GBBBAS <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: Amber-Rigg <[email protected]> Co-authored-by: NooraKubati <[email protected]> * v0.4.0 (#304) * EdgeX Transformer (#271) * added transformer component Signed-off-by: rodalynbarce <[email protected]> * added edgeX test Signed-off-by: JamesKnBr <[email protected]> * update test Signed-off-by: JamesKnBr <[email protected]> * test updates Signed-off-by: rodalynbarce <[email protected]> * added kafka destination write format Signed-off-by: JamesKnBr <[email protected]> * component update Signed-off-by: rodalynbarce <[email protected]> * removed timestamp format Signed-off-by: JamesKnBr <[email protected]> * updated docs Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> * Feature/00264 (#272) * EdgeX Transformer (#271) * added transformer component Signed-off-by: rodalynbarce <[email protected]> * added edgeX test Signed-off-by: JamesKnBr <[email protected]> * update test Signed-off-by: JamesKnBr <[email protected]> * test updates Signed-off-by: rodalynbarce <[email protected]> * added kafka destination write format Signed-off-by: JamesKnBr <[email protected]> * component update Signed-off-by: rodalynbarce <[email protected]> * removed timestamp format Signed-off-by: JamesKnBr <[email protected]> * updated docs Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> * Adding support for MISO Sources Signed-off-by: Shivam Saxena <[email protected]> * Fixing sonar smells Signed-off-by: Shivam Saxena <[email protected]> * Fixing sonar smells Signed-off-by: Shivam Saxena <[email protected]> * Removing ISO Runner file Signed-off-by: Shivam Saxena <[email protected]> * Making ISO methods internal Signed-off-by: Shivam Saxena <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> * Add Partition Pruning to non broadcast join PCDM Merge (#274) * Add Partition Pruning on non broadcast join merge Signed-off-by: GBBBAS <[email protected]> * Update DropDuplicates Signed-off-by: GBBBAS <[email protected]> * Fix Drop Duplicates Logic Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * update docs (#286) * update docs Signed-off-by: JamesKnBr <[email protected]> * linked issues page Signed-off-by: JamesKnBr <[email protected]> * changed issues link Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: JamesKnBr <[email protected]> * Fix mkdocs colour bug (#291) * update Signed-off-by: cching95 <[email protected]> * fix colour issue on mkdocs Signed-off-by: cching95 <[email protected]> * fix colour bug on homepage Signed-off-by: cching95 <[email protected]> --------- Signed-off-by: cching95 <[email protected]> * Add Filter Parameter and TagName field mapping for OPC Publisher Transformer (#296) * Additional OPC Publisher Logic Signed-off-by: GBBBAS <[email protected]> * Add Device ID Filter Signed-off-by: GBBBAS <[email protected]> * Update Docs Signed-off-by: GBBBAS <[email protected]> * Update to Filter Logic Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Weather Domain and Data Model (#294) * added new overview and data model Signed-off-by: PaveeG <[email protected]> * adding signoff Signed-off-by: PaveeG <[email protected]> * added overview and data model Signed-off-by: PaveeG <[email protected]> * Adding signoff Signed-off-by: Author Name <[email protected]> Signed-off-by: PaveeG <[email protected]> --------- Signed-off-by: PaveeG <[email protected]> * Refactor for odbc and functions folders (#299) * Folder Refactor Signed-off-by: GBBBAS <[email protected]> * Code smell fixes Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Pyspark 3.4.0 and delta-spark 2.4.0 support (#297) * Preview Support of Pyspark 3.4.0 Signed-off-by: GBBBAS <[email protected]> * Fix for running github actions tests Signed-off-by: GBBBAS <[email protected]> * Updates for delta-spark release candidate Signed-off-by: GBBBAS <[email protected]> * Update tests Signed-off-by: GBBBAS <[email protected]> * Downgrade Turbodbc to include 3.8 Signed-off-by: GBBBAS <[email protected]> * Add Spark Connect Signed-off-by: GBBBAS <[email protected]> * Updates to packages Signed-off-by: GBBBAS <[email protected]> * Updates for release of 2.4.0 Signed-off-by: GBBBAS <[email protected]> * Add hvac back Signed-off-by: GBBBAS <[email protected]> * Fixes for tests Signed-off-by: GBBBAS <[email protected]> * Remove Whitespace Signed-off-by: GBBBAS <[email protected]> * Build sdist and wheel Signed-off-by: GBBBAS <[email protected]> * Change micromamba Action Signed-off-by: GBBBAS <[email protected]> * Update Tests Workflow Signed-off-by: GBBBAS <[email protected]> * Code Smells Signed-off-by: GBBBAS <[email protected]> * Sonarqube Bug Fix Signed-off-by: GBBBAS <[email protected]> * Update Readme with logo Signed-off-by: GBBBAS <[email protected]> * Center table Signed-off-by: GBBBAS <[email protected]> * Update readme Signed-off-by: GBBBAS <[email protected]> * Folder Refactor Signed-off-by: GBBBAS <[email protected]> * Code smell fixes Signed-off-by: GBBBAS <[email protected]> * Updates for refactoriung spark connector Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Add OPC UA Transformer parameters (#301) Signed-off-by: GBBBAS <[email protected]> * Update for Micromamba Github Actions Parameters (#302) * Update micromamba parameters Signed-off-by: GBBBAS <[email protected]> * Update for Sonarqube Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: GBBBAS <[email protected]> * Upgrade FastAPI package (#303) Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> Signed-off-by: GBBBAS <[email protected]> Signed-off-by: cching95 <[email protected]> Signed-off-by: PaveeG <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: IW-SS <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: cching95 <[email protected]> Co-authored-by: PaveeG <[email protected]> * Resolve imports on deprecated folders Signed-off-by: GBBBAS <[email protected]> * Updates to remove unwanted files Signed-off-by: GBBBAS <[email protected]> --------- Signed-off-by: cching95 <[email protected]> Signed-off-by: GBBBAS <[email protected]> Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> Signed-off-by: PaveeG <[email protected]> Co-authored-by: cching95 <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: Amber-Rigg <[email protected]> Co-authored-by: NooraKubati <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: IW-SS <[email protected]> Co-authored-by: PaveeG <[email protected]> * Merge main to develop Signed-off-by: GBBBAS <[email protected]> * added weather transformer, docs, test - modified source weather Signed-off-by: PaveeG <[email protected]> * refactored code Signed-off-by: PaveeG <[email protected]> * modified weather source unit test Signed-off-by: PaveeG <[email protected]> * reformated code Signed-off-by: PaveeG <[email protected]> * updated camel casing in weather transformer and tests Signed-off-by: PaveeG <[email protected]> * removed spark session from tests Signed-off-by: PaveeG <[email protected]> * changed source test case Signed-off-by: PaveeG <[email protected]> * added weather transformer Signed-off-by: PaveeG <[email protected]> * added component Signed-off-by: PaveeG <[email protected]> * removed duplicates Signed-off-by: PaveeG <[email protected]> * changed if expression Signed-off-by: PaveeG <[email protected]> * merge Signed-off-by: PaveeG <[email protected]> * fixed logic expression Signed-off-by: PaveeG <[email protected]> * merge Signed-off-by: PaveeG <[email protected]> * removed duplicate strings in test Signed-off-by: PaveeG <[email protected]> --------- Signed-off-by: cching95 <[email protected]> Signed-off-by: GBBBAS <[email protected]> Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> Signed-off-by: PaveeG <[email protected]> Co-authored-by: GBBBAS <[email protected]> Co-authored-by: Amber-Rigg <[email protected]> Co-authored-by: NooraKubati <[email protected]> Co-authored-by: cching95 <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: IW-SS <[email protected]> * API Updates (#404) Signed-off-by: GBBBAS <[email protected]> * AMQP value decoding from Kafka Headers (#403) Signed-off-by: GBBBAS <[email protected]> * Return Properties initial value if not populated (#405) Signed-off-by: GBBBAS <[email protected]> * Delete run_delta_stream.py Signed-off-by: JamesKnBr <[email protected]> * fix code smells Signed-off-by: JamesKnBr <[email protected]> * Added test for dataframe type Signed-off-by: rodalynbarce <[email protected]> * updated delta destination and yaml Signed-off-by: JamesKnBr <[email protected]> * remove delta-sharing from yaml Signed-off-by: JamesKnBr <[email protected]> * add python delta sharing source Signed-off-by: JamesKnBr <[email protected]> --------- Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: rodalynbarce <[email protected]> Signed-off-by: cching95 <[email protected]> Signed-off-by: GBBBAS <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Signed-off-by: Shivam Saxena <[email protected]> Signed-off-by: PaveeG <[email protected]> Signed-off-by: JamesKnBr <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: cching95 <[email protected]> Co-authored-by: PaveeG <[email protected]> Co-authored-by: GBBBAS <[email protected]> Co-authored-by: Amber-Rigg <[email protected]> Co-authored-by: NooraKubati <[email protected]> Co-authored-by: rodalynbarce <[email protected]> Co-authored-by: IW-SS <[email protected]>
rtdip · Aug 2, 2023 · 534d2fb · 534d2fb
1 parent ead4316
commit 534d2fb
Show file tree

Hide file tree

Showing 17 changed files with 508 additions and 1 deletion.
diff --git a/docs/sdk/code-reference/pipelines/destinations/python/delta.md b/docs/sdk/code-reference/pipelines/destinations/python/delta.md
@@ -0,0 +1,2 @@
+# Write to Delta
+::: src.sdk.python.rtdip_sdk.pipelines.destinations.python.delta
diff --git a/docs/sdk/code-reference/pipelines/sources/python/delta.md b/docs/sdk/code-reference/pipelines/sources/python/delta.md
@@ -0,0 +1,2 @@
+# Read from Delta
+::: src.sdk.python.rtdip_sdk.pipelines.sources.python.delta
diff --git a/docs/sdk/code-reference/pipelines/sources/python/delta_sharing.md b/docs/sdk/code-reference/pipelines/sources/python/delta_sharing.md
@@ -0,0 +1,2 @@
+# Read from Delta with Delta Sharing
+::: src.sdk.python.rtdip_sdk.pipelines.sources.python.delta_sharing
diff --git a/environment.yml b/environment.yml
@@ -70,4 +70,6 @@ dependencies:
         - deltalake==0.10.0
         - moto[s3]==4.1.13
         - build==0.10.0
+        - polars==0.18.8
+        - delta-sharing==0.7.3
 
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -160,6 +160,9 @@ nav:
                           - Base Weather: sdk/code-reference/pipelines/sources/spark/weather/base_weather.md
                           - Weather Forecast API V1: sdk/code-reference/pipelines/sources/spark/weather/weather_forecast_api_v1.md
                           - Weather Forecast API V1 Multi: sdk/code-reference/pipelines/sources/spark/weather/weather_forecast_api_v1_multi.md
+                  - Python:
+                      - Delta: sdk/code-reference/pipelines/sources/python/delta.md
+                      - Delta Sharing: sdk/code-reference/pipelines/sources/python/delta_sharing.md
               - Transformers:
                   - Spark:
                       - Binary To String: sdk/code-reference/pipelines/transformers/spark/binary_to_string.md
@@ -186,6 +189,8 @@ nav:
                       - Kinesis: sdk/code-reference/pipelines/destinations/spark/kinesis.md
                       - Rest API: sdk/code-reference/pipelines/destinations/spark/rest_api.md
                       - Process Control Data Model To Delta: sdk/code-reference/pipelines/destinations/spark/pcdm_to_delta.md
+                  - Python:
+                      - Delta: sdk/code-reference/pipelines/destinations/python/delta.md
                   - Blockchain:
                       - EVM: sdk/code-reference/pipelines/destinations/blockchain/evm.md
 

diff --git a/src/sdk/python/rtdip_sdk/pipelines/destinations/__init__.py b/src/sdk/python/rtdip_sdk/pipelines/destinations/__init__.py
@@ -19,4 +19,5 @@
 from .spark.kinesis import *
 from .spark.rest_api import *
 from .spark.pcdm_to_delta import *
-from .blockchain.evm import *
+from .blockchain.evm import *
+from .python.delta import *
diff --git a/src/sdk/python/rtdip_sdk/pipelines/destinations/python/__init__.py b/src/sdk/python/rtdip_sdk/pipelines/destinations/python/__init__.py
@@ -0,0 +1,13 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/src/sdk/python/rtdip_sdk/pipelines/destinations/python/delta.py b/src/sdk/python/rtdip_sdk/pipelines/destinations/python/delta.py
@@ -0,0 +1,93 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import time
+import pandas as pd
+from deltalake import write_deltalake, DeltaTable
+from typing import Literal
+import pyarrow as pa
+import polars as pl
+from polars import LazyFrame
+from typing import Callable
+from ..interfaces import DestinationInterface
+from ..._pipeline_utils.models import Libraries, SystemType
+from ..._pipeline_utils.constants import get_default_package
+
+class PythonDeltaDestination(DestinationInterface):
+    '''
+    The Python Delta Destination is used to write data to a Delta table from a Polars LazyFrame.
+
+    Args:
+        data (LazyFrame): Polars LazyFrame to be written to Delta
+        path (str): Path to Delta table to be written to; either local or [remote](https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table){ target="_blank" }. **Locally** if the Table does't exist one will be created, but to write to AWS or Azure, you must have an existing Delta Table
+        options (Optional dict): Used if writing to a remote location. For AWS use format {"aws_access_key_id": "<>", "aws_secret_access_key": "<>"}. For Azure use format {"azure_storage_account_name": "storageaccountname", "azure_storage_access_key": "<>"}
+        mode (Literal['error', 'append', 'overwrite', 'ignore']): Defaults to error if table exists, 'ignore' won't write anything if table exists
+        overwrite_schema (bool): If True will allow for the table schema to be overwritten
+        delta_write_options (dict): Options when writing to a Delta table. See [here](https://delta-io.github.io/delta-rs/python/api_reference.html#writing-deltatables){ target="_blank" } for all options
+    '''
+    data: LazyFrame
+    path: str
+    options: dict
+    mode: Literal['error', 'append', 'overwrite', 'ignore']
+    overwrite_schema: bool
+    delta_write_options: bool
+
+    def __init__(self, data: LazyFrame, path: str, options: dict = None, mode: Literal['error', 'append', 'overwrite', 'ignore'] = 'error', overwrite_schema: bool = False, delta_write_options: bool = False, query_name = None) -> None:
+        self.data = data
+        self.path = path
+        self.options = options
+        self.mode = mode
+        self.overwrite_schema = overwrite_schema
+        self.delta_write_options = delta_write_options
+
+    @staticmethod
+    def system_type():
+        '''
+        Attributes:
+            SystemType (Environment): Requires PYTHON
+        '''             
+        return SystemType.PYTHON
+
+    @staticmethod
+    def libraries():
+        libraries = Libraries()
+        return libraries
+
+    @staticmethod
+    def settings() -> dict:
+        return {}
+
+    def pre_write_validation(self):
+        return True
+
+    def post_write_validation(self):
+        return True
+
+    def write_batch(self):
+        '''
+        Writes batch data to Delta without using Spark.
+        '''  
+        if isinstance(self.data, pl.LazyFrame):
+            df = self.data.collect()
+            df.write_delta(self.path, mode=self.mode, overwrite_schema= self.overwrite_schema, storage_options=self.options, delta_write_options=self.delta_write_options)
+        else:
+            raise ValueError("Data must be a Polars LazyFrame. See https://pola-rs.github.io/polars/py-polars/html/reference/lazyframe/index.html")
+
+    def write_stream(self):
+        '''
+        Raises:
+            NotImplementedError: Writing to a Delta table using Python is only possible for batch writes. To perform a streaming read, use the write_stream method of the SparkDeltaDestination component.
+        '''
+        raise NotImplementedError("Writing to a Delta table using Python is only possible for batch writes. To perform a streaming read, use the write_stream method of the SparkDeltaDestination component")
diff --git a/src/sdk/python/rtdip_sdk/pipelines/sources/__init__.py b/src/sdk/python/rtdip_sdk/pipelines/sources/__init__.py
@@ -22,3 +22,5 @@
 from .spark.kinesis import *
 from .spark.iso import *
 from .spark.weather import *
+from .python.delta import *
+from .python.delta_sharing import *
diff --git a/src/sdk/python/rtdip_sdk/pipelines/sources/python/__init__.py b/src/sdk/python/rtdip_sdk/pipelines/sources/python/__init__.py
@@ -0,0 +1,13 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
diff --git a/src/sdk/python/rtdip_sdk/pipelines/sources/python/delta.py b/src/sdk/python/rtdip_sdk/pipelines/sources/python/delta.py
@@ -0,0 +1,82 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ..interfaces import SourceInterface
+from ..._pipeline_utils.models import Libraries, SystemType
+from ..._pipeline_utils.constants import get_default_package
+import polars as pl
+from polars import LazyFrame
+
+class PythonDeltaSource(SourceInterface):
+    '''
+    The Python Delta Source is used to read data from a Delta table without using Apache Spark, returning a Polars LazyFrame
+
+    Args:
+        path (str): Path to the Delta table. Can be local or in S3/Azure storage
+        version (optional int): Specify the Delta table version to read from. Defaults to the latest version
+        storage_options (optional dict): Used to read from AWS/Azure storage. For AWS use format {"aws_access_key_id": "<>", "aws_secret_access_key":"<>"}. For Azure use format {"azure_storage_account_name": "<>", "azure_storage_account_key": "<>"}.
+        pyarrow_options (optional dict): Data Access and Efficiency options when reading from Delta. See [to_pyarrow_dataset](https://delta-io.github.io/delta-rs/python/api_reference.html#deltalake.table.DeltaTable.to_pyarrow_dataset){ target="_blank" }.
+        without_files (optional bool): If True loads the table without tracking files
+    '''
+    path: str
+    version: int
+    storage_options: dict
+    pyarrow_options: dict
+    without_files: bool
+
+    def __init__(self, path: str, version: int = None, storage_options: dict = None, pyarrow_options: dict = None, without_files: bool = False):
+        self.path = path
+        self.version = version
+        self.storage_options = storage_options
+        self.pyarrow_options = pyarrow_options
+        self.without_files = without_files
+
+    @staticmethod
+    def system_type():
+        '''
+        Attributes:
+            SystemType (Environment): Requires PYTHON
+        '''            
+        return SystemType.PYTHON
+
+    @staticmethod
+    def libraries():
+        libraries = Libraries()
+        return libraries
+
+    @staticmethod
+    def settings() -> dict:
+        return {}
+
+    def pre_read_validation(self):
+        return True
+
+    def post_read_validation(self):
+        return True
+
+    def read_batch(self) -> LazyFrame:
+        '''
+        Reads data from a Delta table into a Polars LazyFrame
+        '''
+        without_files_dict = {"without_files": self.without_files}
+        lf = pl.scan_delta(source= self.path, version= self.version, storage_options= self.storage_options, delta_table_options= without_files_dict, pyarrow_options= self.pyarrow_options)
+        return lf
+
+    def read_stream(self):
+        '''
+        Raises:
+            NotImplementedError: Reading from a Delta table using Python is only possible for batch reads. To perform a streaming read, use the read_stream method of the SparkDeltaSource component.
+        '''
+        raise NotImplementedError("Reading from a Delta table using Python is only possible for batch reads. To perform a streaming read, use the read_stream method of the SparkDeltaSource component")
+
diff --git a/src/sdk/python/rtdip_sdk/pipelines/sources/python/delta_sharing.py b/src/sdk/python/rtdip_sdk/pipelines/sources/python/delta_sharing.py
@@ -0,0 +1,81 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ..interfaces import SourceInterface
+from ..._pipeline_utils.models import Libraries, SystemType
+from ..._pipeline_utils.constants import get_default_package
+import delta_sharing
+import polars as pl
+from polars import LazyFrame
+
+
+class PythonDeltaSharingSource(SourceInterface):
+    '''
+    The Python Delta Sharing Source is used to read data from a Delta table with Delta Sharing configured, without using Apache Spark.
+
+    Args:
+        profile_path (str): Location of the credential file. Can be any URL supported by [FSSPEC](https://filesystem-spec.readthedocs.io/en/latest/index.html){ target="_blank" }
+        share_name (str): The value of 'share=' for the table
+        schema_name (str): The value of 'schema=' for the table
+        table_name (str): The value of 'name=' for the table
+
+    '''
+    profile_path: str
+    share_name: str
+    schema_name: str
+    table_name: str
+
+    def __init__(self, profile_path: str, share_name: str, schema_name: str, table_name: str):
+        self.profile_path = profile_path
+        self.share_name = share_name
+        self.schema_name = schema_name
+        self.table_name = table_name
+
+    @staticmethod
+    def system_type():
+        '''
+        Attributes:
+            SystemType (Environment): Requires PYTHON
+        '''            
+        return SystemType.PYTHON
+
+    @staticmethod
+    def libraries():
+        libraries = Libraries()
+        return libraries
+
+    @staticmethod
+    def settings() -> dict:
+        return {}
+
+    def pre_read_validation(self):
+        return True
+
+    def post_read_validation(self):
+        return True
+
+    def read_batch(self) -> LazyFrame:
+        '''
+        Reads data from a Delta table with Delta Sharing into a Polars LazyFrame.
+        '''
+        pandas_df = delta_sharing.load_as_pandas(f"{self.profile_path}#{self.share_name}.{self.schema_name}.{self.table_name}")
+        polars_lazyframe = pl.from_pandas(pandas_df).lazy()
+        return polars_lazyframe
+
+    def read_stream(self):
+        '''
+        Raises:
+            NotImplementedError: Reading from a Delta table with Delta Sharing using Python is only possible for batch reads.
+        '''
+        raise NotImplementedError("Reading from a Delta table with Delta Sharing using Python is only possible for batch reads.")
diff --git a/tests/sdk/python/rtdip_sdk/pipelines/destinations/python/__init__.py b/tests/sdk/python/rtdip_sdk/pipelines/destinations/python/__init__.py
@@ -0,0 +1,13 @@
+# Copyright 2022 RTDIP
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		# Write to Delta
		::: src.sdk.python.rtdip_sdk.pipelines.destinations.python.delta
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		# Read from Delta
		::: src.sdk.python.rtdip_sdk.pipelines.sources.python.delta
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		# Read from Delta with Delta Sharing
		::: src.sdk.python.rtdip_sdk.pipelines.sources.python.delta_sharing