From 510ebe6875161300582635fba25e12328d924d9e Mon Sep 17 00:00:00 2001 From: open-metadata Date: Thu, 5 Dec 2024 06:34:46 +0000 Subject: [PATCH] See https://github.com/open-metadata/OpenMetadata/commit/4be0bed056697f92be902ba213eac5bd5c4b0ab4 from refs/heads/main --- .../build-code-and-run-tests/prerequisites.md | 2 +- .../build-code-and-run-tests/prerequisites.md | 2 +- .../develop-ingestion-code.md | 11 +---------- 3 files changed, 3 insertions(+), 12 deletions(-) diff --git a/content/v1.5.x/developers/contribute/build-code-and-run-tests/prerequisites.md b/content/v1.5.x/developers/contribute/build-code-and-run-tests/prerequisites.md index e096aa61..8bc120d1 100644 --- a/content/v1.5.x/developers/contribute/build-code-and-run-tests/prerequisites.md +++ b/content/v1.5.x/developers/contribute/build-code-and-run-tests/prerequisites.md @@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development: - [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli` - [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu) - [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11) -- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/) +- [Python 3.8 or 3.9](https://www.python.org/downloads/) - [Node 18.x](https://nodejs.org/en/download/) - [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/) - [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/) diff --git a/content/v1.6.x-SNAPSHOT/developers/contribute/build-code-and-run-tests/prerequisites.md b/content/v1.6.x-SNAPSHOT/developers/contribute/build-code-and-run-tests/prerequisites.md index e096aa61..8bc120d1 100644 --- a/content/v1.6.x-SNAPSHOT/developers/contribute/build-code-and-run-tests/prerequisites.md +++ b/content/v1.6.x-SNAPSHOT/developers/contribute/build-code-and-run-tests/prerequisites.md @@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development: - [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli` - [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu) - [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11) -- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/) +- [Python 3.8 or 3.9](https://www.python.org/downloads/) - [Node 18.x](https://nodejs.org/en/download/) - [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/) - [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/) diff --git a/content/v1.6.x-SNAPSHOT/developers/contribute/developing-a-new-connector/develop-ingestion-code.md b/content/v1.6.x-SNAPSHOT/developers/contribute/developing-a-new-connector/develop-ingestion-code.md index 2b2917bb..e2cd41ec 100644 --- a/content/v1.6.x-SNAPSHOT/developers/contribute/developing-a-new-connector/develop-ingestion-code.md +++ b/content/v1.6.x-SNAPSHOT/developers/contribute/developing-a-new-connector/develop-ingestion-code.md @@ -15,14 +15,10 @@ The main takes for developing a new connector are: When developing a new database ingestion connector in OpenMetadata, ensure all necessary components are correctly configured. This guide outlines the steps required to define the connector's ingestion capabilities using a `service_spec.py` file. ---- - ### Why Use `service_spec.py`? The `service_spec.py` file centralizes the definitions of sources, profilers, lineage, and other ingestion-related components for a connector. This approach helps standardize implementations across connectors, making it easier to manage ingestion workflows. ---- - ### Steps to Develop a New Connector #### 1. Create the `service_spec.py` File @@ -57,17 +53,13 @@ ServiceSpec = DefaultDatabaseSpec( Replace the example classes (e.g., `BigquerySource`, `BigqueryLineageSource`, etc.) with those specific to your connector. Depending on the connector's features, you may include or exclude certain components like usage or profiling. ---- - ### Components of `service_spec.py` - **`metadata_source_class`**: Defines the class for metadata ingestion. - **`lineage_source_class`**: Defines the class for lineage extraction. - **`usage_source_class`**: Tracks data usage patterns. - **`profiler_class`**: Profiles data for quality and insights. -- **`sampler_class`**: Samples data for efficient ingestion. - ---- +- **`sampler_class`**: Samples data for efficient ingestion. ### Example Workflow @@ -80,7 +72,6 @@ Define the `ServiceSpec` using the required classes, adjusting for your connecto #### Step 3: Verify Integration Run the ingestion workflow to test the connector and ensure all components are functioning correctly. - ## Service Topology The Topology defines a series of Nodes and Stages that get executed in a hierarchical way and describe how we extract the needed data from the sources.