Skip to content

Commit

Permalink
See open-metadata/OpenMetadata@4be0bed from refs/heads/main
Browse files Browse the repository at this point in the history
  • Loading branch information
open-metadata committed Dec 5, 2024
1 parent dd181f7 commit 510ebe6
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 12 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development:
- [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli`
- [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu)
- [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11)
- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/)
- [Python 3.8 or 3.9](https://www.python.org/downloads/)
- [Node 18.x](https://nodejs.org/en/download/)
- [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/)
- [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ OpenMetadata being a full stack project, we use the following for development:
- [Antlr 4.9.2](https://www.antlr.org/) - `sudo make install_antlr_cli`
- [JQ](https://jqlang.github.io/jq/) - `brew install jq` (osx) `apt-get install jq` (Ubuntu)
- [Maven 3.5.x or higher](https://maven.apache.org/install.html) - (with Java JDK 11)
- [Python 3.7, 3.8 or 3.9](https://www.python.org/downloads/)
- [Python 3.8 or 3.9](https://www.python.org/downloads/)
- [Node 18.x](https://nodejs.org/en/download/)
- [Yarn ^1.22.0](https://classic.yarnpkg.com/lang/en/docs/install/)
- [Rpm (Optional, only to run RPM profile with maven)](https://macappstore.org/rpm/)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,10 @@ The main takes for developing a new connector are:

When developing a new database ingestion connector in OpenMetadata, ensure all necessary components are correctly configured. This guide outlines the steps required to define the connector's ingestion capabilities using a `service_spec.py` file.

---

### Why Use `service_spec.py`?

The `service_spec.py` file centralizes the definitions of sources, profilers, lineage, and other ingestion-related components for a connector. This approach helps standardize implementations across connectors, making it easier to manage ingestion workflows.

---

### Steps to Develop a New Connector

#### 1. Create the `service_spec.py` File
Expand Down Expand Up @@ -57,17 +53,13 @@ ServiceSpec = DefaultDatabaseSpec(

Replace the example classes (e.g., `BigquerySource`, `BigqueryLineageSource`, etc.) with those specific to your connector. Depending on the connector's features, you may include or exclude certain components like usage or profiling.

---

### Components of `service_spec.py`

- **`metadata_source_class`**: Defines the class for metadata ingestion.
- **`lineage_source_class`**: Defines the class for lineage extraction.
- **`usage_source_class`**: Tracks data usage patterns.
- **`profiler_class`**: Profiles data for quality and insights.
- **`sampler_class`**: Samples data for efficient ingestion.

---
- **`sampler_class`**: Samples data for efficient ingestion.

### Example Workflow

Expand All @@ -80,7 +72,6 @@ Define the `ServiceSpec` using the required classes, adjusting for your connecto
#### Step 3: Verify Integration
Run the ingestion workflow to test the connector and ensure all components are functioning correctly.


## Service Topology

The Topology defines a series of Nodes and Stages that get executed in a hierarchical way and describe how we extract the needed data from the sources.
Expand Down

0 comments on commit 510ebe6

Please sign in to comment.