-
Notifications
You must be signed in to change notification settings - Fork 2.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
e279403
commit c735cf4
Showing
347 changed files
with
136,812 additions
and
87 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,185 @@ | ||
--- | ||
description: >- | ||
DataHub is a data discovery application built on an extensible metadata | ||
platform that helps you tame the complexity of diverse data ecosystems. | ||
hide_title: true | ||
title: Introduction | ||
slug: /introduction | ||
custom_edit_url: "https://github.com/datahub-project/datahub/blob/master/README.md" | ||
--- | ||
|
||
import useBaseUrl from '@docusaurus/useBaseUrl'; | ||
|
||
export const Logo = (props) => { | ||
return ( | ||
|
||
<div style={{ display: "flex", justifyContent: "center", padding: "20px", height: "190px" }}> | ||
<img | ||
alt="DataHub Logo" | ||
src={useBaseUrl("/img/datahub-logo-color-mark.svg")} | ||
{...props} | ||
/> | ||
</div> | ||
); | ||
}; | ||
|
||
<Logo /> | ||
|
||
<!-- | ||
<p align="center"> | ||
<img alt="DataHub" src="docs/imgs/datahub-logo-color-mark.svg" height="150" /> | ||
</p> | ||
<!-- --> | ||
|
||
# DataHub: The Metadata Platform for the Modern Data Stack | ||
|
||
## Built with ❤️ by <img src="https://datahubproject.io/img/acryl-logo-light-mark.png" width="25"/> [Acryl Data](https://acryldata.io) and <img src="https://datahubproject.io/img/LI-In-Bug.png" width="25"/> [LinkedIn](https://engineering.linkedin.com) | ||
|
||
[![Version](https://img.shields.io/github/v/release/datahub-project/datahub?include_prereleases)](https://github.com/datahub-project/datahub/releases/latest) | ||
[![PyPI version](https://badge.fury.io/py/acryl-datahub.svg)](https://badge.fury.io/py/acryl-datahub) | ||
[![build & test](https://github.com/datahub-project/datahub/workflows/build%20&%20test/badge.svg?branch=master&event=push)](https://github.com/datahub-project/datahub/actions?query=workflow%3A%22build+%26+test%22+branch%3Amaster+event%3Apush) | ||
[![Docker Pulls](https://img.shields.io/docker/pulls/linkedin/datahub-gms.svg)](https://hub.docker.com/r/linkedin/datahub-gms) | ||
[![Slack](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://slack.datahubproject.io) | ||
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/datahub-project/datahub/blob/master/docs/CONTRIBUTING.md) | ||
[![GitHub commit activity](https://img.shields.io/github/commit-activity/m/datahub-project/datahub)](https://github.com/datahub-project/datahub/pulls?q=is%3Apr) | ||
[![License](https://img.shields.io/github/license/datahub-project/datahub)](https://github.com/datahub-project/datahub/blob/master/LICENSE) | ||
[![YouTube](https://img.shields.io/youtube/channel/subscribers/UC3qFQC5IiwR5fvWEqi_tJ5w?style=social)](https://www.youtube.com/channel/UC3qFQC5IiwR5fvWEqi_tJ5w) | ||
[![Medium](https://img.shields.io/badge/Medium-12100E?style=for-the-badge&logo=medium&logoColor=white)](https://medium.com/datahub-project) | ||
[![Follow](https://img.shields.io/twitter/follow/datahubproject?label=Follow&style=social)](https://twitter.com/datahubproject) | ||
|
||
### 🏠 Hosted DataHub Docs (Courtesy of Acryl Data): [datahubproject.io](/docs) | ||
|
||
--- | ||
|
||
[Quickstart](/docs/quickstart) | | ||
[Features](/docs/features) | | ||
[Roadmap](https://feature-requests.datahubproject.io/roadmap) | | ||
[Adoption](#adoption) | | ||
[Demo](https://demo.datahubproject.io/) | | ||
[Town Hall](/docs/townhalls) | ||
|
||
--- | ||
|
||
> 📣 DataHub Town Hall is the 4th Thursday at 9am US PT of every month - [add it to your calendar!](https://rsvp.datahubproject.io/) | ||
> | ||
> - Town-hall Zoom link: [zoom.datahubproject.io](https://zoom.datahubproject.io) | ||
> - [Meeting details](docs/townhalls.md) & [past recordings](docs/townhall-history.md) | ||
> ✨ DataHub Community Highlights: | ||
> | ||
> - Read our Monthly Project Updates [here](https://blog.datahubproject.io/tagged/project-updates). | ||
> - Bringing The Power Of The DataHub Real-Time Metadata Graph To Everyone At Acryl Data: [Data Engineering Podcast](https://www.dataengineeringpodcast.com/acryl-data-datahub-metadata-graph-episode-230/) | ||
> - Check out our most-read blog post, [DataHub: Popular Metadata Architectures Explained](https://engineering.linkedin.com/blog/2020/datahub-popular-metadata-architectures-explained) @ LinkedIn Engineering Blog. | ||
> - Join us on [Slack](docs/slack.md)! Ask questions and keep up with the latest announcements. | ||
## Introduction | ||
|
||
DataHub is an open-source metadata platform for the modern data stack. Read about the architectures of different metadata systems and why DataHub excels [here](https://engineering.linkedin.com/blog/2020/datahub-popular-metadata-architectures-explained). Also read our | ||
[LinkedIn Engineering blog post](https://engineering.linkedin.com/blog/2019/data-hub), check out our [Strata presentation](https://speakerdeck.com/shirshanka/the-evolution-of-metadata-linkedins-journey-strata-nyc-2019) and watch our [Crunch Conference Talk](https://www.youtube.com/watch?v=OB-O0Y6OYDE). You should also visit [DataHub Architecture](docs/architecture/architecture.md) to get a better understanding of how DataHub is implemented. | ||
|
||
## Features & Roadmap | ||
|
||
Check out DataHub's [Features](docs/features.md) & [Roadmap](https://feature-requests.datahubproject.io/roadmap). | ||
|
||
## Demo and Screenshots | ||
|
||
There's a [hosted demo environment](https://demo.datahubproject.io/) courtesy of [Acryl Data](https://acryldata.io) where you can explore DataHub without installing it locally | ||
|
||
## Quickstart | ||
|
||
Please follow the [DataHub Quickstart Guide](/docs/quickstart) to get a copy of DataHub up & running locally using [Docker](https://docker.com). As the guide assumes some basic knowledge of Docker, we'd recommend you to go through the "Hello World" example of [A Docker Tutorial for Beginners](https://docker-curriculum.com) if Docker is completely foreign to you. | ||
|
||
## Development | ||
|
||
If you're looking to build & modify datahub please take a look at our [Development Guide](/docs/developers). | ||
|
||
<p align="center"> | ||
<img width="70%" href="https://demo.datahubproject.io/" src="https://raw.githubusercontent.com/acryldata/static-assets-test/master/imgs/entity.png"/> | ||
</p> | ||
|
||
## Source Code and Repositories | ||
|
||
- [datahub-project/datahub](https://github.com/datahub-project/datahub): This repository contains the complete source code for DataHub's metadata model, metadata services, integration connectors and the web application. | ||
- [acryldata/datahub-actions](https://github.com/acryldata/datahub-actions): DataHub Actions is a framework for responding to changes to your DataHub Metadata Graph in real time. | ||
- [acryldata/datahub-helm](https://github.com/acryldata/datahub-helm): Repository of helm charts for deploying DataHub on a Kubernetes cluster | ||
- [acryldata/meta-world](https://github.com/acryldata/meta-world): A repository to store recipes, custom sources, transformations and other things to make your DataHub experience magical | ||
|
||
## Releases | ||
|
||
See [Releases](https://github.com/datahub-project/datahub/releases) page for more details. We follow the [SemVer Specification](https://semver.org) when versioning the releases and adopt the [Keep a Changelog convention](https://keepachangelog.com/) for the changelog format. | ||
|
||
## Contributing | ||
|
||
We welcome contributions from the community. Please refer to our [Contributing Guidelines](docs/CONTRIBUTING.md) for more details. We also have a [contrib](https://github.com/datahub-project/datahub/blob/master/contrib) directory for incubating experimental features. | ||
|
||
## Community | ||
|
||
Join our [Slack workspace](https://slack.datahubproject.io) for discussions and important announcements. You can also find out more about our upcoming [town hall meetings](docs/townhalls.md) and view past recordings. | ||
|
||
## Adoption | ||
|
||
Here are the companies that have officially adopted DataHub. Please feel free to add yours to the list if we missed it. | ||
|
||
- [ABLY](https://ably.team/) | ||
- [Adevinta](https://www.adevinta.com/) | ||
- [Banksalad](https://www.banksalad.com) | ||
- [Cabify](https://cabify.tech/) | ||
- [ClassDojo](https://www.classdojo.com/) | ||
- [Coursera](https://www.coursera.org/) | ||
- [DefinedCrowd](http://www.definedcrowd.com) | ||
- [DFDS](https://www.dfds.com/) | ||
- [Digital Turbine](https://www.digitalturbine.com/) | ||
- [Expedia Group](http://expedia.com) | ||
- [Experius](https://www.experius.nl) | ||
- [Geotab](https://www.geotab.com) | ||
- [Grofers](https://grofers.com) | ||
- [Haibo Technology](https://www.botech.com.cn) | ||
- [hipages](https://hipages.com.au/) | ||
- [inovex](https://www.inovex.de/) | ||
- [IOMED](https://iomed.health) | ||
- [Klarna](https://www.klarna.com) | ||
- [LinkedIn](http://linkedin.com) | ||
- [Moloco](https://www.moloco.com/en) | ||
- [N26](https://n26brasil.com/) | ||
- [Optum](https://www.optum.com/) | ||
- [Peloton](https://www.onepeloton.com) | ||
- [PITS Global Data Recovery Services](https://www.pitsdatarecovery.net/) | ||
- [Razer](https://www.razer.com) | ||
- [Saxo Bank](https://www.home.saxo) | ||
- [Showroomprive](https://www.showroomprive.com/) | ||
- [SpotHero](https://spothero.com) | ||
- [Stash](https://www.stash.com) | ||
- [Shanghai HuaRui Bank](https://www.shrbank.com) | ||
- [ThoughtWorks](https://www.thoughtworks.com) | ||
- [TypeForm](http://typeform.com) | ||
- [Udemy](https://www.udemy.com/) | ||
- [Uphold](https://uphold.com) | ||
- [Viasat](https://viasat.com) | ||
- [Wikimedia](https://www.wikimedia.org) | ||
- [Wolt](https://wolt.com) | ||
- [Zynga](https://www.zynga.com) | ||
|
||
## Select Articles & Talks | ||
|
||
- [DataHub Blog](https://blog.datahubproject.io/) | ||
- [DataHub YouTube Channel](https://www.youtube.com/channel/UC3qFQC5IiwR5fvWEqi_tJ5w) | ||
- [Optum: Data Mesh via DataHub](https://optum.github.io/blog/2022/03/23/data-mesh-via-datahub/) | ||
- [Saxo Bank: Enabling Data Discovery in Data Mesh](https://medium.com/datahub-project/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f) | ||
- [Bringing The Power Of The DataHub Real-Time Metadata Graph To Everyone At Acryl Data](https://www.dataengineeringpodcast.com/acryl-data-datahub-metadata-graph-episode-230/) | ||
- [DataHub: Popular Metadata Architectures Explained](https://engineering.linkedin.com/blog/2020/datahub-popular-metadata-architectures-explained) | ||
- [Driving DataOps Culture with LinkedIn DataHub](https://www.youtube.com/watch?v=ccsIKK9nVxk) @ [DataOps Unleashed 2021](https://dataopsunleashed.com/#shirshanka-session) | ||
- [The evolution of metadata: LinkedIn’s story](https://speakerdeck.com/shirshanka/the-evolution-of-metadata-linkedins-journey-strata-nyc-2019) @ [Strata Data Conference 2019](https://conferences.oreilly.com/strata/strata-ny-2019.html) | ||
- [Journey of metadata at LinkedIn](https://www.youtube.com/watch?v=OB-O0Y6OYDE) @ [Crunch Data Conference 2019](https://crunchconf.com/2019) | ||
- [DataHub Journey with Expedia Group](https://www.youtube.com/watch?v=ajcRdB22s5o) | ||
- [Data Discoverability at SpotHero](https://www.slideshare.net/MaggieHays/data-discoverability-at-spothero) | ||
- [Data Catalogue — Knowing your data](https://medium.com/albert-franzi/data-catalogue-knowing-your-data-15f7d0724900) | ||
- [DataHub: A Generalized Metadata Search & Discovery Tool](https://engineering.linkedin.com/blog/2019/data-hub) | ||
- [Open sourcing DataHub: LinkedIn’s metadata search and discovery platform](https://engineering.linkedin.com/blog/2020/open-sourcing-datahub--linkedins-metadata-search-and-discovery-p) | ||
- [Emerging Architectures for Modern Data Infrastructure](https://future.com/emerging-architectures-for-modern-data-infrastructure-2020/) | ||
|
||
See the full list [here](docs/links.md). | ||
|
||
## License | ||
|
||
[Apache License 2.0](https://github.com/datahub-project/datahub/blob/master/LICENSE). |
97 changes: 97 additions & 0 deletions
97
docs-website/versioned_docs/version-0.10.4/datahub-frontend/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
--- | ||
title: datahub-frontend | ||
sidebar_label: datahub-frontend | ||
slug: /datahub-frontend | ||
custom_edit_url: >- | ||
https://github.com/datahub-project/datahub/blob/master/datahub-frontend/README.md | ||
--- | ||
|
||
# DataHub Frontend Proxy | ||
|
||
DataHub frontend is a [Play](https://www.playframework.com/) service written in Java. It is served as a mid-tier | ||
between [DataHub GMS](https://github.com/datahub-project/datahub/blob/master/metadata-service) which is the backend service and [DataHub Web](../datahub-web-react/README.md). | ||
|
||
## Pre-requisites | ||
|
||
- You need to have [JDK11](https://openjdk.org/projects/jdk/11/) | ||
installed on your machine to be able to build `DataHub Frontend`. | ||
- You need to have [Chrome](https://www.google.com/chrome/) web browser | ||
installed to be able to build because UI tests have a dependency on `Google Chrome`. | ||
|
||
## Build | ||
|
||
`DataHub Frontend` is already built as part of top level build: | ||
|
||
``` | ||
./gradlew build | ||
``` | ||
|
||
However, if you only want to build `DataHub Frontend` specifically: | ||
|
||
``` | ||
./gradlew :datahub-frontend:dist | ||
``` | ||
|
||
## Dependencies | ||
|
||
Before starting `DataHub Frontend`, you need to make sure that [DataHub GMS](https://github.com/datahub-project/datahub/blob/master/metadata-service) and | ||
all its dependencies have already started and running. | ||
|
||
## Start via Docker image | ||
|
||
Quickest way to try out `DataHub Frontend` is running the [Docker image](https://github.com/datahub-project/datahub/blob/master/docker/datahub-frontend). | ||
|
||
## Start via command line | ||
|
||
If you do modify things and want to try it out quickly without building the Docker image, you can also run | ||
the application directly from command line after a successful [build](#build): | ||
|
||
``` | ||
cd datahub-frontend/run && ./run-local-frontend | ||
``` | ||
|
||
## Checking out DataHub UI | ||
|
||
After starting your application in one of the two ways mentioned above, you can connect to it by typing below | ||
into your favorite web browser: | ||
|
||
``` | ||
http://localhost:9002 | ||
``` | ||
|
||
To be able to sign in, you need to provide your user name. The default account is `datahub`, password `datahub`. | ||
|
||
## Authentication | ||
|
||
DataHub frontend leverages [Java Authentication and Authorization Service (JAAS)](https://docs.oracle.com/javase/7/docs/technotes/guides/security/jaas/JAASRefGuide.html) to perform the authentication. By default we provided a [DummyLoginModule](https://github.com/datahub-project/datahub/blob/master/datahub-frontend/app/security/DummyLoginModule.java) which will accept any username/password combination. You can update [jaas.conf](https://github.com/datahub-project/datahub/blob/master/datahub-frontend/conf/jaas.conf) to match your authentication requirement. For example, use the following config for LDAP-based authentication, | ||
|
||
``` | ||
WHZ-Authentication { | ||
com.sun.security.auth.module.LdapLoginModule sufficient | ||
userProvider="ldaps://<host>:636/dc=<domain>" | ||
authIdentity="{USERNAME}" | ||
userFilter="(&(objectClass=person)(uid={USERNAME}))" | ||
java.naming.security.authentication="simple" | ||
debug="false" | ||
useSSL="true"; | ||
}; | ||
``` | ||
|
||
### Authentication in React | ||
|
||
The React app supports both JAAS as described above and separately OIDC authentication. To learn about configuring OIDC for React, | ||
see the [OIDC in React](../docs/authentication/guides/sso/configure-oidc-react.md) document. | ||
|
||
### API Debugging | ||
|
||
Most DataHub frontend API endpoints are protected using [Play Authentication](https://www.playframework.com/documentation/2.1.0/JavaGuide4), which means it requires authentication information stored in the cookie for the request to go through. This makes debugging using curl difficult. One option is to first make a curl call against the `/authenticate` endpoint and stores the authentication info in a cookie file like this | ||
|
||
``` | ||
curl -c cookie.txt -d '{"username":"datahub", "password":"datahub"}' -H 'Content-Type: application/json' http://localhost:9002/authenticate | ||
``` | ||
|
||
You can then make all subsequent calls using the same cookie file to pass the authentication check. | ||
|
||
``` | ||
curl -b cookie.txt "http://localhost:9001/api/v2/search?type=dataset&input=page" | ||
``` |
Oops, something went wrong.