refactor: shared dependencies unification (amundsen-io#1163)

* unify_deps | 🎉 Initial commit. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | ♻️ Refactoring code. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🚨 Removing linter warnings. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🚨 Removing linter warnings. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🚨 Removing linter warnings. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | ✅ Adding tests. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | ⬆️ Upgrading dependencies. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | ✨ Introducing new features. (run tests when reqs change) Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | ♻️ Refactoring code. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 💡 Documenting source code. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 📝 Writing docs. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 💄 Updating the UI and style files. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | ⬆️ Upgrading dependencies. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | ♻️ Refactoring code. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 💡 Documenting source code. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🚨 Removing linter warnings. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 👌 Updating code due to code review changes. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🎉 Initial commit. Signed-off-by: mgorsk1 <[email protected]> * unify_deps | 🐛 Fixing a bug. Signed-off-by: mgorsk1 <[email protected]> Co-authored-by: Dorian Johnson <[email protected]>
DataChefHQ · Jun 30, 2022 · 4165e11 · 4165e11
1 parent b6f3a59
commit 4165e11
Show file tree

Hide file tree

Showing 59 changed files with 309 additions and 397 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,5 @@
+**
+!frontend
+!metadata
+!search
+!requirements*txt
diff --git a/.github/workflows/common_pull_request.yml b/.github/workflows/common_pull_request.yml
@@ -3,6 +3,7 @@ on:
   pull_request:
     paths:
       - "common/**"
+      - "requirements*txt"
 jobs:
   test-unit-python:
     runs-on: ubuntu-18.04
@@ -17,7 +18,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
-        run: pip3 install -r requirements.txt && pip3 install codecov && pip install .
+        run: make install_deps && pip3 uninstall -y amundsen-common
         working-directory: ./common
       - name: Run python unit tests
         run: make test

diff --git a/.github/workflows/databuilder_pull_request.yml b/.github/workflows/databuilder_pull_request.yml
@@ -3,6 +3,7 @@ on:
   pull_request:
     paths:
       - "databuilder/**"
+      - "requirements-dev.txt"
 jobs:
   test-unit-python:
     runs-on: ubuntu-18.04
@@ -17,7 +18,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
-        run: pip3 install ../common && pip3 install -r requirements.txt && pip3 install .[all] && pip3 install codecov
+        run: make install_deps && pip3 uninstall -y amundsen-common && pip3 install -e ../common
         working-directory: ./databuilder
       - name: Run python unit tests
         run: make test

diff --git a/.github/workflows/frontend_pull_request.yml b/.github/workflows/frontend_pull_request.yml
@@ -3,6 +3,7 @@ on:
   pull_request:
     paths:
       - "frontend/**"
+      - "requirements*txt"
 jobs:
   test-unit-python:
     runs-on: ubuntu-18.04
@@ -17,7 +18,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
-        run: pip3 install -r requirements.txt && pip3 install .[all] && pip3 install codecov
+        run: make install_deps && pip3 uninstall -y amundsen-common && pip3 install -e ../common
         working-directory: ./frontend
       - name: Run python unit tests
         run: make test

diff --git a/.github/workflows/metadata_pull_request.yml b/.github/workflows/metadata_pull_request.yml
@@ -3,6 +3,7 @@ on:
   pull_request:
     paths:
       - "metadata/**"
+      - "requirements*txt"
 jobs:
   test-unit-python:
     runs-on: ubuntu-18.04
@@ -17,7 +18,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
-        run: pip3 install ../common && pip3 install -r requirements.txt && pip3 install .[all] && pip3 install codecov && pip install -e '.[atlas,rds]'
+        run: make install_deps && pip3 uninstall -y amundsen-common && pip3 install -e ../common
         working-directory: ./metadata
       - name: Run python unit tests
         run: make test

diff --git a/.github/workflows/search_pull_request.yml b/.github/workflows/search_pull_request.yml
@@ -3,6 +3,7 @@ on:
   pull_request:
     paths:
       - "search/**"
+      - "requirements*txt"
 jobs:
   test-unit-python:
     runs-on: ubuntu-18.04
@@ -17,7 +18,7 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
-        run: pip3 install -r requirements.txt && pip3 install . && pip3 install codecov
+        run: make install_deps && pip3 uninstall -y amundsen-common && pip3 install -e ../common
         working-directory: ./search
       - name: Run python unit tests
         run: make test

diff --git a/Dockerfile.frontend.local b/Dockerfile.frontend.local
@@ -0,0 +1,29 @@
+ARG METADATASERVICE_BASE
+ARG SEARCHSERVICE_BASE
+
+FROM node:12-slim as node-stage
+WORKDIR /app/amundsen_application/static
+
+COPY ./frontend/amundsen_application/static/package.json /app/amundsen_application/static/package.json
+COPY ./frontend/amundsen_application/static/package-lock.json /app/amundsen_application/static/package-lock.json
+RUN npm install
+
+COPY ./frontend/amundsen_application/static/ /app/amundsen_application/static/
+RUN npm rebuild node-sass
+RUN npm run dev-build
+
+COPY ./frontend /app
+
+FROM python:3.7-slim
+WORKDIR /app
+
+COPY ./frontend /app
+COPY requirements-dev.txt /app/requirements-dev.txt
+COPY requirements-common.txt /app/requirements-common.txt
+
+COPY --from=node-stage /app /app
+
+RUN pip3 install -e .
+
+ENTRYPOINT [ "python3" ]
+CMD [ "amundsen_application/wsgi.py" ]
diff --git a/frontend/public.Dockerfile → Dockerfile.frontend.public b/frontend/public.Dockerfile → Dockerfile.frontend.public
@@ -1,22 +1,19 @@
 FROM node:12-slim as node-stage
 WORKDIR /app/amundsen_application/static
 
-COPY amundsen_application/static/package.json /app/amundsen_application/static/package.json
-COPY amundsen_application/static/package-lock.json /app/amundsen_application/static/package-lock.json
+COPY ./frontend/amundsen_application/static/package.json /app/amundsen_application/static/package.json
+COPY ./frontend/amundsen_application/static/package-lock.json /app/amundsen_application/static/package-lock.json
 RUN npm install
 
-COPY amundsen_application/static /app/amundsen_application/static
+COPY ./frontend/amundsen_application/static /app/amundsen_application/static
 RUN npm run build
 
 FROM python:3.7-slim as base
 WORKDIR /app
 RUN pip3 install gunicorn
 
-COPY requirements.txt /app/requirements.txt
-RUN pip3 install -r requirements.txt
-
 COPY --from=node-stage /app /app
-COPY . /app
+COPY ./frontend /app
 RUN pip3 install -e .
 
 CMD [ "python3",  "amundsen_application/wsgi.py" ]

diff --git a/metadata/public.Dockerfile → Dockerfile.metadata.public b/metadata/public.Dockerfile → Dockerfile.metadata.public
@@ -6,17 +6,17 @@ RUN apt-get update
 RUN apt-get upgrade -y
 RUN apt-get install git -y
 
-COPY requirements.txt /app/requirements.txt
-RUN pip3 install -r requirements.txt
-
-COPY . /app
+COPY ./metadata/ /app
+COPY requirements-dev.txt /app/requirements-dev.txt
+COPY requirements-common.txt /app/requirements-common.txt
 
 CMD [ "python3", "metadata_service/metadata_wsgi.py" ]
 
 FROM base as oidc-release
 
-RUN pip3 install .[oidc]
-RUN python3 setup.py install
+RUN  pip3 install -e .&& \
+     pip3 install -e .[oidc]
+
 ENV FLASK_APP_MODULE_NAME flaskoidc
 ENV FLASK_APP_CLASS_NAME FlaskOIDC
 ENV FLASK_OIDC_WHITELISTED_ENDPOINTS status,healthcheck,health
@@ -28,4 +28,4 @@ ENV SQLALCHEMY_DATABASE_URI sqlite:///sessions.db
 # You will also need to mount a volume for the clients_secrets.json file.
 
 FROM base as release
-RUN python3 setup.py install
+RUN pip3 install -e .
diff --git a/Dockerfile.search.public b/Dockerfile.search.public
@@ -0,0 +1,10 @@
+FROM python:3.7-slim
+WORKDIR /app
+RUN pip3 install gunicorn
+
+COPY ./search/ /app/
+COPY requirements-dev.txt /app/requirements-dev.txt
+COPY requirements-common.txt /app/requirements-common.txt
+RUN pip3 install -e .
+
+CMD [ "python3", "search_service/search_wsgi.py" ]
diff --git a/common/MANIFEST.in b/common/MANIFEST.in
@@ -0,0 +1,3 @@
+include requirements.txt
+
+global-include requirements-*.txt
diff --git a/common/Makefile b/common/Makefile
@@ -17,3 +17,7 @@ mypy:
 
 
 test: test_unit lint mypy
+
+.PHONY: install_deps
+install_deps:
+	pip3 install -e ".[all]"
diff --git a/common/requirements-common.txt b/common/requirements-common.txt
@@ -0,0 +1 @@
+../requirements-common.txt
diff --git a/common/requirements-dev.txt b/common/requirements-dev.txt
@@ -0,0 +1 @@
+../requirements-dev.txt
diff --git a/common/requirements.txt b/common/requirements.txt
diff --git a/common/setup.cfg b/common/setup.cfg
@@ -1,6 +1,6 @@
 [flake8]
 format = pylint
-exclude = .svc,CVS,.bzr,.hg,.git,__pycache__,venv
+exclude = .svc,CVS,.bzr,.hg,.git,__pycache__,venv,.venv
 max-complexity = 10
 max-line-length = 120
 

diff --git a/common/setup.py b/common/setup.py
@@ -1,41 +1,39 @@
 # Copyright Contributors to the Amundsen project.
 # SPDX-License-Identifier: Apache-2.0
+import os
 
 from setuptools import find_packages, setup
 
 __version__ = '0.13.0'
 
+requirements_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements-common.txt')
+with open(requirements_path) as requirements_file:
+    requirements_common = requirements_file.readlines()
+
+requirements_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements-dev.txt')
+with open(requirements_path) as requirements_file:
+    requirements_dev = requirements_file.readlines()
+
+# avoid circular references
+requirements_common = [r for r in requirements_common if not r.startswith('amundsen-common')]
+
+all_deps = requirements_common + requirements_dev
+
 setup(
     name='amundsen-common',
     version=__version__,
     description='Common code library for Amundsen',
     long_description=open('README.md').read(),
     long_description_content_type='text/markdown',
-    url='https://github.com/amundsen-io/amundsencommon',
+    url='https://github.com/amundsen-io/amundsen/tree/main/common',
     maintainer='Amundsen TSC',
     maintainer_email='[email protected]',
     packages=find_packages(exclude=['tests*']),
-    install_requires=[
-        # Packages in here should rarely be pinned. This is because these
-        # packages (at the specified version) are required for project
-        # consuming this library. By pinning to a specific version you are the
-        # number of projects that can consume this or forcing them to
-        # upgrade/downgrade any dependencies pinned here in their project.
-        #
-        # Generally packages listed here are pinned to a major version range.
-        #
-        # e.g.
-        # Python FooBar package for foobaring
-        # pyfoobar>=1.0, <2.0
-        #
-        # This will allow for any consuming projects to use this library as
-        # long as they have a version of pyfoobar equal to or greater than 1.x
-        # and less than 2.x installed.
-        'Flask>=1.0.2',
-        'attrs>=19.0.0',
-        'marshmallow>=3.0',
-        'marshmallow3-annotations>=1.0.0'
-    ],
+    install_requires=requirements_common,
+    extras_require={
+        'all': all_deps,
+        'dev': requirements_dev
+    },
     python_requires=">=3.6",
     package_data={'amundsen_common': ['py.typed']},
 )
diff --git a/databuilder/MANIFEST.in b/databuilder/MANIFEST.in
@@ -1 +1,3 @@
 include requirements.txt
+
+global-include requirements-dev.txt
diff --git a/databuilder/Makefile b/databuilder/Makefile
@@ -24,3 +24,7 @@ isort_check:
 
 .PHONY: test
 test: test_unit lint mypy isort_check
+
+.PHONY: install_deps
+install_deps:
+	pip3 install -e ".[all]"
diff --git a/databuilder/databuilder/models/feature/feature_generation_code.py b/databuilder/databuilder/models/feature/feature_generation_code.py
@@ -65,7 +65,7 @@ def _create_node_iterator(self) -> Iterator[GraphNode]:
             FeatureGenerationCode.TEXT_ATTR: self.text,
         }
         if self.last_executed_timestamp:
-            attrs[FeatureGenerationCode.LAST_EXECUTED_TIMESTAMP_ATTR] = self.last_executed_timestamp
+            attrs[FeatureGenerationCode.LAST_EXECUTED_TIMESTAMP_ATTR] = self.last_executed_timestamp  # type: ignore
 
         yield GraphNode(
             key=self._get_generation_code_key(),

diff --git a/databuilder/databuilder/models/feature/feature_metadata.py b/databuilder/databuilder/models/feature/feature_metadata.py
@@ -132,7 +132,7 @@ def _create_next_node(self) -> Iterator[GraphNode]:
 
         if self.description:
             yield self.description.get_node(
-                node_key=self.description.get_description_default_key(
+                node_key=self.description.get_description_default_key(  # type: ignore
                     start_key=self._get_feature_key()),
             )
 

diff --git a/databuilder/requirements-dev.txt b/databuilder/requirements-dev.txt
@@ -0,0 +1 @@
+../requirements-dev.txt
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		include requirements.txt

		global-include requirements-*.txt
Original file line number	Diff line number	Diff line change
		@@ -1 +1,3 @@
		include requirements.txt

		global-include requirements-dev.txt