refactor: Migrate from Pipenv to poetry (#2460)

This moves Pipenv to poetry, and removes Pipfile Pipfile.lock, and other
mentions or Pipenv from the repository.

Changes in addition to a direct migration:
- Appengine
- Removes gcloud appengine related files, as we are now fully on cloud
run via building a docker image (With the exception of index.yaml, which
is for appengine datastore)
- Add a `build.sh` script for building the docker image similar to other
`build.sh` scripts.
- Removed most dependencies in deployment as it's originally only needed
for appengine (see the deleted `gcp/appengine/deploy.sh`)
- Python environment
- For many of the docker images, pipenv installs to the **system**
python dependencies. This is not really ideal, as even in a docker
container, it is good practice to compartmentalise into separate
virtualenvs.
- For the worker image, I have retained this behaviour, and will leave
this migration to a future PR, as it affects a large number of different
images based on it, e.g.:
    - ci
    - cron
    - exporter
    - importer
- For other non worker based images, I have switched to using
virtualenvs, this includes:
    - Appengine/Cloud Run
    - API
    - Debian (already uses virtualenvs)


TODO/looking for suggestions:

The poetry version is defined in every file that it is installed with
`==`, is there a way to set the version number for the whole repository
in one place?

.github/workflows/lint.yaml

@@ -35,15 +35,15 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: '3.11'
-      - name: Install pipenv
-        run: pip install pipenv==2023.12.1
-      - name: Set up pipenv
-        run: pipenv verify && pipenv sync --dev
+      - name: Install poetry
+        run: pip install poetry==1.8.3
+      - name: Set up poetry
+        run: poetry install
       - name: Set up Go
         uses: actions/setup-go@v5
         with:
           go-version: '>=1.22.5'
       - name: Set up terraform
         uses: hashicorp/setup-terraform@v2
       - name: Run pylint and yapf, go vet
-        run: pipenv run ./tools/lint_and_format.sh
+        run: poetry run ./tools/lint_and_format.sh

CONTRIBUTING.md

@@ -40,21 +40,20 @@ You must install:
 1.  [Yapf](https://github.com/google/yapf)
 1.  [Make](https://www.gnu.org/software/make/)
 1.  [Poetry](https://python-poetry.org/)
-1.  [Pipenv](https://pipenv.pypa.io/en/latest/)
-1.  [Google Cloud SDK](https://cloud.google.com/sdk)
-1.  [Hugo](https://gohugo.io/installation/)
-1.  [Node JS](https://nodejs.org/) >= 18.17.x
-1.  [Terraform](https://developer.hashicorp.com/terraform/downloads) >= 1.5 (for infrastructure changes)
+2.  [Google Cloud SDK](https://cloud.google.com/sdk)
+3.  [Hugo](https://gohugo.io/installation/)
+4.  [Node JS](https://nodejs.org/) >= 18.17.x
+5.  [Terraform](https://developer.hashicorp.com/terraform/downloads) >= 1.5 (for infrastructure changes)
 
 Then you can set up the development environment by cloning the OSV repo and
-installing the Pipfile dependencies.
+installing the Poetry dependencies.
 
 ```shell
 git clone https://github.com/google/osv.dev
 cd osv.dev
 git submodule update --init --recursive
-pipenv sync --dev
-pipenv shell
+poetry install
+poetry shell
 ```
 
 ### Running tests

Makefile

@@ -12,14 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-ifeq "$(USE_POETRY)" "true"
-	install-cmd := poetry install
-	run-cmd := poetry run
-else
-	install-cmd := pipenv verify && pipenv sync
-	run-cmd := pipenv run
-endif
-
+install-cmd := poetry install
+run-cmd := poetry run
 
 lib-tests:
 	./run_tests.sh

cloudbuild.yaml

@@ -57,7 +57,7 @@ steps:
     - CLOUD_BUILD=1
     # same as worker
     - DATASTORE_EMULATOR_PORT=8003
-  # importer uses same pipenv as worker, 'pipenv sync' may break if run concurrently
+  # importer uses same poetry as worker, 'poetry install' may break if run concurrently
   waitFor: ['init', 'worker-tests']
 
 - name: 'gcr.io/oss-vdb/ci'
@@ -67,7 +67,7 @@ steps:
   env:
     # same as worker/importer
     - DATASTORE_EMULATOR_PORT=8003
-  # alias uses same pipenv as worker, 'pipenv sync' may break if run concurrently
+  # alias uses same poetry as worker, 'poetry install' may break if run concurrently
   waitFor: ['init', 'importer-tests']
 
 - name: 'gcr.io/oss-vdb/ci'

docker/alias/run_tests.sh

@@ -15,13 +15,5 @@
 
 cd ../worker
 
-if [ "$USE_POETRY" == "true" ]
-then
-  poetry install
-  poetry run python ../alias/alias_computation_test.py
-  exit 0
-fi
-
-export PIPENV_IGNORE_VIRTUALENVS=1
-pipenv sync
-pipenv run python ../alias/alias_computation_test.py
+poetry install
+poetry run python ../alias/alias_computation_test.py

docker/deployment/Dockerfile

@@ -1,15 +1,8 @@
 FROM ubuntu:22.04
 
 RUN apt-get update && \
-    apt-get install -y curl jq nodejs npm python3-pip
+    apt-get install -y curl jq
 
 RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
     curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
     apt-get update && apt-get install -y google-cloud-sdk
-
-RUN pip3 install pipenv==2023.12.1
-# Install the latest npm version 8 release (last release is in Feb 2023)
-# This will take precedence over the system installed version on the PATH
-RUN npm install -g npm@8
-
-COPY --from=gcr.io/oss-vdb/ci /root/go/bin/hugo /usr/local/bin/hugo

docker/importer/run_tests.sh

@@ -15,14 +15,5 @@
 
 cd ../worker
 
-if [ "$USE_POETRY" == "true" ]
-then
-  poetry install
-  poetry run python ../importer/importer_test.py
-  exit 0
-fi
-
-export PIPENV_IGNORE_VIRTUALENVS=1
-pipenv verify
-pipenv sync
-pipenv run python ../importer/importer_test.py
+poetry install
+poetry run python ../importer/importer_test.py

docker/worker-base/Dockerfile

@@ -17,16 +17,16 @@ FROM ubuntu:20.04
 ENV DEBIAN_FRONTEND noninteractive
 RUN apt-get update && apt-get upgrade -y && \
     apt-get install -y \
-        apt-transport-https \
-        build-essential \
-        ca-certificates \
-        curl \
-        git \
-        gnupg-agent \
-        zlib1g-dev \
-        libffi-dev \
-        libssl-dev \
-        software-properties-common
+    apt-transport-https \
+    build-essential \
+    ca-certificates \
+    curl \
+    git \
+    gnupg-agent \
+    zlib1g-dev \
+    libffi-dev \
+    libssl-dev \
+    software-properties-common
 
 ARG PYTHON_VERSION=3.11.4
 RUN curl -sS https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz | tar -C /tmp -xzv && \
@@ -35,15 +35,24 @@ RUN curl -sS https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON
     make -j && \
     make install && \
     rm -rf /tmp/Python-${PYTHON_VERSION}
-RUN pip3 install pipenv==2023.12.1
+
+# Setup Poetry in its own virtual environment.
+# So when poetry changes the system dependencies, it doesn't mess with its own dependencies
+# when managing our dependencies.
+# See: https://python-poetry.org/docs/#installation 
+ENV POETRY_HOME "/opt/poetry"
+RUN python3 -m venv $POETRY_HOME
+RUN $POETRY_HOME/bin/pip install poetry==1.8.3
+# Link in PATH to make it possible to directly call poetry
+RUN ln -s $POETRY_HOME/bin/poetry /usr/local/bin/poetry
 
 # Install Docker.
 # Pin the version to an older one due to gVisor incompatibilities.
 # See https://github.com/google/osv.dev/issues/1019#issuecomment-1427418758
 ARG DOCKER_VERSION=5:20.10.22~3-0~ubuntu-bionic
 RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/trusted.gpg.d/docker.gpg && \
     add-apt-repository \
-        "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" && \
+    "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" && \
     apt-get install -y docker-ce=$DOCKER_VERSION docker-ce-cli=$DOCKER_VERSION && \
     apt-mark hold docker-ce docker-ce-cli  # Prevent subsequent upgrades from bumping the version.
 

docker/worker/Dockerfile

@@ -19,12 +19,14 @@ RUN apt-get update && apt-get upgrade -y
 RUN mkdir /work && mkdir -p /env/docker/worker
 VOLUME /var/lib/docker
 
-# Replicate project structure to make relative editable pipenv dependency work.
+# Replicate project structure to make relative editable poetry dependency work.
 # TODO(ochang): Just copy the entire project (needs a clean checkout).
-COPY docker/worker/Pipfile* /env/docker/worker/
-COPY setup.py Pipfile* README.md /env/
+COPY docker/worker/poetry.lock docker/worker/pyproject.toml /env/docker/worker/
+COPY setup.py poetry.lock pyproject.toml README.md /env/
 COPY osv /env/osv
-RUN cd /env/docker/worker && pipenv install --deploy --system
+
+# Set virtualenv creation to false to install globally
+RUN cd /env/docker/worker && POETRY_VIRTUALENVS_CREATE=false poetry install
 
 COPY docker/worker/oss_fuzz.py docker/worker/worker.py /usr/local/bin/
 RUN chmod 755 /usr/local/bin/worker.py