Installing python dependencies in a virtual environment
# Creation of the virtual environment
python -m venv ./bigd
And then you can mount this environment with:
# when using fish
source ./bigd/bin/activate.fish
# or other (multiple activation scripts are in this folder)
source ./bigd/bin/activate
# Installation of dependencies
pip install -r requirements.txt
To start the spark cluster, use docker-compose (in Mac this comes out of the box with Docker Desktop, for others refer to the Install Instructions)
# verify whether docker-compose is installed already, else install it
docker-compose -v
# to start all containers
docker-compose up
# or to just start a dedicated container by name
docker-compose up spark