Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Updating the dev setup #267

Merged
merged 6 commits into from
Nov 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 32 additions & 16 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -20,26 +20,29 @@ endif
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1

DOCKER_COMPOSE=docker-compose --env-file=${ENV_FILE}
DOCKER_COMPOSE=docker compose --env-file=${ENV_FILE}
# tweak some config to avoid port conflicts
DOCKER_COMPOSE_TEST=COMPOSE_PROJECT_NAME=test_taxonomy NEO4J_ADMIN_EXPOSE=127.0.0.1:7475 NEO4J_BOLT_EXPOSE=127.0.0.1:7688 docker-compose --env-file=${ENV_FILE}
DOCKER_COMPOSE_TEST=COMPOSE_PROJECT_NAME=test_taxonomy NEO4J_ADMIN_EXPOSE=127.0.0.1:7475 NEO4J_BOLT_EXPOSE=127.0.0.1:7688 docker compose --env-file=${ENV_FILE}


.PHONY: tests

#------------#
# dev setup #
#------------#

build:
build: ## Build docker images
@echo "🍜 Building docker images"
${DOCKER_COMPOSE} build
@echo "🍜 Project setup done"

up:
up: ## Run the project
@echo "🍜 Running project (ctrl+C to stop)"
@echo "🍜 The React app will be available on http://ui.taxonomy.localhost:8091"
@echo "🍜 The API will be exposed on http://api.taxonomy.localhost:8091"
@echo "🍜 The Neo4j admin console will be available on http://localhost:7474/browser/"
${DOCKER_COMPOSE} up

dev: build up
dev: build up ## Build and run the project


#-----------#
Expand All @@ -48,28 +51,28 @@ dev: build up


# lint code
lint: backend_lint frontend_lint
lint: backend_lint frontend_lint ## Run all linters

backend_lint:
backend_lint: ## Run lint on backend code
@echo "🍜 Linting python code"
${DOCKER_COMPOSE} run --rm taxonomy_api isort .
${DOCKER_COMPOSE} run --rm taxonomy_api black .

frontend_lint:
frontend_lint: ## Run lint on frontend code
@echo "🍜 Linting react code"
${DOCKER_COMPOSE} run --rm taxonomy_node npx prettier -w src/


# check code quality
quality: backend_quality frontend_quality
quality: backend_quality frontend_quality ## Run all quality checks

backend_quality:
backend_quality: ## Run quality checks on backend code
@echo "🍜 Quality checks python"
${DOCKER_COMPOSE} run --rm taxonomy_api flake8 .
${DOCKER_COMPOSE} run --rm taxonomy_api isort --check-only .
${DOCKER_COMPOSE} run --rm taxonomy_api black --check .

frontend_quality:
frontend_quality: ## Run quality checks on frontend code
@echo "🍜 Quality checks JS"
${DOCKER_COMPOSE} run --rm taxonomy_node npx prettier -c src/
${DOCKER_COMPOSE} run --rm -e CI=true taxonomy_node npm run build
Expand All @@ -78,22 +81,35 @@ frontend_quality:



tests: backend_tests
tests: backend_tests ## Run all tests

backend_tests:
backend_tests: ## Run python tests
@echo "🍜 Running python tests"
${DOCKER_COMPOSE_TEST} up -d neo4j
${DOCKER_COMPOSE_TEST} run --rm taxonomy_api pytest /parser /parser
${DOCKER_COMPOSE_TEST} run --rm taxonomy_api pytest /code/tests
${DOCKER_COMPOSE_TEST} stop neo4j

checks: quality tests
checks: quality tests ## Run all checks (quality + tests)


#------------#
# production #
#------------#

create_external_volumes:
create_external_volumes: ## Create external volumes (production only)
@echo "🍜 Creating external volumes (production only) …"
docker volume create ${COMPOSE_PROJECT_NAME}_neo4j-data


#------------#
# utils #
#------------#

.DEFAULT_GOAL := help

.PHONY: build up dev lint backend_lint frontend_lint quality backend_quality frontend_quality tests backend_tests checks create_external_volumes help

# this command will grep targets and the following ## comment will be used as description
help: ## Description of the available commands
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m\033[0m\n"} /^[$$()% a-zA-Z_-]+:.*?##/ { printf " \033[36m%-30s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
34 changes: 18 additions & 16 deletions doc/how-to-guides/develop-with-neo4j.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,23 @@
# Developing with Neo4j

We provide a simple docker-compose for neo4j to have it available on your localhost.
We provide a simple docker compose for neo4j to have it available on your localhost.

> **Note**: this may be a temporary setting, as later on we may have all running in docker.

## Launching the database

To start the database, simply type `docker-compose up neo4j` at root of project.
To start the database, simply type `docker compose up neo4j` at root of project.

The neo4j database will be available at `localhost:7687` using bolt protocol.

Moreover you can access an admin tool at http://localhost:7474/.

To stop the database, simply use *Ctrl+C* in the console where neo4j is running, or type `docker-compose stop` in another terminal.
To stop the database, simply use _Ctrl+C_ in the console where neo4j is running, or type `docker compose stop` in another terminal.

If you want to **destroy** neo4j data, simply use `docker-compose rm -sf neo4j` and then `docker volume rm taxonomy-editor_neo4j-data`[^vol_name]
If you want to **destroy** neo4j data, simply use `docker compose rm -sf neo4j` and then `docker volume rm taxonomy-editor_neo4j-data`[^vol_name]

[^vol_name]: (it may have another name, look at what `docker volume list|grep neo4j-data`) gives you.


## Importing data into Neo4J database

### Import sample data
Expand All @@ -27,40 +26,43 @@ We have a `backend/sample/test-neo4j.json` file to load sample data in the datab

To load it:

* ensure your neo4j database is launched:
- ensure your neo4j database is launched:
```bash
docker-compose up -d neo4j
docker compose up -d neo4j
```
* create you local env if you do not have one yet:
- create you local env if you do not have one yet:
```bash
python3 -m venv .venv
```
* activate your local env, if needed:
- activate your local env, if needed:
```bash
. .venv/bin/activate
```
* if needed, install neo4j driver:
- if needed, install neo4j driver:
```bash
pip install neo4j
```
* finally load the data:
- finally load the data:

```bash
cd backend/sample
python3 load.py --reset test-neo4j.json
```

note that if you want to see more options, you can use `--help`
* enjoy it in the neo4j browser at http://localhost:7474
(Hint: After connecting the database, on the left click on the "*" node types to visualize a beautiful graph of the loaded taxonomy)

- enjoy it in the neo4j browser at http://localhost:7474
(Hint: After connecting the database, on the left click on the "\*" node types to visualize a beautiful graph of the loaded taxonomy)

### From a generic file

Note that if you want to import some other data, (eg for a specific test of Neo4J capabilities),
Note that if you want to import some other data, (eg for a specific test of Neo4J capabilities),
the file needs to be placed inside the `neo4j/import` folder.

Then see [neo4j documentation on importing data](https://neo4j.com/docs/operations-manual/current/docker/operations/#docker-neo4jlabs-pluginsneo4j.com) but you can run the simpler command:

```
docker-compose exec neo4j neo4j-admin import ...
docker compose exec neo4j neo4j-admin import ...
```

> **Note**: it haven't been tested so far, and some tweaks might be necessary.
Expand All @@ -72,4 +74,4 @@ If you are using a local python environment, the database will be accessible at
## Use it in another docker service

If you run another docker service and want to access neo4j, it will be accessible at `neo4j:7687`
provided your container runs in the same network (should default to `taxonomy-editor_default`)
provided your container runs in the same network (should default to `taxonomy-editor_default`)
15 changes: 10 additions & 5 deletions doc/introduction/setup-dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,23 +12,24 @@ Docker provides an isolated environment, very close to a Virtual Machine. This e
### Prerequisites

- [Install Docker CE](https://docs.docker.com/install/#supported-platforms)
> If you run e.g. Debian, don't forget to add your user to the `docker` group!
> If you run e.g. Debian, don't forget to add your user to the `docker` group!
- [Install Docker Compose](https://docs.docker.com/compose/install/)
- [Enable command-line completion](https://docs.docker.com/compose/completion/)

### Setup

The docker-compose for Taxonomy Editor is designed in such a way that all development servers can be built rapidly.

After first checkout, or on requirements or important element changes, run the following commands in the root directory:

```bash
DOCKER_BUILDKIT=1 docker-compose build
make build
```

And every time, to get the server running, just use:

```bash
docker-compose up
make up
```

You should have a running environment.
Expand All @@ -42,7 +43,7 @@ You can also access the Neo4j Admin Console at `http://localhost:7474/browser/`
If you modify any file in the React App, the changes will be taken into account instantly.
However, this feature is not compatible with Windows systems. In order to use live reload on a Windows machine, you will need to install and use [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install). This will allow you to run the development server in a Linux environment, and the live reload feature will work as expected.

If you modify any files related to the Python API, you need to restart the `taxonomy_api` container in Docker: `docker-compose restart taxonomy_api`
If you modify any files related to the Python API, you need to restart the `taxonomy_api` container in Docker: `docker compose restart taxonomy_api`

You can modify the `.env` file to fit your needs, but you should not commit any changes that are not defaults to the project.
Notably, if you use a `uid` which is not 1000, you should personalize the `USER_UID` variable.
Expand All @@ -62,7 +63,7 @@ That's it! Now, you'll be able to view any created PR's in your fork of Taxonomy
### Importing some test data

```bash
docker-compose run --rm taxonomy_api sample/load.py sample/test-neo4j.json
docker compose run --rm taxonomy_api sample/load.py sample/test-neo4j.json
```

Add the `--reset` switch if you already have data you want to remove.
Expand Down Expand Up @@ -90,6 +91,7 @@ Create a virtualenv with python:
```bash
python3 -m venv .venv
```

and activate it:

```bash
Expand All @@ -111,6 +113,7 @@ uvicorn editor.api:app --host 127.0.0.1 --port 8080
Open http://127.0.0.1:8080 in your browser to check out the API.

You will see the following:

```
{"message": "Hello user! Tip: open /docs or /redoc for documentation"}
```
Expand All @@ -135,6 +138,7 @@ Run the server with:
```bash
REACT_APP_API_URL="http://localhost:8080/" npm start
```

(or)

```bash
Expand All @@ -151,6 +155,7 @@ To import some test data, in your Python virtualenv (don't forget to activate it
```bash
python3 sample/load.py sample/test-neo4j.json
```

Add the `--reset` switch if you already have data you want to remove.

Going into the Neo4j Admin Console, you should be able to see the data.
2 changes: 1 addition & 1 deletion parser/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ quality:
black --check .

tests:
cd .. && docker-compose up -d neo4j
cd .. && docker compose up -d neo4j
pytest .
# we do not shutdown neo4j

Expand Down