Skip to content

Latest commit

 

History

History
59 lines (48 loc) · 2.59 KB

TESTING.md

File metadata and controls

59 lines (48 loc) · 2.59 KB

Testing

Testing the file ingestor locally on your machine from src

In order to test the ingestor locally, you need to have an instance of kafka running locally and proper instance of User Office and SciCat Backend v4.x accessible.

Create your own local copy of the configuration file config.json.local.test.src from the template config.json.local.test.src,template.
Add in the information for your local User Office and SciCat instance that you are going to use for testing.
THis file assumes that you are running a dockerize kafka instance as it is configured in the docker compose file available under the docker folder.

Run local kafka containers

    > cd docker  
    > docker-compose -f docker-compose-kafka-with-ui.yml up  

This command will not release the terminal, so we can see all the logging from the containers.
If we want to run it in the background, add the -d option:

    > docker-compose -f docker-compose-kafka-with-ui.yml -d up  

old setup
Create and run the generator container which simulate the filewriter end message to the kafka topic:

    > docker run -d 
        --name scicat-filewriter-generator 
        -v ./sfi_generator_config_docker_local.json:/app/sfi_generator_config.json 
        --network=host 
        scicat-filewriter-ingest-generator:latest

Start the jupyter lab with the generator notebook, which allows to simulate the filewriter done writing message.

    > cd generator
    > micromamba run -n sfi-gen jupyter lab

This assume that the micromamba environment sfi-gen has already been created and is available in your system. To create the sfi-gen under the micromamba tool, use the following command:

    > micromamba env create -n sfi-gen -f requirements_sfi_gen.yaml

The following command will start the ingestor. Make sure that you select the correct configuration file

    > micromamba run -n sfi python online_ingestor.py --config-file configs/config.json.local.test.src -v --debug DEBUG

This command assumes that you have the correct environment setup. To create the sfi environment, run this command in the command line

    > micromamba env create -n sfi -f requirements_sfi.yaml

Now you are ready to generate messages that will be ingested. Open a new terminal and run the following command anytime you want to simulate a filewriter message:

    > docker start scicat-filewriter-ingest-generator