Skip to content

Commit

Permalink
add administrative files
Browse files Browse the repository at this point in the history
  • Loading branch information
jeromeleonard committed Sep 6, 2022
1 parent b340273 commit 7543e85
Show file tree
Hide file tree
Showing 10 changed files with 284 additions and 31 deletions.
1 change: 1 addition & 0 deletions CNAME
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
docs.thehive-project.org
3 changes: 2 additions & 1 deletion docs/cortex/.pages
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ nav:
- 'Download': 'download'
- Installation & configuration: 'installation-and-configuration'
- 'user-guides'
- 'operations'
- 'operations'
- 'API': 'api'
2 changes: 1 addition & 1 deletion docs/cortex/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Please [open an issue on GitHub](https://github.com/TheHive-Project/Cortex/issue
If you need to contact the Project's team, send an email to <[email protected]>.

!!! Note
- If you have problems with [Cortex4py](https://github.com/TheHive-Project/Cortex4y), please [open an issue on its dedicated repository](https://github.com/TheHive-Project/Cortex4py/issues/new).
- If you have problems with [Cortex4py](https://github.com/TheHive-Project/Cortex4py), please [open an issue on its dedicated repository](https://github.com/TheHive-Project/Cortex4py/issues/new).
- If you encounter an issue with Cortex or would like to request a Cortex-related feature, please [open an issue on its dedicated GitHub repository](https://github.com/TheHive-Project/Cortex/issues/new).
- If you have troubles with a Cortex analyzer or would like to request a new one or an improvement to an existing analyzer, please open an issue on the [analyzers' dedicated GitHub repository](https://github.com/TheHive-Project/cortex-analyzers/issues/new).

Expand Down
5 changes: 3 additions & 2 deletions docs/cortex/installation-and-configuration/.pages
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
nav:
- index.md
- Step by step guide: step-by-step-guide.md
- Set up a secret key: secret-key.md
- Set up a secret key: secret.md
- Authentication options: authentication.md
- Database configuration: database.md
- Configure Analyzers & Responders: analyzers-responders.md
- Docker parameters: docker.md
- Using Cortex behind a proxy: proxy-settings.md
- SSL configuration: ssl.md
- Advanced configuration: advanced-configuration.md
- Advanced configuration: advanced-configuration.md
- 'Run Cortex with Docker': run-cortex-with-docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ To run Analyzers&Responders with Docker images, Cortex should be able have acces
[..]
```


## Store & run programs on the host

### Additionnal packages
Expand Down
3 changes: 2 additions & 1 deletion docs/cortex/installation-and-configuration/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@
- `docker.container.memorySwappiness`: (integer) Tune a container’s memory swappiness behavior. Accepts an integer between 0 and 100
- `docker.container.networkMode`: (string) name of the network
- `docker.container.privileged`: (boolean) Give extended privileges to this container

- `job.directory`: (string) Folder used by Cortex binary inside the container to share input and output data of Analyzers & Responders
- `job.dockerDirectory` = (string) Folder on the host used by Analyzers & Responders to share input and output data with Cortex

## Dockerized analyzers / responders

Expand Down
70 changes: 45 additions & 25 deletions docs/cortex/installation-and-configuration/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,41 +4,33 @@
## Overview
Cortex relies on Elasticsearch to store its data. A basic setup to install Elasticsearch, then Cortex on a standalone and dedicated server (physical or virtual).

Before starting the install Cortex, this is important to know how _Analyzers_ and _Responders_ will be managed and run. 2 solutions are available to run them:

### Run the _Analyzers_ & _Responders_ locally
The programs are downloaded and installed on the system running Cortex.
## Hardware requirements
Hardware requirements depends on the usage of the system. We recommend starting with dedicated resources:

There are many disadvantages with this option:
* :fontawesome-solid-microchip: 8 vCPU
* :fontawesome-solid-memory: 16 GB of RAM

* Some public _Analyzers_ or _Responders_, or you own custom program might required specific applications installed on the system,
* All of the programs published are written in Python and come with dependancies. To run successfully, the dependancies of all programs should be installed on the same operating system ; so there is a high risk of incompatibilities (some program might require a specific version of a librarie with the latest is also required by another one)
* The goal of _Analyzers_ is to extract or gather information or intelligence about observables ; and some of them might be malicious. Depending on the analysis, like a code analysis, you might want to ensure the Analyzer has not been compromised - and the host - by the observable itself
* You might want to ensure that when you run an _Analyzer_, there is no question about the integrity of its programs
* Updating them might be a pain regarding Operating System used and dependancies

### Run the _Analyzers_ & _Responders_ with Docker
_Analyzers_ & _Responders_ we publish also have their own [Docker images](https://hub.docker.com/search?q=cortexneurons).
## Operating systems
Cortex has been tested and is supported on the following operating systems:

There are several benefits to use Docker images of _Analyzers_ & _Responders_.
- :material-ubuntu: Ubuntu 20.04 LTS
- :material-debian: Debian 11
- :material-redhat: RHEL 8
- :material-fedora: Fedora 35

* No need to worry about applications required or libraries, it just work
* When requested, Cortex downloads the docker image of a program and instanciate a container running the program. When finished, the container is trashed and a new one is created the next time. No need to worry about the integrity of the program
* This is simple to use and maintain

!!! Tip "This is the recommended option. It requires installing Docker engine as well."

This is not an exclusive choice, both solutions can be used by the same instance of Cortex.
## Installation Guide

!!! Tip "Too much in a hurry to read ? "

## Hardware requirements
Hardware requirements depends on the usage of the system. We recommend starting with dedicated resources:
If you are using [one of the supported](#operating-systems) operating systems, use our all-in-one **installation script**:

* 8 vCPU
* 16 GB of RAM
``` bash
wget -q -O /tmp/install-thehive.sh https://archives.strangebee.com/scripts/install.sh ; sudo -v ; bash /tmp/install.sh
```

For each release, DEB, RPM and ZIP binary packages are built and provided.

## Installation Guide

The [following Guide](step-by-step-guide.md) let you **prepare**, **install** and **configure** Cortex and its prerequisites for Debian and RPM packages based Operating Systems, as well as for other systems and using our binary packages.

Expand All @@ -63,3 +55,31 @@ Various aspects can configured in the `application.conf` file:
- [database](database.md)
- [Authentication](authentication.md)
- [Analyzers & Responders](analyzers-responders.md)


## _Analyzers_ & _Responders_
Before starting the installation of Cortex, this is important to know how _Analyzers_ and _Responders_ will be managed and run. 2 solutions are available to run them:

### Run locally
The programs are downloaded and installed on the system running Cortex.

There are many disadvantages with this option:

* Some public _Analyzers_ or _Responders_, or you own custom program might required specific applications installed on the system,
* All of the programs published are written in Python and come with dependancies. To run successfully, the dependancies of all programs should be installed on the same operating system ; so there is a high risk of incompatibilities (some program might require a specific version of a librarie with the latest is also required by another one)
* The goal of _Analyzers_ is to extract or gather information or intelligence about observables ; and some of them might be malicious. Depending on the analysis, like a code analysis, you might want to ensure the Analyzer has not been compromised - and the host - by the observable itself
* You might want to ensure that when you run an _Analyzer_, there is no question about the integrity of its programs
* Updating them might be a pain regarding Operating System used and dependancies

### Run with Docker
_Analyzers_ & _Responders_ we publish also have their own [Docker images](https://hub.docker.com/search?q=cortexneurons).

There are several benefits to use Docker images of _Analyzers_ & _Responders_.

* No need to worry about applications required or libraries, it just work
* When requested, Cortex downloads the docker image of a program and instanciate a container running the program. When finished, the container is trashed and a new one is created the next time. No need to worry about the integrity of the program
* This is simple to use and maintain

!!! Tip "This is the recommended option. It requires installing Docker engine as well."

This is not an exclusive choice, both solutions can be used by the same instance of Cortex.
100 changes: 100 additions & 0 deletions docs/cortex/installation-and-configuration/run-cortex-with-docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
### Docker
To use the Docker image, you must use [Docker](https://www.docker.com/) (courtesy of Captain Obvious).

By default, the docker image generate a configuration file for Cortex with:
- the Elasticsearch uri is determined by resolving the host name "elasticsearch",
- the [analyzers](https://download.thehive-project.org/analyzers.json) and [responders](https://download.thehive-project.org/responders.json) official location,
- a generated secret (used to protect the user sessions).
The behaviour of the Cortex Docker image can be customized using environment variables or parameters:

| Parameter | Env variable | Description |
| ------ | ------------ | ----------- |
| `--no-config` | `no_config=1` | Do not configure Cortex |
| `--no-config-secret` | `no_config_secret=1` | Do not add the random secret to the configuration |
| `--no-config-es` | `no_config_es=1` | do not add elasticsearch hosts to configuration
| `--es-uri <uri>` | `es_uri=<uri>` | use this string to configure elasticsearch hosts (format: http(s)://host:port,host:port(/prefix)?querystring)
| `--es-hostname <host>` | `es_hostname=host` | resolve this hostname to find elasticsearch instances
| `--secret <secret>` | `secret=<secret>` | secret to secure sessions
| `--show-secret` | `show_secret=1` | show the generated secret
| `--job-directory <dir>` | `job_directory=<dir>` | use this directory to store job files
| `--docker-job-directory <dir>` | `docker_job_directory=<dir>` | indicate the job directory in the host (not inside container)
| `--analyzer-url <url>` | `analyzer_urls=<url>,<url>,...` | where analyzers are located (url or path)
| `--responder-url <url>` | `responder_urls=<url>,<url>,...` | where responders are located (url or path)
| `--start-docker` | `start_docker=1` | start an internal docker (inside container) to run analyzers/responders
| `--daemon-user <user>` | `daemon_user=<user>` | run cortex using this user

At the end of the generated configuration, the file `/etc/cortex/application.conf` is included. Thus you can override any setting by binding your own `application.conf` into this file:

!!! Example ""
```
docker run --volume /path/to/my/application.conf:/etc/cortex/application.conf thehiveproject/cortex:3.1.0-1 --es-uri http://elasticsearch.local:9200
```

Cortex uses docker to run analyzers and responders. If you run Cortex inside a docker, you can:

- give Cortex access to docker service (recommended solution)
- start a docker service inside Cortex docker container

#### Cortex uses main docker service
In order to use docker service the docker socket must be bound into Cortex container. Moreover, as Cortex shares files with analyzers, a folder must be bound between them.

!!! Example ""
```
docker run --volume /var/run/docker.sock:/var/run/docker.sock --volume /var/run/cortex/jobs:/tmp/cortex-jobs thehiveproject/cortex:3.1.0-1 --job-directory /tmp/cortex-jobs --docker-job-directory /var/run/cortex/jobs
```

Cortex can instantiate docker container by using the docker socket `/var/run/docker.sock`. The folder `/var/run/cortex/jobs` is used to store temporary file of jobs. The folder `/tmp/cortex-jobs` is job folder inside the docker. In order to make job file visible to analyzer docker, Cortex needs to know both folders (parameters `--job-directory` and `-docker-job-directory`). On most cases, job directories are the same and `--docker-job-directory` can be omitted.

If you run Cortex in Windows, the docker service is accessible through the named pipe `\\.\pipe\docker_engine`. The command becomes

!!! Example ""
```
docker run --volume //./pipe/docker_engine://./pipe/docker_engine --volume C:\\CORTEX\\JOBS:/tmp/cortex-jobs thehiveproject/cortex:latest --job-directory /tmp/cortex-jobs --docker-job-directory C:\\CORTEX\\JOBS
```

#### Docker in docker (docker-ception)
You can also run docker service inside Cortex container, a docker in a docker with `--start-docker` parameter. The container must be run in privileged mode.

!!! Example ""
```
docker run --privileged thehiveproject/cortex:3.1.0-1 --start-docker
```
In this case you don't need to bind job directory.

#### Use Docker-compose
Cortex requires [Elasticsearch](#elasticsearch-inside-a-docker) to run. You can use `docker-compose` to start them together in Docker or install and configure Elasticsearch manually.
[Docker-compose](https://docs.docker.com/compose/install/) can start multiple dockers and link them together.

The following [docker-compose.yml](https://raw.githubusercontent.com/TheHive-Project/Cortex/master/docker/cortex/docker-compose.yml)
file starts Elasticsearch and Cortex:

!!! Example ""
```
version: "2"
services:
elasticsearch:
image: elasticsearch:7.9.1
environment:
- http.host=0.0.0.0
- discovery.type=single-node
- script.allowed_types=inline
- thread_pool.search.queue_size=100000
- thread_pool.write.queue_size=10000
volumes:
- /path/to/data:/usr/share/elasticsearch/data
cortex:
image: thehiveproject/cortex:3.1.1
environment:
- job_directory=${job_directory}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${job_directory}:${job_directory}
depends_on:
- elasticsearch
ports:
- "0.0.0.0:9001:9001"
```

Put this [docker-compose file](https://raw.githubusercontent.com/TheHive-Project/Cortex/master/docker/docker-compose.yaml) and [.env](https://raw.githubusercontent.com/TheHive-Project/Cortex/master/docker/cortex/.env) in an empty folder and run `docker-compose up`. Cortex is exposed on 9001/tcp port. These ports can be changed by modifying the `docker-compose` file.

!!! Tip "For advanced configuration, visit [our Docker Templates repository](https://github.com/TheHive-Project/Docker-Templates)"
3 changes: 3 additions & 0 deletions robots.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
User-agent: *
Allow:
Sitemap: https://docs.thehive-project.org/sitemap.xml
Loading

0 comments on commit 7543e85

Please sign in to comment.