Skip to content

Commit

Permalink
feat(guides): add screenshots from circleci and descriptions (#330)
Browse files Browse the repository at this point in the history
* feat: add screenshots from circleci and descriptions

* Update contributors-guide/tools-and-technologies/ci_cd_pipelines.md

Co-authored-by: Sam <[email protected]>

Co-authored-by: Sam <[email protected]>
  • Loading branch information
Lewis Daly and elnyry-sam-k authored Sep 24, 2021
1 parent be01a2b commit a808be7
Show file tree
Hide file tree
Showing 8 changed files with 137 additions and 10 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ We have a suite of tools bundled under the [license-scanner](https://github.com/
For more information, refer to the [readme](https://github.com/mojaloop/license-scanner) in the license-scanner repository.


## Blacklisting and Whitelisting
## Blocking and Allowing Licenses

The license-scanner works by blacklisting unwanted licenses, and whitelisting packages which have been manually audited, and we are comfortable with using.
The license-scanner works by blocklisting unwanted licenses, and allowlisting packages which have been manually audited, and we are comfortable with using.

In the [`config.toml`](https://github.com/mojaloop/license-scanner/blob/master/config.toml) file, we configure two arrays of strings.

__Adding a new License identifier to the blacklist:__
__Adding a new License identifier to the blocklist:__

Edit `config.tml`, and add the license string into the `failList` array:
```toml
Expand All @@ -33,14 +33,14 @@ failList = [
]
```

The license-scanner will pick out licenses that it finds in the `node_modules` folder, and if the license string is in the `failList` (using a fuzzy string search), the license-scanner will fail, unless the package is found in the whitelist.
The license-scanner will pick out licenses that it finds in the `node_modules` folder, and if the license string is in the `failList` (using a fuzzy string search), the license-scanner will fail, unless the package is found in the allowlist.

__Adding a new package to the whitelist:__
__Adding a new package to the allowlist:__

In addition to maintaining a blacklist of licenses, we whitelist packages that we have manually audited and are happy to include.
In addition to maintaining a blocklist of licenses, we allowlist packages that we have manually audited and are happy to include.
The most common case for this is packages that don't have a license entry in the `package.json` file, which the npm license scan tool lists as `UNKNOWN`.

To add a package to the whitelist, simply add an entry to the `excludeList` in `config.toml`. For example:
To add a package to the allowlist, simply add an entry to the `excludeList` in `config.toml`. For example:
```toml
excludeList = [
"[email protected]",
Expand All @@ -51,11 +51,11 @@ excludeList = [
]
```

>___FAQ:__ Why keep a whitelist?_
>___FAQ:__ Why keep a allowlist?_
>
>The license-scanner is not perfect, and sometimes there are packages that have been incorrectly identified, or contain no license entry in the `package.json` file, but do contain a valid license in the package's git repository.
>
>By maintaining a whitelist of packages we have manually audited, we can get around incorrectly labelled packages.
>By maintaining a allowlist of packages we have manually audited, we can get around incorrectly labelled packages.

## Running Inside CI/CD
Expand All @@ -78,7 +78,7 @@ The license-scanner does the following:
* 4. Uploads the results as a `.csv` artifact


Should the license scanner pass (i.e. find no licenses that are blacklisted), the build will succeed.
Should the license scanner pass (i.e. find no licenses that are blocklisted), the build will succeed.

<img src="./assets/diagrams/automated-license-scanning/audit-licenses-pr.svg" alt="CircleCI license scanning for PR" width=900>

Expand Down
127 changes: 127 additions & 0 deletions contributors-guide/tools-and-technologies/ci_cd_pipelines.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# CI/CD Pipelines

The Mojaloop Community uses [CircleCI](https://circleci.com/) to automatically build, test and deploy our
software. This document describes how we use CI/CD in Mojaloop, the different checks we perform on the
software, and how we distribute the software.

Broadly speaking, there are 2 types of workflow we use, depending on the type of project:
- Library: build node project -> test -> publish to [npm](https://www.npmjs.com/search?q=%40mojaloop)
- Service: build docker image -> test -> publish to [Docker Hub](https://hub.docker.com/u/mojaloop)

Additionally, we also maintain a set of [Mojaloop Helm Charts](http://docs.mojaloop.io/helm/), which are
built from the [mojaloop/helm](https://github.com/mojaloop/helm)

## Libraries

> For a good example of this CI/CD pattern see [central-services-shared](https://github.com/mojaloop/central-services-shared/blob/master/.circleci/config.yml)
### Pull Request (PR) Flow:

The PR flow runs on pull requests, and during the [PR review process](https://github.com/mojaloop/documentation/blob/master/contributors-guide/standards/creating-new-features.md#creating-new-features), these checks must be satisfied
for the code to be merged in.

![](./assets/images/ci_cd_lib_pr.png)

| Step | Description | More Info |
| --- | ----------- | --------- |
| pr-title-check | Checks that the PR title conforms to the conventional commits specification | Defined in a CircleCI orb here: [mojaloop/ci-config](https://github.com/mojaloop/ci-config) |
| test-coverage | Runs the unit-tests and checks that code coverage is above the specified limit. | Typically this is 90% |
| test-unit | Runs the unit-tests. Fails if any unit test fails. | |
| vulnerability-check | Runs the `npm audit` tool to search for any vulnerabilities in dependencies | `npm audit` is full of false positives, or security issues that don't apply to our codebase. We use `npm-audit-resolver` to give us flexibility over vulnerabilities that may be ignored, e.g `devDependencies` |
| audit-licenses | Runs the mojaloop `license-scanner-tool` and fails if any license found doesn't match an allowlist specified in the `license-scanner-tool` | [license-scanner-tool repo](https://github.com/mojaloop/license-scanner-tool) |


### Master and Release Flow:

This CI/CD pipeline runs on the master/main branch:

![](./assets/images/ci_cd_lib_master.png)

| Step | Description | More Info |
| --- | ----------- | --------- |
| pr-title-check | See above | |
| test-coverage | See above | |
| test-unit | See above | |
| vulnerability-check | See above | |
| audit-licenses | See above | |
| release | Runs a release which creates a git tag and pushes the git tag | |
| github-release | Adds the release metadata (e.g. changelog) to github, sends a slack alert to #announcements | |


### Tag Flow:

Once a git tag is pushed to the repository, it triggers a workflow that ends
in publishing to `npm`. Importantly, the checks are all run again, to ensure
nothing has changed (e.g. dependencies) inbetween the main/master and the actual
artefact that is published to `npm`.

![](./assets/images/ci_cd_lib_tag.png)

| Step | Description | More Info |
| --- | ----------- | --------- |
| pr-title-check | See above | |
| test-coverage | See above | |
| test-unit | See above | |
| vulnerability-check | See above | |
| audit-licenses | See above | |
| publish | Publishes the latest version of the library based on the git tag | |


## Services

> For a good example of this CI/CD pattern see [central-ledger](https://github.com/mojaloop/central-ledger/blob/master/.circleci/config.yml)
### Pull Request (PR) Flow:

The PR flow runs on pull requests, and during the PR review process, these checks must be satisfied
for the code to be merged in.

![](./assets/images/ci_cd_svc_pr.png)

| Step | Description | More Info |
| --- | ----------- | --------- |
| pr-title-check | Checks that the PR title conforms to the conventional commits specification | Defined in a CircleCI orb here: [mojaloop/ci-config](https://github.com/mojaloop/ci-config) |
| test-coverage | Runs the unit-tests and checks that code coverage is above the specified limit. | Typically this is 90% |
| test-unit | Runs the unit-tests. Fails if any unit test fails. | |
| test-integration | Runs the integration-tests. Typically by building a docker image locally | |
| vulnerability-check | Runs the `npm audit` tool to search for any vulnerabilities in dependencies | `npm audit` is full of false positives, or security issues that don't apply to our codebase. We use `npm-audit-resolver` to give us flexibility over vulnerabilities that may be ignored, e.g `devDependencies` |
| audit-licenses | Runs the mojaloop `license-scanner-tool` and fails if any license found doesn't match an allowlist specified in the `license-scanner-tool` | [license-scanner-tool repo](https://github.com/mojaloop/license-scanner-tool) |


### Master and Release Flow:

This CI/CD pipeline runs on the master/main branch:

![](./assets/images/ci_cd_svc_master.png)

| Step | Description | More Info |
| --- | ----------- | --------- |
| pr-title-check | See above | |
| test-coverage | See above | |
| test-unit | See above | |
| test-integration | See above | |
| vulnerability-check | See above | |
| audit-licenses | See above | |
| release | Runs a release which creates a git tag and pushes the git tag | |
| github-release | Adds the release metadata (e.g. changelog) to github, sends a slack alert to #announcements | |


### Tag Flow:

Once a git tag is pushed to the repository, it triggers a workflow that ends
in publishing a docker image to Docker Hub. Importantly, the checks are all run again,
and further scans are made on the docker image before the image is pushed.

![](./assets/images/ci_cd_svc_tag.png)

| Step | Description | More Info |
| --- | ----------- | --------- |
| pr-title-check | See above | |
| test-coverage | See above | |
| test-unit | See above | |
| vulnerability-check | See above | |
| audit-licenses | See above | |
| build | Builds the docker image | |
| image-scan | Runs `anchore/analyze_local_image` to scan the image | See [anchore-engine](https://circleci.com/developer/orbs/orb/anchore/anchore-engine) CircleCI Orb for more information. |
| license-scan | Runs the mojaloop `license-scanner-tool` on the licenses contained in the docker image | |
| publish | Publishes the latest version of the library based on the git tag | |

0 comments on commit a808be7

Please sign in to comment.