-
Notifications
You must be signed in to change notification settings - Fork 104
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(guides): add screenshots from circleci and descriptions (#330)
* feat: add screenshots from circleci and descriptions * Update contributors-guide/tools-and-technologies/ci_cd_pipelines.md Co-authored-by: Sam <[email protected]> Co-authored-by: Sam <[email protected]>
- Loading branch information
1 parent
be01a2b
commit a808be7
Showing
8 changed files
with
137 additions
and
10 deletions.
There are no files selected for viewing
Binary file added
BIN
+31 KB
contributors-guide/tools-and-technologies/assets/images/ci_cd_lib_master.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+24.7 KB
contributors-guide/tools-and-technologies/assets/images/ci_cd_lib_pr.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+25.4 KB
contributors-guide/tools-and-technologies/assets/images/ci_cd_lib_tag.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+40.3 KB
contributors-guide/tools-and-technologies/assets/images/ci_cd_svc_master.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+26.8 KB
contributors-guide/tools-and-technologies/assets/images/ci_cd_svc_pr.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+56 KB
contributors-guide/tools-and-technologies/assets/images/ci_cd_svc_tag.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,13 +14,13 @@ We have a suite of tools bundled under the [license-scanner](https://github.com/ | |
For more information, refer to the [readme](https://github.com/mojaloop/license-scanner) in the license-scanner repository. | ||
|
||
|
||
## Blacklisting and Whitelisting | ||
## Blocking and Allowing Licenses | ||
|
||
The license-scanner works by blacklisting unwanted licenses, and whitelisting packages which have been manually audited, and we are comfortable with using. | ||
The license-scanner works by blocklisting unwanted licenses, and allowlisting packages which have been manually audited, and we are comfortable with using. | ||
|
||
In the [`config.toml`](https://github.com/mojaloop/license-scanner/blob/master/config.toml) file, we configure two arrays of strings. | ||
|
||
__Adding a new License identifier to the blacklist:__ | ||
__Adding a new License identifier to the blocklist:__ | ||
|
||
Edit `config.tml`, and add the license string into the `failList` array: | ||
```toml | ||
|
@@ -33,14 +33,14 @@ failList = [ | |
] | ||
``` | ||
|
||
The license-scanner will pick out licenses that it finds in the `node_modules` folder, and if the license string is in the `failList` (using a fuzzy string search), the license-scanner will fail, unless the package is found in the whitelist. | ||
The license-scanner will pick out licenses that it finds in the `node_modules` folder, and if the license string is in the `failList` (using a fuzzy string search), the license-scanner will fail, unless the package is found in the allowlist. | ||
|
||
__Adding a new package to the whitelist:__ | ||
__Adding a new package to the allowlist:__ | ||
|
||
In addition to maintaining a blacklist of licenses, we whitelist packages that we have manually audited and are happy to include. | ||
In addition to maintaining a blocklist of licenses, we allowlist packages that we have manually audited and are happy to include. | ||
The most common case for this is packages that don't have a license entry in the `package.json` file, which the npm license scan tool lists as `UNKNOWN`. | ||
|
||
To add a package to the whitelist, simply add an entry to the `excludeList` in `config.toml`. For example: | ||
To add a package to the allowlist, simply add an entry to the `excludeList` in `config.toml`. For example: | ||
```toml | ||
excludeList = [ | ||
"[email protected]", | ||
|
@@ -51,11 +51,11 @@ excludeList = [ | |
] | ||
``` | ||
|
||
>___FAQ:__ Why keep a whitelist?_ | ||
>___FAQ:__ Why keep a allowlist?_ | ||
> | ||
>The license-scanner is not perfect, and sometimes there are packages that have been incorrectly identified, or contain no license entry in the `package.json` file, but do contain a valid license in the package's git repository. | ||
> | ||
>By maintaining a whitelist of packages we have manually audited, we can get around incorrectly labelled packages. | ||
>By maintaining a allowlist of packages we have manually audited, we can get around incorrectly labelled packages. | ||
|
||
## Running Inside CI/CD | ||
|
@@ -78,7 +78,7 @@ The license-scanner does the following: | |
* 4. Uploads the results as a `.csv` artifact | ||
|
||
|
||
Should the license scanner pass (i.e. find no licenses that are blacklisted), the build will succeed. | ||
Should the license scanner pass (i.e. find no licenses that are blocklisted), the build will succeed. | ||
|
||
<img src="./assets/diagrams/automated-license-scanning/audit-licenses-pr.svg" alt="CircleCI license scanning for PR" width=900> | ||
|
||
|
127 changes: 127 additions & 0 deletions
127
contributors-guide/tools-and-technologies/ci_cd_pipelines.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,127 @@ | ||
# CI/CD Pipelines | ||
|
||
The Mojaloop Community uses [CircleCI](https://circleci.com/) to automatically build, test and deploy our | ||
software. This document describes how we use CI/CD in Mojaloop, the different checks we perform on the | ||
software, and how we distribute the software. | ||
|
||
Broadly speaking, there are 2 types of workflow we use, depending on the type of project: | ||
- Library: build node project -> test -> publish to [npm](https://www.npmjs.com/search?q=%40mojaloop) | ||
- Service: build docker image -> test -> publish to [Docker Hub](https://hub.docker.com/u/mojaloop) | ||
|
||
Additionally, we also maintain a set of [Mojaloop Helm Charts](http://docs.mojaloop.io/helm/), which are | ||
built from the [mojaloop/helm](https://github.com/mojaloop/helm) | ||
|
||
## Libraries | ||
|
||
> For a good example of this CI/CD pattern see [central-services-shared](https://github.com/mojaloop/central-services-shared/blob/master/.circleci/config.yml) | ||
### Pull Request (PR) Flow: | ||
|
||
The PR flow runs on pull requests, and during the [PR review process](https://github.com/mojaloop/documentation/blob/master/contributors-guide/standards/creating-new-features.md#creating-new-features), these checks must be satisfied | ||
for the code to be merged in. | ||
|
||
 | ||
|
||
| Step | Description | More Info | | ||
| --- | ----------- | --------- | | ||
| pr-title-check | Checks that the PR title conforms to the conventional commits specification | Defined in a CircleCI orb here: [mojaloop/ci-config](https://github.com/mojaloop/ci-config) | | ||
| test-coverage | Runs the unit-tests and checks that code coverage is above the specified limit. | Typically this is 90% | | ||
| test-unit | Runs the unit-tests. Fails if any unit test fails. | | | ||
| vulnerability-check | Runs the `npm audit` tool to search for any vulnerabilities in dependencies | `npm audit` is full of false positives, or security issues that don't apply to our codebase. We use `npm-audit-resolver` to give us flexibility over vulnerabilities that may be ignored, e.g `devDependencies` | | ||
| audit-licenses | Runs the mojaloop `license-scanner-tool` and fails if any license found doesn't match an allowlist specified in the `license-scanner-tool` | [license-scanner-tool repo](https://github.com/mojaloop/license-scanner-tool) | | ||
|
||
|
||
### Master and Release Flow: | ||
|
||
This CI/CD pipeline runs on the master/main branch: | ||
|
||
 | ||
|
||
| Step | Description | More Info | | ||
| --- | ----------- | --------- | | ||
| pr-title-check | See above | | | ||
| test-coverage | See above | | | ||
| test-unit | See above | | | ||
| vulnerability-check | See above | | | ||
| audit-licenses | See above | | | ||
| release | Runs a release which creates a git tag and pushes the git tag | | | ||
| github-release | Adds the release metadata (e.g. changelog) to github, sends a slack alert to #announcements | | | ||
|
||
|
||
### Tag Flow: | ||
|
||
Once a git tag is pushed to the repository, it triggers a workflow that ends | ||
in publishing to `npm`. Importantly, the checks are all run again, to ensure | ||
nothing has changed (e.g. dependencies) inbetween the main/master and the actual | ||
artefact that is published to `npm`. | ||
|
||
 | ||
|
||
| Step | Description | More Info | | ||
| --- | ----------- | --------- | | ||
| pr-title-check | See above | | | ||
| test-coverage | See above | | | ||
| test-unit | See above | | | ||
| vulnerability-check | See above | | | ||
| audit-licenses | See above | | | ||
| publish | Publishes the latest version of the library based on the git tag | | | ||
|
||
|
||
## Services | ||
|
||
> For a good example of this CI/CD pattern see [central-ledger](https://github.com/mojaloop/central-ledger/blob/master/.circleci/config.yml) | ||
### Pull Request (PR) Flow: | ||
|
||
The PR flow runs on pull requests, and during the PR review process, these checks must be satisfied | ||
for the code to be merged in. | ||
|
||
 | ||
|
||
| Step | Description | More Info | | ||
| --- | ----------- | --------- | | ||
| pr-title-check | Checks that the PR title conforms to the conventional commits specification | Defined in a CircleCI orb here: [mojaloop/ci-config](https://github.com/mojaloop/ci-config) | | ||
| test-coverage | Runs the unit-tests and checks that code coverage is above the specified limit. | Typically this is 90% | | ||
| test-unit | Runs the unit-tests. Fails if any unit test fails. | | | ||
| test-integration | Runs the integration-tests. Typically by building a docker image locally | | | ||
| vulnerability-check | Runs the `npm audit` tool to search for any vulnerabilities in dependencies | `npm audit` is full of false positives, or security issues that don't apply to our codebase. We use `npm-audit-resolver` to give us flexibility over vulnerabilities that may be ignored, e.g `devDependencies` | | ||
| audit-licenses | Runs the mojaloop `license-scanner-tool` and fails if any license found doesn't match an allowlist specified in the `license-scanner-tool` | [license-scanner-tool repo](https://github.com/mojaloop/license-scanner-tool) | | ||
|
||
|
||
### Master and Release Flow: | ||
|
||
This CI/CD pipeline runs on the master/main branch: | ||
|
||
 | ||
|
||
| Step | Description | More Info | | ||
| --- | ----------- | --------- | | ||
| pr-title-check | See above | | | ||
| test-coverage | See above | | | ||
| test-unit | See above | | | ||
| test-integration | See above | | | ||
| vulnerability-check | See above | | | ||
| audit-licenses | See above | | | ||
| release | Runs a release which creates a git tag and pushes the git tag | | | ||
| github-release | Adds the release metadata (e.g. changelog) to github, sends a slack alert to #announcements | | | ||
|
||
|
||
### Tag Flow: | ||
|
||
Once a git tag is pushed to the repository, it triggers a workflow that ends | ||
in publishing a docker image to Docker Hub. Importantly, the checks are all run again, | ||
and further scans are made on the docker image before the image is pushed. | ||
|
||
 | ||
|
||
| Step | Description | More Info | | ||
| --- | ----------- | --------- | | ||
| pr-title-check | See above | | | ||
| test-coverage | See above | | | ||
| test-unit | See above | | | ||
| vulnerability-check | See above | | | ||
| audit-licenses | See above | | | ||
| build | Builds the docker image | | | ||
| image-scan | Runs `anchore/analyze_local_image` to scan the image | See [anchore-engine](https://circleci.com/developer/orbs/orb/anchore/anchore-engine) CircleCI Orb for more information. | | ||
| license-scan | Runs the mojaloop `license-scanner-tool` on the licenses contained in the docker image | | | ||
| publish | Publishes the latest version of the library based on the git tag | | |