diff --git a/Makefile b/Makefile index 088f37b..f1a7415 100644 --- a/Makefile +++ b/Makefile @@ -1,8 +1,8 @@ install: - pip3 install -r requirements.txt -e . + pip3 install -r requirements.txt --no-deps -e . -install-dev: - pip3 install -r requirements.txt -e ".[dev]" +install-dev: install + pip3 install -r requirements-dev.txt flake8-lint: flake8 whispers/ tests/unit/ setup.py diff --git a/README.md b/README.md index 81e3c57..b4089a1 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ > "My little birds are everywhere, even in the North, they whisper to me the strangest stories." - _Varys_ -Whispers is an information security analysis tool designed for identifying **hardcoded secrets in structured text and static code** ([CWE-798](https://cwe.mitre.org/data/definitions/798.html)). Whispers can be used as a [standalone binary](https://github.com/adeptex/whispers#download), or as a [Python module](https://github.com/adeptex/whispers#install), which is meant to facilitate its usage individually and as part of automated processes and pipelines at scale. +Whispers is an information security analysis tool designed for identifying **hardcoded secrets in structured text and static code** ([CWE-798](https://cwe.mitre.org/data/definitions/798.html)). Whispers can be used as a [standalone binary](https://github.com/adeptex/whispers#download), or as a [Python module](https://github.com/adeptex/whispers#install), which is meant to facilitate its usage individually, and in automated processes and pipelines at scale. * :clipboard: [Release notes](https://github.com/adeptex/whispers/blob/master/RELEASE_NOTES.md) * :gear: [Request a feature](https://github.com/adeptex/whispers/issues/new?assignees=&labels=&template=feature_request.md&title=) @@ -252,28 +252,26 @@ Simple filtering based on rules and severity can also be done with CLI arguments | Group | Rule ID | Severity | |----------------------|----------------------|-----------------| -| files | file-known | Low | +| keys | aws-secret | Critical | +| keys | aws-token | Critical | +| keys | privatekey | Critical | +| keys | apikey-known | Critical | +| keys | apikey | High | +| keys | aws-id | Medium | +| keys | aws-account | Low | +| keys | apikey-maybe | Low | +| passwords | password | High | +| passwords | uri | High | | infra | dockercfg | High | -| infra | htpasswd | Medium | | infra | npmrc | High | | infra | pip | High | | infra | pypirc | High | -| keys | apikey | Medium | -| keys | apikey-known | High | -| keys | apikey-maybe | Low | -| keys | aws-id | Critical | -| keys | aws-secret | Critical | -| keys | aws-token | Critical | -| keys | privatekey | High | -| misc | comment | Info | +| infra | htpasswd | Medium | +| misc | webhook | Medium | | misc | creditcard | Low | | misc | secret | Low | -| misc | webhook | Low | -| passwords | password | High | -| passwords | uri | High | -| python | cors | Low | -| python | system | Low | - +| misc | comment | Info | +| files | file-known | Low | ### Custom rules diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 778d338..268abd9 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,74 +1,99 @@ -# Whispers 2.3.0 release notes +# Whispers 2.4.0 release notes -* **New Feature:** 💫 Static Code Analysis 💫 is now supported! - * The present release complements classic Whispers' structured text analysis with [Semgrep](https://semgrep.dev)'s AST generator for [common programming languages](https://semgrep.dev/docs/supported-languages) like Python, PHP, Java/Scala/Kotlin, JavaScript/TypeScript, Go, etc etc. - * New argument `--ast` for enabling this feature via the CLI (it is disabled by default) - * New setting `ast: true` for enabling this feature via a custom config file (set to `ast: false` by default) - * Replaced [`astroid`](https://github.com/adeptex/whispers/blob/8f17f77e2199c55458ff125e3fb477a2a9349593/whispers/plugins/python.py) Python AST generator with [`semgrep`](https://github.com/adeptex/whispers/blob/master/whispers/plugins/semgrep.py) +* 💫 **Remove Semgrep telemetry** 💫 +* Lazy-load parsers +* Severity levels reassignment +* Detection rule improvements + * URI credentials + * AWS Account ID +* Generalize default config -* [Detection rule](https://github.com/adeptex/whispers/blob/master/whispers/rules) improvements - * Known API keys - * AWS account ID - * Passwords - * Creditcards -* Drop end-of-life Python support - * Versions 3.6 and 3.7 are no longer supported. Oldest supported version is Python 3.8. - * Last release that supports Python 3.6 and 3.7 is [Whispers 2.2.1](https://github.com/adeptex/whispers/releases/tag/2.2.1) +## 💫 Remove Semgrep telemetry 💫 -* Dependency tracking improvements - * New [`requirements-dev.txt`](https://github.com/adeptex/whispers/blob/master/requirements-dev.txt) file allows Dependabot updates for dev dependencies - * Modified [`setup.py`](https://github.com/adeptex/whispers/blob/master/setup.py) to read from `requirements.txt` and `requirements-dev.txt` - * Updated build CI to use Python 3.12.3 +It's a better world now that corporations build telemetry into every single piece of software... **not really** 😒... It was shoking to see telemetry packages installed as part of Whispers. But how did this happen? -* Debugging and troubleshooting - * Modified [`config.yml`](https://github.com/adeptex/whispers/blob/master/whispers/config.yml) to exclude known false positives - * Fixed [`Dockerfile`](https://github.com/adeptex/whispers/blob/master/Dockerfile) to work with `docker build -t whispers .` or the same `make image` - * New arg `--dump` for generating an AST of a file: `whispers --dump src/example.ts` +As it turns out, Semgrep includes A LOT of code to support exfiltrating metadata and usage information from your machine. This Whispers release is largely dedicated to stripping out all unnecessary spyware garbage "required" by Semgrep. Apart from privacy, a nice side effect of this is that now Semgrep runs a lot faster when parsing ASTs! Win-win. +The following are 24 (out of 32.. wtf??) "required" Semgrep dependencies that are now excluded: -## 💫 New Feature: Static Code Analysis 💫 - -With the release of Whispers 2.3, it is now possible to accurately apply Whispers' secret detection techniques for structured text to static code. Before this release, Whispers only supported structured text formats, such as JSON or XML. [Semgrep](https://semgrep.dev) is an open source SAST tool, which has a built-in feature for generating Abstract Structure Trees (ASTs) for [many common programming languages](https://semgrep.dev/docs/supported-languages). Generating an AST for static code yields an accurate structured text representation, which can be checked for secrets with Whispers' rules and plugins. As such, generating ASTs requires an additional "format conversion" step, which naturally affects runtime speed. When AST is enabled it will take longer to scan the same scope if any source code files are present. The increased amount of runtime time would be however long it takes to run the following command on all static code files in scope: - -```sh -semgrep scan --metrics=off --quiet --dump-ast --json --lang $LANG $SRCFILE ``` - -Consider the following benchmarks: - -```sh -time whispers -F " " tests/fixtures -# 313 detected secrets -# 0,51s user 0,03s system 99% cpu 0,540 total -# 0,60s user 0,04s system 99% cpu 0,642 total - -time whispers -a -F " " tests/fixtures -# 421 detected secrets -# 2,20s user 0,40s system 100% cpu 2,589 total -# 2,32s user 0,46s system 100% cpu 2,772 total +certifi==2024.8.30 +charset-normalizer==3.3.2 +Deprecated==1.2.14 +googleapis-common-protos==1.65.0 +idna==3.10 +importlib_metadata==7.1.0 +markdown-it-py==3.0.0 +mdurl==0.1.2 +opentelemetry-api==1.25.0 +opentelemetry-exporter-otlp-proto-common==1.25.0 +opentelemetry-exporter-otlp-proto-http==1.25.0 +opentelemetry-instrumentation==0.46b0 +opentelemetry-instrumentation-requests==0.46b0 +opentelemetry-proto==1.25.0 +opentelemetry-sdk==1.25.0 +opentelemetry-semantic-conventions==0.46b0 +opentelemetry-util-http==0.46b0 +protobuf==4.25.5 +Pygments==2.18.0 +requests==2.32.3 +rich==13.9.1 +setuptools==75.1.0 +urllib3==2.2.3 +zipp==3.20.2 ``` -AST conversion is **disabled by default** - `semgrep` will **not** execute at all unless explicitly enabled. Custom config files that are missing `ast: false` or `ast: true` will default to `false`. +The confirmation of this great success can be seen in every `pip3 install whispers` log in the form of these amazing error messages: -```yaml -ast: true # enable AST in config.yml ``` - -```sh -whispers --ast target/dir/or/file # enable AST in CLI +ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. +semgrep 1.85.0 requires attrs>=21.3, which is not installed. +semgrep 1.85.0 requires boltons~=21.0, which is not installed. +semgrep 1.85.0 requires click-option-group~=0.5, which is not installed. +semgrep 1.85.0 requires colorama~=0.4.0, which is not installed. +semgrep 1.85.0 requires defusedxml~=0.7.1, which is not installed. +semgrep 1.85.0 requires exceptiongroup~=1.2.0, which is not installed. +semgrep 1.85.0 requires glom~=22.1, which is not installed. +semgrep 1.85.0 requires opentelemetry-api~=1.25.0, which is not installed. +semgrep 1.85.0 requires opentelemetry-exporter-otlp-proto-http~=1.25.0, which is not installed. +semgrep 1.85.0 requires opentelemetry-instrumentation-requests~=0.46b0, which is not installed. +semgrep 1.85.0 requires opentelemetry-sdk~=1.25.0, which is not installed. +semgrep 1.85.0 requires peewee~=3.14, which is not installed. +semgrep 1.85.0 requires ruamel.yaml<0.18,>=0.16.0, which is not installed. +semgrep 1.85.0 requires tomli~=2.0.1, which is not installed. +semgrep 1.85.0 requires wcmatch~=8.3, which is not installed. ``` ## ❌ Breaking changes ❌ -### ❌ Replaced `astroid` with `semgrep` ❌ - -Before Whispers 2.3, only Python AST scanning was natively supported by `astroid`, and integrated via [`plugins/python.py`](https://github.com/adeptex/whispers/blob/8f17f77e2199c55458ff125e3fb477a2a9349593/whispers/plugins/python.py). With the release of Whispers 2.3, this functionality is superseded by `semgrep`, and integrated via [`plugins/semgrep.py`](https://github.com/adeptex/whispers/blob/master/whispers/plugins/semgrep.py). As a base line, the new `semgrep` plugin detects the same findings as the `astroid` plugin, but supports more programming languages. - -Unfortunately `semgrep` has telemetry enabled by default, but can be turned off via [`--metrics=off`](https://github.com/adeptex/whispers/blob/master/whispers/plugins/semgrep.py#L57). In any case, `semgrep` will not execute unless explicitly enabled via args or config. - -⚠️ **NOTE:** At the time of writing, `semgrep` [does not support Windows OS natively](https://github.com/semgrep/semgrep/issues/1330), and can only be installed through WSL. As such, compiled Whispers PE32+ executable comes without Static Code Analysis support. Installing Whispers on Windows via WSL with `pip3 install whispers` *does* have Static Code Analysis support. +### ❌ Severity levels reassignment ❌ + +Adjusted rule severity levels to add structure. New severity levels are the following: + +| Group | Rule ID | Severity Before | Severity Now | +|----------------------|----------------------|-----------------|--------------| +| keys | aws-secret | Critical | Critical | +| keys | aws-token | Critical | Critical | +| keys | privatekey | High | Critical | +| keys | apikey-known | High | Critical | +| keys | apikey | Medium | High | +| keys | aws-id | Critical | Medium | +| keys | aws-account | Low | Low | +| keys | apikey-maybe | Low | Low | +| passwords | password | High | High | +| passwords | uri | High | High | +| infra | dockercfg | High | High | +| infra | npmrc | High | High | +| infra | pip | High | High | +| infra | pypirc | High | High | +| infra | htpasswd | Medium | Medium | +| misc | webhook | Low | Medium | +| misc | creditcard | Low | Low | +| misc | secret | Low | Low | +| misc | comment | Info | Info | +| files | file-known | Low | Low | # Changelog @@ -79,3 +104,4 @@ Unfortunately `semgrep` has telemetry enabled by default, but can be turned off |2022-07-29|2.1.0|https://github.com/adeptex/whispers/releases/tag/2.1.0| |2023-10-23|2.2.0|https://github.com/adeptex/whispers/releases/tag/2.2.0| |2024-06-16|2.3.0|https://github.com/adeptex/whispers/releases/tag/2.3.0| +|2024-10-05|2.4.0|https://github.com/adeptex/whispers/releases/tag/2.4.0| diff --git a/requirements.txt b/requirements.txt index cb5ab74..943d6a9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,9 +2,13 @@ beautifulsoup4==4.12.3 crossplane==0.5.8 jellyfish==1.1.0 jproperties==2.1.1 +jsonschema==4.23.0 +jsonschema-specifications==2023.12.1 luhn==0.2.0 lxml==5.3.0 -pyyaml==6.0.2 +PyYAML==6.0.2 semgrep==1.85.0; sys_platform != 'win32' six==1.16.0 +soupsieve==2.6 +typing_extensions==4.12.2 wrapt==1.16.0 \ No newline at end of file diff --git a/tests/fixtures/.gitlab-ci.yml b/tests/fixtures/.gitlab-ci.yml new file mode 100644 index 0000000..3271cb3 --- /dev/null +++ b/tests/fixtures/.gitlab-ci.yml @@ -0,0 +1,16 @@ +variables: + USERNAME: admin + PASSWORD: HardcodedAdminPassword01 + APIKEY: 23d968ff-10b9-4e6f-a33a-hardcoded02 + +compliant: + stage: test + script: + - curl https://$USERNAME:$PASSWORD@git.hosting-name.com + - curl https://git.hosting-name.com/?APIKEY=$APIKEY + +noncompliant: + stage: test + script: + - curl https://admin:HardcodedAdminPassword03@git.hosting-name.com + - curl https://git.hosting-name.com/?APIKEY=23d968ff-10b9-4e6f-a33a-hardcoded04 diff --git a/tests/fixtures/arn.xml b/tests/fixtures/arn.xml index b9ba3ec..e7fb223 100644 --- a/tests/fixtures/arn.xml +++ b/tests/fixtures/arn.xml @@ -5,8 +5,8 @@ arn:aws:kms:{REGION}:{ACCOUNT}:key/{KEY_ID} - arn:aws:kms:eu-central-1:123456123456:key/hardcoded - arn:aws:kms:ap-southeast-1:123456123456:key/hardcoded - arn:aws:iam::123456123456:oidc-provider/auth-dev.mozilla.auth0.com + arn:aws:kms:eu-central-1:111122223333:key/hardcoded + arn:aws:kms:ap-southeast-1:111122223333:key/hardcoded + arn:aws:iam::111122223333:oidc-provider/auth-dev.mozilla.auth0.com diff --git a/tests/fixtures/arn.yml b/tests/fixtures/arn.yml index 71ec09e..dcb6bec 100644 --- a/tests/fixtures/arn.yml +++ b/tests/fixtures/arn.yml @@ -4,8 +4,8 @@ compliant: noncompliant: - arn01: arn:aws:kms:eu-central-1:123456123456:key/hardcoded - arn02: arn:aws:kms:ap-southeast-1:123456123456:key/hardcoded - arn03: arn:aws:iam::123456123456:oidc-provider/auth-dev.mozilla.auth0.com + arn01: arn:aws:kms:eu-central-1:111122223333:key/hardcoded + arn02: arn:aws:kms:ap-southeast-1:111122223333:key/hardcoded + arn03: arn:aws:iam::111122223333:oidc-provider/auth-dev.mozilla.auth0.com arn_list: - - arn:aws:kms:eu-central-1:123456123456:key/hardcoded + - arn:aws:kms:eu-central-1:111122223333:key/hardcoded diff --git a/tests/fixtures/aws.json b/tests/fixtures/aws.json index a80c575..3cf0fda 100644 --- a/tests/fixtures/aws.json +++ b/tests/fixtures/aws.json @@ -4,7 +4,7 @@ "aws_id": "{{ AWS ID }}", "aws_key": "${AWS_KEY}", "commit_id": "912ec803b2ce49e4a541068d495ab57000000000", - "role": "arn:aws:iam::123456789000:role/role-name" + "role": "arn:aws:iam::111122223333:role/role-name" }, { "aws_account01": "000000000000", @@ -18,7 +18,7 @@ "aws_token": "FakeYXdzELv//////////wEldj3948yOJRO84jgpoip239232hEOHhfkjhefkwue97jorhfiuh+XjFC9Je/YG7JCqKjrspab2lB+7/Fb1NJFjgwur47Dbhs/L7nh+/VGnwLoAo8CIqoPBLRmXItaoiuuofZnr+ktihZk1Yi55sYZ12hfRMPVbDmhf9Ke683+e9bJirhUEghw9424JOhgwrgqq99MvzCEFe4eXPOSgAcQcD2xqnnKO738tjhoh23HFqjflhefibWegfqefgqUF12hvgfwegqf" }, { - "aws_account01": "123456789123" + "aws_account01": "111122223333" } ] } \ No newline at end of file diff --git a/tests/fixtures/aws.xml b/tests/fixtures/aws.xml index 6a17fbf..68562ff 100644 --- a/tests/fixtures/aws.xml +++ b/tests/fixtures/aws.xml @@ -4,7 +4,7 @@ {{ AWS ID }} ${AWS_KEY} 912ec803b2ce49e4a541068d495ab57000000000 - arn:aws:iam::123456789000:role/role-name + arn:aws:iam::111122223333:role/role-name 000000000000 111111111111 @@ -12,6 +12,6 @@ AKIAHI38FAKE1IWUQEEN PA3XsxZ8d8cPQLmnZzFAKEdzC6ND2a8vhbyXU/Dw FakeYXdzELv//////////wEldj3948yOJRO84jgpoip239232hEOHhfkjhefkwue97jorhfiuh+XjFC9Je/YG7JCqKjrspab2lB+7/Fb1NJFjgwur47Dbhs/L7nh+/VGnwLoAo8CIqoPBLRmXItaoiuuofZnr+ktihZk1Yi55sYZ12hfRMPVbDmhf9Ke683+e9bJirhUEghw9424JOhgwrgqq99MvzCEFe4eXPOSgAcQcD2xqnnKO738tjhoh23HFqjflhefibWegfqefgqUF12hvgfwegqf - 123456789123 + 111122223333 \ No newline at end of file diff --git a/tests/fixtures/aws.yml b/tests/fixtures/aws.yml index 0a32e8c..c609139 100644 --- a/tests/fixtures/aws.yml +++ b/tests/fixtures/aws.yml @@ -2,7 +2,7 @@ compliant: aws_id: "{{ AWS ID }}" aws_key: "${AWS_KEY}" commit_id: 912ec803b2ce49e4a541068d495ab57000000000 - role: arn:aws:iam::123456789000:role/role-name + role: arn:aws:iam::111122223333:role/role-name aws_account01: '000000000000' aws_account02: '111111111111' @@ -10,4 +10,4 @@ noncompliant: aws_id: AKIAHI38FAKE1IWUQEEN aws_key: PA3XsxZ8d8cPQLmnZzFAKEdzC6ND2a8vhbyXU/Dw aws_token: FakeYXdzELv//////////wEldj3948yOJRO84jgpoip239232hEOHhfkjhefkwue97jorhfiuh+XjFC9Je/YG7JCqKjrspab2lB+7/Fb1NJFjgwur47Dbhs/L7nh+/VGnwLoAo8CIqoPBLRmXItaoiuuofZnr+ktihZk1Yi55sYZ12hfRMPVbDmhf9Ke683+e9bJirhUEghw9424JOhgwrgqq99MvzCEFe4eXPOSgAcQcD2xqnnKO738tjhoh23HFqjflhefibWegfqefgqUF12hvgfwegqf - aws_account: '123456789123' \ No newline at end of file + aws_account: '111122223333' \ No newline at end of file diff --git a/tests/fixtures/nginx.conf b/tests/fixtures/nginx.conf index ce39e0d..d1724db 100644 --- a/tests/fixtures/nginx.conf +++ b/tests/fixtures/nginx.conf @@ -15,6 +15,6 @@ http { server { set $apikey 23d968ff-10b9-4e6f-a33a-hardcoded01; proxy_set_header x-apikey 23d968ff-10b9-4e6f-a33a-hardcoded02; - proxy_set_header x-url https://admin:hardcoded03@localhost.local; + proxy_set_header x-url https://admin:23d968ff-10b9-4e6f-a33a-hardcoded03@localhost.local; } } diff --git a/tests/fixtures/severity.yml b/tests/fixtures/severity.yml index 365c12f..fd479e3 100644 --- a/tests/fixtures/severity.yml +++ b/tests/fixtures/severity.yml @@ -1,13 +1,11 @@ -blocker: - aws_id: AKIAHI38FAKHARDCODED - critical: rsa: "-----BEGIN RSA PRIVATE KEY-----\nQyNTUxOQAAACCtrF27B/zd9DEpd38IbVBy93wSeYXKU0AGXMyO8ePu2QAAAKBSzpYEUs6W\n-----END RSA PRIVATE KEY-----" -major: +high: apikey: YXNkZmZmZmZm_HARDcoded -minor: +medium: + aws_id: AKIAHI38FAKHARDCODED slack-webhook: https://hooks.slack.com/services/HARDC0D3D/HARDC0D3D/hardcodedWebhook01 info: diff --git a/tests/unit/core/test_pairs.py b/tests/unit/core/test_pairs.py index 138977c..ace525b 100644 --- a/tests/unit/core/test_pairs.py +++ b/tests/unit/core/test_pairs.py @@ -106,6 +106,7 @@ def test_filter_static(key, value, expected): (".aws/credentials", False, Config), (".dockercfg", False, Dockercfg), (".env", False, Shell), + (".gitlab-ci.yml", False, Yml), (".htpasswd", False, Htpasswd), (".npmrc", False, Npmrc), (".pypirc", False, Pypirc), @@ -126,6 +127,7 @@ def test_filter_static(key, value, expected): ("language/fixture.py", True, Semgrep), ("language/fixture.py", False, None), ("language/fixture.sh", False, Shell), + ("nginx.conf", False, Config), ("page.html", False, Html), ("passwords.yml", False, Yml), ("pip.conf", False, Pip), diff --git a/tests/unit/core/test_secrets.py b/tests/unit/core/test_secrets.py index d3f70fa..8b99bb7 100644 --- a/tests/unit/core/test_secrets.py +++ b/tests/unit/core/test_secrets.py @@ -46,30 +46,31 @@ def test_detect_secrets_by_key(src, expected): @pytest.mark.parametrize( ("src", "severity", "expected"), [ - (".aws/credentials", "Critical", 3), + (".aws/credentials", "Critical", 2), (".dockercfg", "High", 1), - (".env", "Medium", 4), + (".env", "High", 4), + (".gitlab-ci.yml", DEFAULT_SEVERITY, 4), (".htpasswd", "Medium", 2), (".npmrc", "High", 5), (".pypirc", "High", 1), - ("apikeys-known.yml", "High", 56), - ("apikeys.json", "Medium", 9), - ("apikeys.xml", "Medium", 9), - ("apikeys.yml", "Medium", 9), + ("apikeys-known.yml", "Critical", 56), + ("apikeys.json", "High", 9), + ("apikeys.xml", "High", 9), + ("apikeys.yml", "High", 9), ("apikeys.json", "Low", 1), ("apikeys.xml", "Low", 2), ("apikeys.yml", "Low", 1), ("arn.yml", "Low", 4), ("arn.xml", "Low", 3), - ("aws.yml", "Critical", 3), + ("aws.yml", "Critical", 2), ("aws.yml", DEFAULT_SEVERITY, 7), - ("aws.json", "Critical", 3), + ("aws.json", "Critical", 2), ("aws.json", DEFAULT_SEVERITY, 7), - ("aws.xml", "Critical", 3), + ("aws.xml", "Critical", 2), ("aws.xml", DEFAULT_SEVERITY, 7), - ("beans.xml", "High", 1), - ("beans.xml.dist", "High", 1), - ("beans.xml.template", "High", 1), + ("beans.xml", "High", 2), + ("beans.xml.dist", "High", 2), + ("beans.xml.template", "High", 2), ("build.gradle", "High", 2), ("cloudformation.json", "High", 1), ("cloudformation.json.template", DEFAULT_SEVERITY, 0), @@ -79,7 +80,7 @@ def test_detect_secrets_by_key(src, expected): ("custom.json", DEFAULT_SEVERITY, 0), ("custom.xml", DEFAULT_SEVERITY, 0), ("custom.yml", DEFAULT_SEVERITY, 0), - ("Dockerfile", "Medium", 3), + ("Dockerfile", "High", 3), ("empty.dockercfg", "Critical,High,Medium,Info", 0), ("excluded.json", "Critical", 0), ("excluded.xml", "Critical", 0), @@ -100,7 +101,7 @@ def test_detect_secrets_by_key(src, expected): ("java.properties", "High,Medium", 3), ("jdbc.xml", "High", 3), ("jenkins.xml", "High,Medium", 2), - ("nginx.conf", DEFAULT_SEVERITY, 4), + ("nginx.conf", "High,Medium", 3), ("page.html", "Info", 3), ("passwords.json", "High", 5), ("passwords.xml", "High", 5), @@ -111,13 +112,13 @@ def test_detect_secrets_by_key(src, expected): ("placeholders.xml", DEFAULT_SEVERITY, 0), ("placeholders.yml", DEFAULT_SEVERITY, 0), ("plaintext.txt", "High", 2), - ("private-pgp-block.txt", "High", 1), - ("privatekey.pem", "High", 1), - ("privatekeys.json", "High", 6), - ("privatekeys.xml", "High", 6), - ("privatekeys.yml", "High", 6), + ("private-pgp-block.txt", "Critical", 1), + ("privatekey.pem", "Critical", 1), + ("privatekeys.json", "Critical", 6), + ("privatekeys.xml", "Critical", 6), + ("privatekeys.yml", "Critical", 6), ("putty.ppk", DEFAULT_SEVERITY, 1), - ("ruleslist.yml", "High", 3), + ("ruleslist.yml", "High", 5), ("script.sh", "High,Medium", 15), ("settings.cfg", "High", 1), ("settings.conf", "High", 1), @@ -127,7 +128,7 @@ def test_detect_secrets_by_key(src, expected): ("severity.yml", "Critical", 1), ("sops.yml", DEFAULT_SEVERITY, 1), ("uri.yml", "High", 4), - ("webhooks.yml", "Low", 6), + ("webhooks.yml", "Medium", 6), ("falsepositive/values.yml", DEFAULT_SEVERITY, 4), ("falsepositive/plain.txt", DEFAULT_SEVERITY, 0), ("falsepositive/semver.json", DEFAULT_SEVERITY, 0), diff --git a/tests/unit/test_main.py b/tests/unit/test_main.py index cfb465b..668286c 100644 --- a/tests/unit/test_main.py +++ b/tests/unit/test_main.py @@ -15,13 +15,13 @@ def test_main(): @pytest.mark.parametrize( ("ast", "expected"), [ - ("--ast", 430), - ("", 318), + ("--ast", 434), + ("", 322), ], ) def test_run(ast, expected): if platform.startswith("win"): - expected = 318 + expected = 322 argv = ["-F", "None"] if ast: diff --git a/tests/unit/test_whispers.py b/tests/unit/test_whispers.py index 75301fa..d375fb8 100644 --- a/tests/unit/test_whispers.py +++ b/tests/unit/test_whispers.py @@ -10,7 +10,7 @@ (f"-c {config_path('integration.yml')} {fixture_path()}", 5), (f"-r apikey-known {fixture_path('apikeys-known.yml')}", 56), (f"-F None --rules file-known {fixture_path('files')}", 3), - (f"-s Critical {fixture_path('aws.yml')}", 3), + (f"-s Critical {fixture_path('aws.yml')}", 2), ], ) def test_whispers(args, expected): diff --git a/whispers/__version__.py b/whispers/__version__.py index a984d40..9812740 100644 --- a/whispers/__version__.py +++ b/whispers/__version__.py @@ -1,4 +1,4 @@ -VERSION = (2, 3, 1) +VERSION = (2, 4, 0) __version__ = ".".join(map(str, VERSION)) diff --git a/whispers/config.yml b/whispers/config.yml index 93f9084..2a18ee4 100644 --- a/whispers/config.yml +++ b/whispers/config.yml @@ -6,9 +6,8 @@ include: exclude: files: - - .*(__pycache__|\.eggs|build|dev|\.vscode|\.git) + - .*(__pycache__|\.eggs|build/|node_modules/|vendor/) - .*(locale|lang(uage)?|spec|test|mock(ing)?|synthetic|dummy|fixture|example|e2e)s? - - .*(integration|node_modules) - .*(package(-lock)?|npm-shrinkwrap)\.json keys: diff --git a/whispers/core/args.py b/whispers/core/args.py index da6562c..5b01026 100644 --- a/whispers/core/args.py +++ b/whispers/core/args.py @@ -5,8 +5,6 @@ from whispers.__version__ import __version__, __whispers__ from whispers.core.constants import DEFAULT_PATH -from whispers.core.utils import default_rules, load_regex -from whispers.plugins.semgrep import AST def argument_parser() -> ArgumentParser: @@ -70,6 +68,8 @@ def parse_args(arguments: list = argv[1:]) -> Namespace: args.src = args.src[0] if args.dump: + from whispers.plugins.semgrep import AST + print(AST.dump(args.src)) exit() @@ -90,6 +90,8 @@ def parse_args(arguments: list = argv[1:]) -> Namespace: args.files = args.files.split(",") if args.xfiles: + from whispers.core.utils import load_regex + args.xfiles = load_regex(args.xfiles) if args.rules: @@ -128,6 +130,8 @@ def show_config(): def show_info(): + from whispers.core.utils import default_rules + argument_parser().print_help() rules_table = [] col_width = 20 diff --git a/whispers/core/pairs.py b/whispers/core/pairs.py index 78831ce..03db2ad 100644 --- a/whispers/core/pairs.py +++ b/whispers/core/pairs.py @@ -6,22 +6,6 @@ from whispers.core.utils import global_exception_handler, is_static, strip_string from whispers.models.appconfig import AppConfig from whispers.models.pair import KeyValuePair -from whispers.plugins.config import Config -from whispers.plugins.dockercfg import Dockercfg -from whispers.plugins.dockerfile import Dockerfile -from whispers.plugins.elixir import Elixir -from whispers.plugins.gradle import Gradle -from whispers.plugins.html import Html -from whispers.plugins.htpasswd import Htpasswd -from whispers.plugins.jproperties import Jproperties -from whispers.plugins.json import Json -from whispers.plugins.npmrc import Npmrc -from whispers.plugins.pip import Pip -from whispers.plugins.plaintext import Plaintext -from whispers.plugins.pypirc import Pypirc -from whispers.plugins.shell import Shell -from whispers.plugins.xml import Xml -from whispers.plugins.yml import Yml def make_pairs(config: AppConfig, file: Path) -> Optional[Iterator[KeyValuePair]]: @@ -107,60 +91,96 @@ def load_plugin(file: Path, ast: bool = False) -> Optional[object]: """ file_name = file.name.lower() + logging.debug(f"load_plugin: ast:{ast} file:{file}") + if file.suffix.lower() in [".dist", ".template"]: filetype = file.stem.split(".")[-1].lower() else: filetype = file_name.split(".")[-1] if filetype in ["yaml", "yml"]: + from whispers.plugins.yml import Yml + return Yml elif filetype == "json": + from whispers.plugins.json import Json + return Json elif filetype == "xml": + from whispers.plugins.xml import Xml + return Xml elif filetype.startswith("npmrc"): + from whispers.plugins.npmrc import Npmrc + return Npmrc elif filetype.startswith("pypirc"): + from whispers.plugins.pypirc import Pypirc + return Pypirc elif file_name == "pip.conf": + from whispers.plugins.pip import Pip + return Pip elif file_name == "build.gradle": + from whispers.plugins.gradle import Gradle + return Gradle elif filetype in ["conf", "cfg", "cnf", "config", "ini", "credentials", "s3cfg"]: + from whispers.plugins.config import Config + return Config elif filetype == "properties": + from whispers.plugins.jproperties import Jproperties + return Jproperties elif filetype.startswith(("sh", "bash", "zsh", "env")) or file_name == "environment": + from whispers.plugins.shell import Shell + return Shell elif "dockerfile" in file_name: + from whispers.plugins.dockerfile import Dockerfile + return Dockerfile elif filetype == "dockercfg": + from whispers.plugins.dockercfg import Dockercfg + return Dockercfg elif filetype.startswith("htpasswd"): + from whispers.plugins.htpasswd import Htpasswd + return Htpasswd elif filetype == "txt": + from whispers.plugins.plaintext import Plaintext + return Plaintext elif filetype.startswith("htm"): + from whispers.plugins.html import Html + return Html elif filetype == "exs": + from whispers.plugins.elixir import Elixir + return Elixir elif REGEX_PRIVKEY_FILE.match(filetype): + from whispers.plugins.plaintext import Plaintext + return Plaintext elif ast and REGEX_AST_FILE.match(filetype): diff --git a/whispers/rules/keys.yml b/whispers/rules/keys.yml index 252539a..98453fa 100644 --- a/whispers/rules/keys.yml +++ b/whispers/rules/keys.yml @@ -2,7 +2,7 @@ group: keys description: Values resembling known API key formats message: API key - severity: Medium + severity: High key: regex: "^(?!.*\\s)[\\w\\d\\$\\.\\-_]+(key|token)$" ignorecase: True @@ -17,7 +17,7 @@ group: keys description: Known API key formats message: API key - severity: High + severity: Critical value: regex: "^(?!.*\\s)(\ akab|aio(?!http)|amzn\\.mws\\.|\ @@ -56,7 +56,7 @@ regex: .*aws.* ignorecase: False value: - regex: "^(?!.*\\s)(?!(.)\\1{11})[0-9]{12}$" + regex: "^(?!.*\\s)(?!(.)\\1{11})(?!.*1234.*)[0-9]{12}$" ignorecase: False @@ -64,7 +64,7 @@ group: keys description: Values formatted like AWS Access Key ID message: AWS Access Key ID - severity: Critical + severity: Medium value: regex: "^(?!.*\\s)(?=.*[A-Z])(?=.*[0-9])A(AGA|CCA|GPA|IDA|IPA|KIA|NPA|NVA|PKA|ROA|SCA|SIA|3T[A-Z0-9])[A-Z0-9]{16}$" ignorecase: False @@ -102,7 +102,7 @@ group: keys description: Text resembling a private key signature message: Private key - severity: High + severity: Critical value: regex: "[\\-]{5}BEGIN (RSA|DSA|EC|OPENSSH|PRIVATE|ENCRYPTED|PGP)? ?(PRIVATE)? KEY( BLOCK)?[\\-]{5}" ignorecase: False diff --git a/whispers/rules/misc.yml b/whispers/rules/misc.yml index 1335390..6e9dee3 100644 --- a/whispers/rules/misc.yml +++ b/whispers/rules/misc.yml @@ -38,7 +38,7 @@ group: misc description: Various known webhook formats message: Webhook - severity: Low + severity: Medium value: regex: "https?://(\ hooks\\.slack\\.com/services/[\\w\\d]{9}/[\\w\\d]{9}/[\\w\\d]+|\ diff --git a/whispers/rules/passwords.yml b/whispers/rules/passwords.yml index 8cd8f83..e29c15b 100644 --- a/whispers/rules/passwords.yml +++ b/whispers/rules/passwords.yml @@ -22,5 +22,5 @@ regex: ^uri_creds$ ignorecase: False value: - regex: "^\\w+:\\w+$" - minlen: 3 \ No newline at end of file + regex: "^[\\w\\-_\\.]+:[\\w\\-_\\.]+$" + minlen: 7 \ No newline at end of file