Merge branch 'master' into updating_rego_titles

CONTRIBUTING.md

@@ -3,16 +3,13 @@
 
 Welcome, and thank you for considering contributing to defsec!
 
-<p align="center">
-    <img alt="Welcome!" src="https://i.giphy.com/media/KczqttEJqm55hE1ccU/giphy.webp"/>
-</p>
-
-
 The following guide gives an overview of the project and some directions on how to make common types of contribution. If something is missing or you get stuck, please [jump on Slack](https://slack.aquasec.com/) or [start a discussion](https://github.com/aquasecurity/defsec/discussions/new) and we'll do our best to help.
 
 ## Project Overview
 
-_defsec_ is a library for defining security rules and policies in code, and the tools to apply those rules/policies to a variety of sources. The general architecture and project layout are defined in [ARCHITECTURE.md](ARCHITECTURE.md) - this is a great place to start exploring.
+_defsec_ is a library for defining security rules and policies in code, and the tools to apply those rules/cloud/policies to a variety of sources. The general architecture and project layout are defined in [ARCHITECTURE.md](ARCHITECTURE.md) - this is a great place to start exploring.
+
+_defsec_ is also the misconfiguration/IaC/Cloud scanning engine for Trivy. Trivy uses defsec internally as a library to perform various scans. 
 
 ## Guides
 
@@ -26,9 +23,9 @@ First of all, you should check if the provider your rule targets is supported by
 
 Next up, you'll need to check if the properties you want to target are supported, and if not, add support for them. The guide on [Adding Support for a New Service](#adding-support-for-a-new-service) covers adding new properties.
 
-At last, it's time to write your rule code! Rules are defined using _OPA Rego_. You can find a number of examples in the `rules/policies` directory. The [OPA documentation](https://www.openpolicyagent.org/docs/latest/policy-language/) is a great place to start learning Rego. You can also check out the [Rego Playground](https://play.openpolicyagent.org/) to experiment with Rego, and [join the OPA Slack](https://slack.openpolicyagent.org/).
+At last, it's time to write your rule code! Rules are defined using _OPA Rego_. You can find a number of examples in the `rules/cloud/policies` directory. The [OPA documentation](https://www.openpolicyagent.org/docs/latest/policy-language/) is a great place to start learning Rego. You can also check out the [Rego Playground](https://play.openpolicyagent.org/) to experiment with Rego, and [join the OPA Slack](https://slack.openpolicyagent.org/).
 
-Create a new file in `rules/policies` with the name of your rule. You should nest it in the existing directory structure as applicable. The package name should be in the format `builtin.PROVIDER.SERVICE.ID`, e.g. `builtin.aws.rds.aws0176`.
+Create a new file in `rules/cloud/policies` with the name of your rule. You should nest it in the existing directory structure as applicable. The package name should be in the format `builtin.PROVIDER.SERVICE.ID`, e.g. `builtin.aws.rds.aws0176`.
 
 Running `make id` will provide you with the next available _ID_ for your rule. You can use this ID in your rule code to identify it. 
 
@@ -63,9 +60,9 @@ deny[res] {
 }
 ```
 
-In fact, this is the code for an actual rule. You can find it in `rules/policies//aws/rds/enable_iam_auth.rego`. 
+In fact, this is the code for an actual rule. You can find it in `rules/cloud/policies/aws/rds/enable_iam_auth.rego`. 
 
-The metadata is the top section that starts with `# METADATA`, and is fairly verbose. You can copy and paste from another rule as a starting point. This format is effectively _yaml_ within a Rego comment, and is [defined as part of Rego itself](https://www.openpolicyagent.org/docs/latest/annotations/).
+The metadata is the top section that starts with `# METADATA`, and is fairly verbose. You can copy and paste from another rule as a starting point. This format is effectively _yaml_ within a Rego comment, and is [defined as part of Rego itself](https://www.openpolicyagent.org/docs/latest/policy-language/#metadata).
 
 Let's break the metadata down. 
 
@@ -97,7 +94,7 @@ The rule should return a result, which can be created using `result.new` (this f
 
 In the example above, you'll notice properties are being accessed from the `input.aws` object. The full set of schemas containing all of these properties is [available here](https://github.com/aquasecurity/defsec/tree/master/pkg/rego/schemas). You can match the schema name to the type of input you want to scan.
 
-You should also write a test for your rule(s). There are many examples of these in the `rules/policies` directory.
+You should also write a test for your rule(s). There are many examples of these in the `rules/cloud/policies` directory.
 
 Finally, you'll want to run `make docs` to generate the documentation for your new policy.
 

Makefile

@@ -1,4 +1,4 @@
-DYNAMIC_REGO_FOLDER=./rules/policies/kubernetes/policies/dynamic
+DYNAMIC_REGO_FOLDER=./rules/kubernetes/policies/dynamic
 
 .PHONY: test
 test:
@@ -17,7 +17,7 @@ schema:
 
 .PHONY: fmt-rego
 fmt-rego:
-	opa fmt -w rules/policies
+	opa fmt -w rules/cloud/policies
 
 .PHONY: test-rego
 test-rego:

go.mod

@@ -26,16 +26,16 @@ require (
 	github.com/aws/aws-sdk-go-v2/service/ec2 v1.89.1
 	github.com/aws/aws-sdk-go-v2/service/ecr v1.17.18
 	github.com/aws/aws-sdk-go-v2/service/ecs v1.18.26
-	github.com/aws/aws-sdk-go-v2/service/efs v1.20.1
-	github.com/aws/aws-sdk-go-v2/service/eks v1.22.1
+	github.com/aws/aws-sdk-go-v2/service/efs v1.20.3
+	github.com/aws/aws-sdk-go-v2/service/eks v1.27.14
 	github.com/aws/aws-sdk-go-v2/service/elasticache v1.26.8
 	github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.19.11
 	github.com/aws/aws-sdk-go-v2/service/elasticsearchservice v1.19.0
 	github.com/aws/aws-sdk-go-v2/service/emr v1.24.4
 	github.com/aws/aws-sdk-go-v2/service/iam v1.19.12
 	github.com/aws/aws-sdk-go-v2/service/kafka v1.19.4
 	github.com/aws/aws-sdk-go-v2/service/kinesis v1.15.19
-	github.com/aws/aws-sdk-go-v2/service/kms v1.21.1
+	github.com/aws/aws-sdk-go-v2/service/kms v1.23.0
 	github.com/aws/aws-sdk-go-v2/service/lambda v1.24.6
 	github.com/aws/aws-sdk-go-v2/service/mq v1.15.0
 	github.com/aws/aws-sdk-go-v2/service/neptune v1.20.7
@@ -57,7 +57,7 @@ require (
 	github.com/liamg/memoryfs v1.4.3
 	github.com/liamg/tml v0.6.0
 	github.com/mitchellh/go-homedir v1.1.0
-	github.com/moby/buildkit v0.11.4
+	github.com/moby/buildkit v0.11.6
 	github.com/olekukonko/tablewriter v0.0.5
 	// using this commit to avoid a bug in the latest version - replace when > v0.44.0 is released
 	github.com/open-policy-agent/opa v0.44.1-0.20220927105354-00e835a7cc15

go.sum

@@ -323,10 +323,10 @@ github.com/aws/aws-sdk-go-v2/service/ecr v1.17.18 h1:uiF/RI+Up8H2xdgT2GWa20YzxiK
 github.com/aws/aws-sdk-go-v2/service/ecr v1.17.18/go.mod h1:DQtDYmexqR+z+B6HBCvY7zK/tuXKv6Zy/IwOXOK3eow=
 github.com/aws/aws-sdk-go-v2/service/ecs v1.18.26 h1:EHJAYkUnlFJ/KwuFMvUs/bPbb0DaqAI+gTfXxffTPZ0=
 github.com/aws/aws-sdk-go-v2/service/ecs v1.18.26/go.mod h1:NpR78BP2STxvF/R1GXLDM4gAEfjz68W/h0nC5b6Jk3s=
-github.com/aws/aws-sdk-go-v2/service/efs v1.20.1 h1:fJFdGkf0Xj04IBnps3YlL6mGPSjzfgAZ7ZX+mbsrPDc=
-github.com/aws/aws-sdk-go-v2/service/efs v1.20.1/go.mod h1:7szMjYu35IWLaJEG23z6x/KOFckAF4/oQdNuDni99Pk=
-github.com/aws/aws-sdk-go-v2/service/eks v1.22.1 h1:f07Bk+xMm0Q8PCzvrBg8Bd6m67CTvZSxQWB0H7ZEJOU=
-github.com/aws/aws-sdk-go-v2/service/eks v1.22.1/go.mod h1:YoafRRQM4SnTFwb49e4LCAel6n99q2DMxkeAfbgvq8s=
+github.com/aws/aws-sdk-go-v2/service/efs v1.20.3 h1:+rQHxWkGK5GyanoetOyOG/U0sgXjlt3vw+jufY7wp4k=
+github.com/aws/aws-sdk-go-v2/service/efs v1.20.3/go.mod h1:UpiMmYILiWWe5wfcz6dJded9/K1XVmcOD3LB1ZCLVdw=
+github.com/aws/aws-sdk-go-v2/service/eks v1.27.14 h1:47HQVuJXgwvuoc4AT3rVdm77H0qGFbFnsuE4PRT+xX0=
+github.com/aws/aws-sdk-go-v2/service/eks v1.27.14/go.mod h1:QxuWcm9rlLkW3aEV8tiDzqZewnNSNUZfnqJvo1Nv9A0=
 github.com/aws/aws-sdk-go-v2/service/elasticache v1.26.8 h1:QBi/EzjmsZAm2qWCftCDHQ5DW1iDd4IwLDvXA6ikj8E=
 github.com/aws/aws-sdk-go-v2/service/elasticache v1.26.8/go.mod h1:HPSFSw7eCcEJFRaNZxnlMxOOctdrSNttq5JR/Q9yusU=
 github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.19.11 h1:IN2XMTLmhIEL5e3o+tY9JsLFSAxmjgM8gI7W2+CPrpw=
@@ -356,8 +356,8 @@ github.com/aws/aws-sdk-go-v2/service/kafka v1.19.4 h1:EmEk3gRJzSEK8DUnkvTGDn/RQc
 github.com/aws/aws-sdk-go-v2/service/kafka v1.19.4/go.mod h1:+O9qi0UC83Lk0KAnC/ixNcw4piXfUtPzXpYn/KC2Mhg=
 github.com/aws/aws-sdk-go-v2/service/kinesis v1.15.19 h1:qVaBkJxFxm6o/9DPNnJU6L9O3V7ycEKhCvRm2BFBQTU=
 github.com/aws/aws-sdk-go-v2/service/kinesis v1.15.19/go.mod h1:9rLNg+J9SEe7rhge/YzKU3QTovlLqOmqH8akb0IB1ko=
-github.com/aws/aws-sdk-go-v2/service/kms v1.21.1 h1:Q03Jqh1enA8keCiGZpLetpk58Ll9iGejE5bOErxyGAU=
-github.com/aws/aws-sdk-go-v2/service/kms v1.21.1/go.mod h1:EEfb4gfSphdVpRo5sGf2W3KvJbelYUno5VaXR5MJ3z4=
+github.com/aws/aws-sdk-go-v2/service/kms v1.23.0 h1:NXYeZBNg35rDBhcus60DFkIP7q6RNSkarLx+37ERX1g=
+github.com/aws/aws-sdk-go-v2/service/kms v1.23.0/go.mod h1:aNfh11Smy55o65PB3MyKbkM8BFyFUcZmj1k+4g8eNfg=
 github.com/aws/aws-sdk-go-v2/service/lambda v1.24.6 h1:N7RkXX2SJbN+TCp295J3LdMR0KRFd2Bhi5nIO+svLQY=
 github.com/aws/aws-sdk-go-v2/service/lambda v1.24.6/go.mod h1:oTJIIluTaJCRT6xP1AZpuU3JwRHBC0Q5O4Hg+SUxFHw=
 github.com/aws/aws-sdk-go-v2/service/mq v1.15.0 h1:Z1q5y9q4feZb/h2H4D+MHqGflTEeyZH6MarBG6o00ng=
@@ -867,8 +867,8 @@ github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RR
 github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
 github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
 github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
-github.com/moby/buildkit v0.11.4 h1:mleVHr+n7HUD65QNUkgkT3d8muTzhYUoHE9FM3Ej05s=
-github.com/moby/buildkit v0.11.4/go.mod h1:P5Qi041LvCfhkfYBHry+Rwoo3Wi6H971J2ggE+PcIoo=
+github.com/moby/buildkit v0.11.6 h1:VYNdoKk5TVxN7k4RvZgdeM4GOyRvIi4Z8MXOY7xvyUs=
+github.com/moby/buildkit v0.11.6/go.mod h1:GCqKfHhz+pddzfgaR7WmHVEE3nKKZMMDPpK8mh3ZLv4=
 github.com/moby/locker v1.0.1 h1:fOXqR41zeveg4fFODix+1Ch4mj/gT0NE1XJbp/epuBg=
 github.com/moby/locker v1.0.1/go.mod h1:S7SDdo5zpBK84bzzVlKr2V0hz+7x9hWbYC/kq7oQppc=
 github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8=

internal/adapters/cloud/aws/rds/rds.go

@@ -336,10 +336,15 @@ func (a *adapter) adaptDBSnapshots(dbSnapshots types.DBSnapshot) (*rds.Snapshots
 		DBSnapshotIdentifier: defsecTypes.String(*dbSnapshots.DBSnapshotIdentifier, metadata),
 		DBSnapshotArn:        defsecTypes.String(*dbSnapshots.DBSnapshotArn, metadata),
 		Encrypted:            defsecTypes.Bool(dbSnapshots.Encrypted, metadata),
-		KmsKeyId:             defsecTypes.String(*dbSnapshots.KmsKeyId, metadata),
+		KmsKeyId:             defsecTypes.String("", metadata),
 		SnapshotAttributes:   SnapshotAttributes,
 	}
 
+	// KMSKeyID is only set if Encryption is enabled
+	if snapshots.Encrypted.IsTrue() {
+		snapshots.KmsKeyId = defsecTypes.StringDefault(*dbSnapshots.KmsKeyId, metadata)
+	}
+
 	return snapshots, nil
 }
 

pkg/rego/scanner_test.go

@@ -886,3 +886,83 @@ deny {
 		"undefined ref: input.evil",
 	)
 }
+
+func Test_RegoScanning_CustomData(t *testing.T) {
+	srcFS := testutil.CreateFS(t, map[string]string{
+		"policies/test.rego": `
+package defsec.test
+import data.settings.DS123.foo_bar_baz
+
+deny {
+    not foo_bar_baz
+}
+`,
+	})
+
+	dataFS := testutil.CreateFS(t, map[string]string{
+		"data/data.json": `{
+	"settings": {
+		"DS123":{
+			"foo_bar_baz":false
+		}
+	}
+}`,
+		"data/junk.txt": "this file should be ignored",
+	})
+
+	scanner := NewScanner(types.SourceJSON)
+	scanner.SetDataFilesystem(dataFS)
+	scanner.SetDataDirs(".")
+
+	require.NoError(
+		t,
+		scanner.LoadPolicies(false, srcFS, []string{"policies"}, nil),
+	)
+
+	results, err := scanner.ScanInput(context.TODO(), Input{})
+	require.NoError(t, err)
+
+	assert.Equal(t, 1, len(results.GetFailed()))
+	assert.Equal(t, 0, len(results.GetPassed()))
+	assert.Equal(t, 0, len(results.GetIgnored()))
+}
+
+func Test_RegoScanning_InvalidFS(t *testing.T) {
+	srcFS := testutil.CreateFS(t, map[string]string{
+		"policies/test.rego": `
+package defsec.test
+import data.settings.DS123.foo_bar_baz
+
+deny {
+    not foo_bar_baz
+}
+`,
+	})
+
+	dataFS := testutil.CreateFS(t, map[string]string{
+		"data/data.json": `{
+	"settings": {
+		"DS123":{
+			"foo_bar_baz":false
+		}
+	}
+}`,
+		"data/junk.txt": "this file should be ignored",
+	})
+
+	scanner := NewScanner(types.SourceJSON)
+	scanner.SetDataFilesystem(dataFS)
+	scanner.SetDataDirs("X://")
+
+	require.NoError(
+		t,
+		scanner.LoadPolicies(false, srcFS, []string{"policies"}, nil),
+	)
+
+	results, err := scanner.ScanInput(context.TODO(), Input{})
+	require.NoError(t, err)
+
+	assert.Equal(t, 1, len(results.GetFailed()))
+	assert.Equal(t, 0, len(results.GetPassed()))
+	assert.Equal(t, 0, len(results.GetIgnored()))
+}

pkg/rego/store.go

@@ -15,9 +15,9 @@ import (
 func initStore(dataFS fs.FS, dataPaths, namespaces []string) (storage.Store, error) {
 	// FilteredPaths will recursively find all file paths that contain a valid document
 	// extension from the given list of data paths.
-	allDocumentPaths, err := loader.FilteredPathsFS(dataFS, dataPaths, func(abspath string, info os.FileInfo, depth int) bool {
+	allDocumentPaths, _ := loader.FilteredPathsFS(dataFS, dataPaths, func(abspath string, info os.FileInfo, depth int) bool {
 		if info.IsDir() {
-			return false
+			return false // filter in, include
 		}
 		ext := strings.ToLower(filepath.Ext(info.Name()))
 		for _, filter := range []string{
@@ -26,14 +26,11 @@ func initStore(dataFS fs.FS, dataPaths, namespaces []string) (storage.Store, err
 			".json",
 		} {
 			if filter == ext {
-				return false
+				return false // filter in, include
 			}
 		}
-		return true
+		return true // filter out, exclude
 	})
-	if err != nil {
-		return nil, fmt.Errorf("filter data paths: %w", err)
-	}
 
 	documents, err := loader.NewFileLoader().WithFS(dataFS).All(allDocumentPaths)
 	if err != nil {

pkg/scanners/azure/arm/parser/armjson/parse_comment.go

@@ -1,6 +1,10 @@
 package armjson
 
-import "github.com/aquasecurity/defsec/pkg/types"
+import (
+	"strings"
+
+	"github.com/aquasecurity/defsec/pkg/types"
+)
 
 func (p *parser) parseComment(parentMetadata *types.Metadata) (Node, error) {
 
@@ -32,7 +36,7 @@ func (p *parser) parseLineComment(parentMetadata *types.Metadata) (Node, error)
 
 	n, _ := p.newNode(KindComment, parentMetadata)
 
-	var comment string
+	var sb strings.Builder
 	for {
 		c, err := p.next()
 		if err != nil {
@@ -43,10 +47,10 @@ func (p *parser) parseLineComment(parentMetadata *types.Metadata) (Node, error)
 			p.position.Line++
 			break
 		}
-		comment += string(c)
+		sb.WriteRune(c)
 	}
 
-	n.raw = comment
+	n.raw = sb.String()
 	n.end = p.position
 
 	if err := p.parseWhitespace(); err != nil {
@@ -59,7 +63,7 @@ func (p *parser) parseBlockComment(parentMetadata *types.Metadata) (Node, error)
 
 	n, _ := p.newNode(KindComment, parentMetadata)
 
-	var comment string
+	var sb strings.Builder
 
 	for {
 		c, err := p.next()
@@ -74,17 +78,17 @@ func (p *parser) parseBlockComment(parentMetadata *types.Metadata) (Node, error)
 			if c == '/' {
 				break
 			}
-			comment += "*"
+			sb.WriteRune('*')
 		} else {
 			if c == '\n' {
 				p.position.Column = 1
 				p.position.Line++
 			}
-			comment += string(c)
+			sb.WriteRune(c)
 		}
 	}
 
-	n.raw = comment
+	n.raw = sb.String()
 
 	if err := p.parseWhitespace(); err != nil {
 		return nil, err