Skip to content

Commit

Permalink
feat: flagd gherkin suite and flags
Browse files Browse the repository at this point in the history
Signed-off-by: Todd Baert <[email protected]>
  • Loading branch information
toddbaert committed Sep 5, 2023
1 parent a463f8f commit 4b8cdbb
Show file tree
Hide file tree
Showing 8 changed files with 253 additions and 49 deletions.
4 changes: 3 additions & 1 deletion .gherkin-lintrc
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@
"Feature": 0,
"Background": 2,
"Scenario": 2,
"Step": 4
"Step": 4,
"Examples": 4,
"example": 6
}
],
"no-trailing-spaces": "on",
Expand Down
18 changes: 17 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,20 @@ See the [flagd docs](https://flagd.dev/) for more information on flagd.
The _sync_-testbed_ container is a docker image built on conforming to the sync.proto - a grpc which flagd or flagd in-process providers can use as a sync-source.
It features an identical set of flags to the [flagd-testbed container](#flagd-testbed-container)

For details on the sync-testbed, see [sync/README.me](sync/README.md)
For details on the sync-testbed, see [sync/README.me](sync/README.md)

## Gherkin test suite

The [gherkin/](gherkin/) dir includes a set of [_gherkin_](https://cucumber.io/docs/gherkin/) tests that define expected behavior associated with the configurations defined in the flagd-testbed (see [flags/](flags/)).
Combined with the _flagd-provider_ for the associated SDK and the flagd-testbed, these comprise a complete integration test suite.
They include tests for:
* default value (zero-value) handling (some proto3 implementations handle these inconsistently).
* basic event handling
* custom JSONLogic operators (`starts_with`, `ends_with`, `fractional`, `sem_ver`)

### Lint Gherkin files

The Gherkin files structure can be linted using [gherkin-lint](https://github.com/vsiakka/gherkin-lint). The following commands require Node.js 10 or later.

1. npm install
1. npm run gherkin-lint
9 changes: 7 additions & 2 deletions flagd/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
FROM ghcr.io/open-feature/flagd:v0.6.3 as flagd
# we NEED flagd v0.6.4 as a minimum
FROM ghcr.io/open-feature/flagd:v0.6.4 as flagd

FROM busybox:1.36

Expand All @@ -7,4 +8,8 @@ COPY flags/* .
COPY scripts/* .
LABEL org.opencontainers.image.source = "https://github.com/open-feature/test-harness"

ENTRYPOINT ["sh", "wrapper.sh", "./flagd", "start", "-f", "file:testing-flags.json", "-f", "file:changing-flag.json"]
ENTRYPOINT ["sh", "wrapper.sh", "./flagd", "start", \
"-f", "file:testing-flags.json", \
"-f", "file:changing-flag.json", \
"-f", "file:custom-ops.json", \
"-f", "file:zero-flags.json"]
104 changes: 104 additions & 0 deletions flags/custom-ops.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
{
"flags": {
"fractional-flag": {
"state": "ENABLED",
"variants": {
"clubs": "clubs",
"diamonds": "diamonds",
"hearts": "hearts",
"spades": "spades",
"wild": "wild"
},
"defaultVariant": "wild",
"targeting": {
"fractional": [
{ "var": "user.name" },
[ "clubs", 25 ],
[ "diamonds", 25 ],
[ "hearts", 25 ],
[ "spades", 25 ]
]
}
},
"starts-ends-flag": {
"state": "ENABLED",
"variants": {
"prefix": "prefix",
"postfix": "postfix",
"none": "none"
},
"defaultVariant": "none",
"targeting": {
"if": [
{
"starts_with": [{"var": "id"}, "abc"]
},
"prefix", {
"if": [
{
"ends_with": [{"var": "id"}, "xyz"]
},
"postfix", "none"
]
}
]
}
},
"equal-greater-lesser-version-flag": {
"state": "ENABLED",
"variants": {
"equal": "equal",
"greater": "greater",
"lesser": "lesser",
"none": "none"
},
"defaultVariant": "none",
"targeting": {
"if": [
{
"sem_ver": [{"var": "version"}, "=", "2.0.0"]
},
"equal", {
"if": [
{
"sem_ver": [{"var": "version"}, ">", "2.0.0"]
},
"greater", {
"if": [
{
"sem_ver": [{"var": "version"}, "<", "2.0.0"]
},
"lesser", "none"
]
}
]
}
]
}
},
"major-minor-version-flag": {
"state": "ENABLED",
"variants": {
"minor": "minor",
"major": "major",
"none": "none"
},
"defaultVariant": "none",
"targeting": {
"if": [
{
"sem_ver": [{"var": "version"}, "~", "3.0.0"]
},
"minor", {
"if": [
{
"sem_ver": [{"var": "version"}, "^", "3.0.0"]
},
"major", "none"
]
}
]
}
}
}
}
44 changes: 0 additions & 44 deletions flags/testing-flags.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@
},
"defaultVariant": "on"
},
"boolean-flag-copy": {
"state": "ENABLED",
"variants": {
"on": true,
"off": false
},
"defaultVariant": "on"
},
"string-flag": {
"state": "ENABLED",
"variants": {
Expand All @@ -24,14 +16,6 @@
},
"defaultVariant": "greeting"
},
"string-flag-copy": {
"state": "ENABLED",
"variants": {
"greeting": "hi",
"parting": "bye"
},
"defaultVariant": "greeting"
},
"integer-flag": {
"state": "ENABLED",
"variants": {
Expand All @@ -40,14 +24,6 @@
},
"defaultVariant": "ten"
},
"integer-flag-copy": {
"state": "ENABLED",
"variants": {
"one": 1,
"ten": 10
},
"defaultVariant": "ten"
},
"float-flag": {
"state": "ENABLED",
"variants": {
Expand All @@ -56,14 +32,6 @@
},
"defaultVariant": "half"
},
"float-flag-copy": {
"state": "ENABLED",
"variants": {
"tenth": 0.1,
"half": 0.5
},
"defaultVariant": "half"
},
"object-flag": {
"state": "ENABLED",
"variants": {
Expand All @@ -76,18 +44,6 @@
},
"defaultVariant": "template"
},
"object-flag-copy": {
"state": "ENABLED",
"variants": {
"empty": {},
"template": {
"showImages": true,
"title": "Check out these pics!",
"imagesPerPage": 100
}
},
"defaultVariant": "template"
},
"context-aware": {
"state": "ENABLED",
"variants": {
Expand Down
36 changes: 36 additions & 0 deletions flags/zero-flags.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"flags": {
"boolean-zero-flag": {
"state": "ENABLED",
"variants": {
"zero": false,
"non-zero": true
},
"defaultVariant": "zero"
},
"string-zero-flag": {
"state": "ENABLED",
"variants": {
"zero": "",
"non-zero": "str"
},
"defaultVariant": "zero"
},
"integer-zero-flag": {
"state": "ENABLED",
"variants": {
"zero": 0,
"non-zero": 1
},
"defaultVariant": "zero"
},
"float-zero-flag": {
"state": "ENABLED",
"variants": {
"zero": 0.0,
"non-zero": 1.0
},
"defaultVariant": "zero"
}
}
}
81 changes: 81 additions & 0 deletions gherkin/flagd.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
Feature: flagd providers

# This test suite contains scenarios to test flagd providers (both RPC and in-process).
# It's associated with the flags configured in flags/custom-ops.json and flags/zero-flags.json
# It should be used in conjunection with the suites supplied by the OpenFeature specification

Background:
Given a flagd provider is set

# events
Scenario: Provider ready event
When a PROVIDER_READY handler is added
Then the PROVIDER_READY handler must run

Scenario: Flag change event
When a PROVIDER_CONFIGURATION_CHANGED handler is added
And a flag with key "changing-flag" is modified
Then the PROVIDER_CONFIGURATION_CHANGED handler must run
And the event details must indicate "changing-flag" was altered

# zero evaluation
Scenario: Resolves boolean zero value
When a boolean flag with key "boolean-zero-flag" is evaluated with default value "true"
Then the resolved boolean value should be "false"

Scenario: Resolves string zero value
When a string flag with key "string-zero-flag" is evaluated with default value "hi"
Then the resolved string value should be ""

Scenario: Resolves integer zero value
When an integer flag with key "integer-zero-flag" is evaluated with default value 1
Then the resolved integer value should be 0

Scenario: Resolves float zero value
When a float flag with key "float-zero-flag" is evaluated with default value 0.1
Then the resolved float value should be 0.0

# custom operators
Scenario Outline: Fractional operator
When a string flag with key "fractional-flag" is evaluated with default value "fallback"
And a context containing a nested property with outer key "user" and inner key "name", with value <name>
Then the returned value should be <value>
Examples:
| name | value |
| jack | clubs |
| queen | diamonds |
| ace | hearts |
| joker | spades |

Scenario Outline: Substring operators
When a string flag with key "starts-ends-flag" is evaluated with default value "fallback"
And a context containing a key "id", with value <id>
Then the returned value should be <value>
Examples:
| id | value |
| abcdef | prefix |
| uvwxyz | postfix |
| abcxyz | prefix |
| lmnopq | nomatch |

Scenario Outline: Semantic version operator numeric comparision
When a string flag with key "equal-greater-lesser-version-flag" is evaluated with default value "fallback"
And a context containing a key "version", with value <version>
Then the returned value should be <value>
Examples:
| version | value |
| 2.0.0 | equal |
| 2.1.0 | greater |
| 1.9.0 | lesser |
| 2.0.0-alpha | lesser |
| 2.0.0.0 | invalid |

Scenario Outline: Semantic version operator semantic comparision
When a string flag with key "major-minor-version-flag" is evaluated with default value "fallback"
And a context containing a key "version", with value <version>
Then the returned value should be <value>
Examples:
| version | value |
| 3.0.1 | minor |
| 3.1.0 | major |
| 4.0.0 | none |
6 changes: 5 additions & 1 deletion sync/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,8 @@ COPY flags/* .
COPY scripts/* .
LABEL org.opencontainers.image.source = "https://github.com/open-feature/test-harness"

ENTRYPOINT ["sh", "wrapper.sh", "./sync", "start", "-f", "testing-flags.json", "-f", "changing-flag.json"]
ENTRYPOINT ["sh", "wrapper.sh", "./sync", "start", \
"-f", "testing-flags.json", \
"-f", "changing-flag.json", \
"-f", "custom-ops.json", \
"-f", "zero-flags.json"]

0 comments on commit 4b8cdbb

Please sign in to comment.