JSON Schema for test files

[alombarte]

This proposal is for:

• Creating a JSON Schema that validates that test files are written properly (it also helps with auto-completion in IDEs)
• Create a richer definition of tests
• Avoid code repetition

In the current implementation, the test command requires you to write a JSON file with the following fields:

• description
• schema
• tests An array with test objects:
  • description
  • data
  • valid

This format is functional and allows us to perform testing. Yet, tests might be unnecessarily verbose, as on one side, you declare all descriptions, and on the other, you might repeat yourself over and over in the data field.

Here is a fictional example of a test using this proposal:

{
  "$schema": "../jsonschema-test-improved.json",
  "target": "https://www.krakend.io/schema/v2.7/modifier/martian.json",
  "description": "Martian test case",
  "tests": [
    {
      "description": "full data must pass",
      "ref_data": "martian_valid_data.json",
      "valid": true
    },
    {
      "description": "empty object must fail",
      "data": {},
      "valid": false
    },
    {
      "subschema": "#/header.Modifier",
      "@comment": "Because schemas can be very extensive, we only want to test a subpath of the schema and define that part.",
      "data": {
        "scope": [
          "response"
        ],
        "name": "Cache-Control",
        "value": "max-age=600, public"
      },
      "valid": true
    }
  ]
}

It introduces the following changes:

• $schema: Because we should add to all tests the schema that validates this test suite, and this schema should be public as part of the JSON schema spec. E.g.: http://json-schema.org/draft-07/tests-schema#
• target: Formerly schema. Any other keyword would be fine, but I would avoid using $schema and schema simultaneously
• ref_data: It could be useful to pre-generate (massive) data in a separate process, reuse it, and inject it into tests. A problem I see with the current system, is that you can repeat a large content in the data section multiple times to complete a basic functional test. It is easy to end-up with a test case of 500 lines to test properly a modest schema.
• @comment: I have added a keyword to add comments (not the same as descriptions) to the tests
• subschema: This would open the door that a test case takes the responsibility of a single property. Just like a unit test can take responsibility of a function. Any JSON Schema out there is going to define multiple properties, many of which are mandatory. Why would you declare all this unnecessary data every time to test a part where the parent properties are not involved? It would be nice to test them one by one separately.

This is a draft of the JSON Schema that would validate this test suite:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "$id": "https://www.json-schema.org/draft-07/test-suite",
    "title": "JSON Schema for test suite files",
    "type": "object",
    "additionalProperties": false,
    "patternProperties": {
        "^[$]": {}
    },
    "required": [
        "tests",
        "target"
    ],
    "properties": {
        "description": {
            "type": "string"
        },
        "target": {
            "type": "string"
        },
        "tests": {
            "type": "array",
            "items": {
                "type": "object",
                "description": "Test case. Use @comment to add text that is not evaluated by the engine",
                "additionalProperties": false,
                "patternProperties": {
                    "^[@$]": {}
                },
                "oneOf": [
                    {
                        "required": [
                            "data",
                            "valid"
                        ]
                    },
                    {
                        "required": [
                            "ref_data",
                            "valid"
                        ]
                    }
                ],
                "properties": {
                    "description": {
                        "type": "string"
                    },
                    "data": {},
                    "ref_data": {
                        "type": "string",
                        "description": "Path to an external definition of the data"
                    },
                    "valid": {
                        "type": "boolean"
                    },
                    "subschema": {
                        "type": "string",
                        "description": "The path to a subschema allows you to test a small part of the schema"
                    }
                }
            }
        }
    }
}

This is just an idea, looking forward to hear thoughts

[jviotti]

All very valid points and workflows I didn't think about and we can do all of them. Expect to see PRs linking into this soon incrementally implementing various of these ideas.

Note that it is already possible to add comments (as the tool will ignore them), though maybe worth using $comment just to be a bit more compliant?

[alombarte]

I added the expression to accept both $comment and @comment, so using $comment is aligned with this thinking.

[jviotti]

I'm explicitly testing for and documenting $comment here: #124

[jviotti]

schema renamed to target here: #125

[jviotti]

I wanted to avoid adding a new keyword for testing subschemas, and came up with an alternative that is equally flexible and simple IMHO: letting target point to the specific subschemas directly. See #126. What do you think @alombarte ?

So in your example above, you can set:

{
    ....
    "target": "https://www.krakend.io/schema/v2.7/modifier/martian.json#/header.Modifier"
    ....
}

[jviotti]

I'm also implementing your ref_data idea here: #127, though I'm calling it dataPath to be more JSON Schema -ish