Query params cause combinatorial explosion in number of generated tests

[matus-m]

Description
Query params cause combinatorial explosion in number of generated tests. Using the existing MAX_TEST_REQUEST_COMBINATIONS slightly mitigates this, but most probably does not work the way most users would expect.

Steps to reproduce

1. Define a simple listing endpoint with a bunch of query params - in this case 10 query params, 2 of which are enums:

openapi: 3.0.3
info:
  title: My service
  description: My service
  version: 1.0.0
servers:
  - url: 'https://localhost:8080'
paths:
  /myapi:
    get:
      parameters:
        - name: first
          in: query
          required: false
          schema:
            type: string
        - name: second
          in: query
          required: false
          schema:
            type: string
        - name: third
          in: query
          required: false
          schema:
            type: string
            enum:
              - A
              - B
              - C
        - name: fourth
          in: query
          required: false
          schema:
            type: string
            enum:
              - RED
              - GREEN
              - BLUE
              - WHITE
              - BLACK
        - name: fifth
          in: query
          required: false
          schema:
            type: string
        - name: sixth
          in: query
          required: false
          schema:
            type: string
        - name: seventh
          in: query
          required: false
          schema:
            type: number
        - name: eighth
          in: query
          required: false
          schema:
            type: number
        - name: nineth
          in: query
          required: false
          schema:
            type: boolean
        - name: tenth
          in: query
          required: false
          schema:
            type: boolean
      responses:
        200:
          description: OK
          content:
            application/json:
              schema:
                type: string

2. set MAX_TEST_REQUEST_COMBINATION to 10
3. run the tests

Expected behavior
Most intuitive outcome should be 10 total requests generated. But instead 4864 testcases are generated. Even when set to 1, there are 1024 testcases generated.
Specmatic should either set a limit to the total number of generated testcases(that the user can override) or have some other safety net mechanism so that people dont smoke their laptops(imaging your api has more of these endpoints which is very common for any enterprise app that has search/listing endpoints)
The MAX_TEST_REQUEST_COMBINATION could be enhanced to really cover max request combinations in total, not only with regards to enums.

[harikrishnan83]

@yntelectual Thanks for bringing this up. I am moving this issue to a discussion format as we will need more ideas and inputs on this from you as well as other contributors and the community.

This is a known problem that we have been trying to tackle. Thanks to @westse, who introduced MAX_TEST_REQUEST_COMBINATIONS to reduce this to an extent. However, I guess the MAX_TEST_REQUEST_COMBINATIONS seems to be applying only to the body and not to query parameters. I will double check on this.

Before going into solutioning, just wanted to step back and look at the problem with optional fields in general. The issue you have highlighted with the query parameters is an extension of the same.

• In order to verify if a field is optional we need to generate two tests, one where we send the field and another where we do not. And we expect both these requests to have a 2xx response in order to ascertain the optionality.
• Extrapolating this to multiple optional fields, we have to assume that each field is independently optional. Thereby creating this explosion of combinations.
• API designs which have several optional fields with inter-parameter dependency creates another level of complexity.
• Enums add to the combinations

If we limit the total number of combinations generated, then we may not be able to ascertain the optional nature of all fields. We are looking into creating the most meaningful combinations and bubble that to the top so that even with a low MAX_TEST_REQUEST_COMBINATIONS we get some useful feedback. Do share your thoughts on this.

In my experience, where I hit this sort of an issue with several optional parameters and inter-parameter dependency, it may point to a API design smell. And I may explore / reevaluate my API design. That said, I do acknowledge there are API designs which need to go down this path of several optional parameters because of genuine constraints.

[philmadden83]

@harikrishnan83 @yntelectual i would tend to agree that restricting the number of generated test through a flag would only reduce the confidence that the provider has indeed implemented the specification correctly.

I would imagine that you could reduce the number of generated tests by being more descriptive in the specification itself, i.e by adding min/max ranges for numbers, adding example request/responses and reducing the number of plain text query params in favor of enums. This should do three things:

• increase the quality of the contract
• protect the API better
• reduce the number of tests generated

thoughts?

[joelrosario]

@philmadden83 good points, and I agree. A more descriptive (and a more specific) specification (pun not intended) would be easier both to understand and test.

Ranges are great and should be added added for sure. Note though that they provide bounds for a number, while the number of tests increases when the number of datatypes increase. For example, when a JSON value can be null or a number, there are two data types, that is to say two types of values. If another JSON value in this same payload can be null or string this means there are 4 combinations in total, resulting in 4 tests. If both values in the payload were just numbers, then there's just one possible combination, resulting in just 1 test. And as @harikrishnan83 mentioned, this could point to a problem in the API design, which has the follow-on effects of making the API harder to understand, and thereby develop and test.

Examples are great way to make sense of optionality in an application. They also act to constrain the count of tests, as Specmatic uses examples if they are present rather than generating tests. Of course this means that the provider now has to make sure to add enough examples that cover all possible requests, and their expected responses. This is where an approach to generating tests from the specification that prioritise the most meaningful combinations would help, and as @harikrishnan83 said is something we're looking into. Let us know if you have any thoughts on this.

[philmadden83]

@joelrosario @yntelectual can we take the contract that is defined above and make it more descriptive to see how effective my points would be? I can recommend Spectral which helps lint OpenAPI specs.

I'd be interested to see how many of the 4864 generated tests we can remove....any why.

fwiw @yntelectual I ran out of memory testing my contract. #729

[joelrosario]

We're exploring a few options here, and will report back soon.

[matus-m]

I agree that the generated scenarios covering all allowed combinations are useful. However I believe that are many scenarios in which a perfectly valid and well designed schema offers an endpoint with a significant number of query params - which would result in such a number of tests, that it might overwhelm the test runner. Maybe the simplest solution to avoid this would be to define a global safeguard limit on number of generated tests(2000?), which if its reached would cause test run to fail with a descriptive error message, with information about number of generated scenarios per endpoint/operation - giving the user an option to restrict the number of scenarios using examples, or split the test run into multiple ones, or lifting that limit. User would be able to override this limit using a simple config property, e.g. MAX_GENERATED_SCENARIOS=-1 would mean no limit (current scenario)

[joelrosario]

Thanks @yntelectual that's a pretty interesting idea. We'll get back soon.

[philmadden83]

@joelrosario do you have any updates on this?