Field validation failing for array of objects

[andrewkroh]

I'm trying to address validation errors for a field that was declared as type: array. This array contains a list of objects. The fields within these objects are fully specified in the mapping. I thought I should be able to remove the field entry where type: array is used (like this). Given how Elasticsearch flattens arrays of objects, the intermediate objects should have been implicit (e.g. for a field foo.bar we don't require a foo entry). But when I do this the pipeline tests begin failing with field "powershell.command.invocation_details" is undefined.

Another potential fix would be to explicitly declare the invocation_details as type: object, but this solution would run afoul with elastic/package-spec#628. I could also make this nested, but I don't need that capability.

I think our goal with the validation is to ensure that we have mappings for all of the leaf fields. To meet that goal I think elastic-package needs to flatten arrays of objects like Elasticsearch does before applying the field validation.

What would you recommend to make this pipeline test field validation pass?

• Definition: https://github.com/elastic/integrations/blob/6719d691048bfd9803c98cd406beb51ae3653fac/packages/windows/data_stream/powershell/fields/fields.yml#L41-L63

• Sample Data: https://github.com/elastic/integrations/blob/75c6fada34f9c4f5b3b4fd81fceb574b8462dc54/packages/windows/data_stream/powershell_operational/_dev/test/pipeline/test-events.json-expected.json#L80-L82

---

This is a simplified example of the issue.

---
- name: foo.bar
  type: keyword

{
  "foo": [
    {
      "bar": "baz"
    },
    {
      "bar": "baz"
    }
  ]
}

[jsoriano]

What would you recommend to make this pipeline test field validation pass?

I think that the only option for this case would be to define invocation_details as a group and define the individual fields inside.

Using arrays of objects this way is actually tested, but not recommended, and its validation is not implemented.

I will try to slip in validation for these cases for Package Spec v3.