conftest read json file failed due to json file contains BOM

[jinweili1]

Hi,

We met some problems when using conftest. If the json file contains BOM, conftest will throw Error: running test: get configurations: get configurations: parser unmarshal: unmarshal json: invalid character 'ï' looking for beginning of value error.
After removing them, conftest works.
So I think conftest need to support it, and could you fix this problem?

[boranx]

Hi,

Thanks for reporting the issue.
That sounds like a small enhancement to yaml/json parser
Is there an example JSON you could share with us that contains BOM? In a way, it can be used in the testing to prove conftest can parse requested form

[jinweili1]

sure, I will share it
testBOM.zip

[jalseth]

This is an upstream issue with the Go JSON parser, it is not specific to conftest. I don't think we should update conftest with custom code for this, the JSON supplied to conftest should be compatible with Go's JSON parser.

$ jq '.' testBOM.json
{
    "test":  "test"
}

$ xxd testBOM.json
00000000: efbb bf7b 0d0a 2020 2020 2274 6573 7422  ...{..    "test"
00000010: 3a20 2022 7465 7374 220d 0a7d 0d0a       :  "test"..}..

$ conftest verify -p foo.rego -d testBOM.json
Error: running verification: load: load documents: 1 error occurred during loading: testBOM.json: invalid character 'ï' looking for beginning of value

$ jq '.' foo.json
{
  "foo": "bar"
}

$ xxd foo.json
00000000: 7b0a 2020 2266 6f6f 223a 2022 6261 7222  {.  "foo": "bar"
00000010: 0a7d 0a                                  .}.

$ conftest verify -p foo.rego -d foo.json

1 test, 1 passed, 0 warnings, 0 failures, 0 exceptions, 0 skipped

$ cat foo.go
package main

import (
  "fmt"
  "os"

  "encoding/json"
)

func main() {
  bytes, err := os.ReadFile("testBOM.json")
  if err != nil {
    panic(err)
  }
  j := make(map[string]any)
  if err := json.Unmarshal(bytes, j); err != nil {
    panic(err)
  }
  fmt.Printf("%+v\n", j)
}

$ go run foo.go
panic: invalid character 'ï' looking for beginning of value

goroutine 1 [running]:
main.main()
	/Users/james/Downloads/foo.go:17 +0xc4
exit status 2

@jinweili1 At this point, I think you can update your JSON input to not have the extra bytes, or file a bug upstream with the Go maintainers.