Report digest and signature

[2byrds]

A discussion to standardize the digest and signature for the reg pilot. @tibor19 @daviddm @lenkan @aydarng @dawgithub @Arsh-Sandhu

We are converging towards a the following report.json in the META-INF directory of the report zip:

{
    "documentInfo": {
        "documentType": "http://xbrl.org/PWD/2020-12-09/report-package",
        "signatures": [
            {
                "file": "../reports/DUMMYLEI123456789012.CON_FR_PILLAR3010000_CONDIS_2023-12-31_20230516103116000.xbrl",
                "aid": "EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw",
                "sigs": [
                    "AACMB3EdY5YEqphXmlt-86vLprnlaMXdWNuxnsPhHwF3N45g_SOOea3X6spezBB2nvyP8HZ0lhpGCfI_dvrP2uMJ"
                ],
                "digest": "e1bce01400cebb494ae68ec295e3faed859132271522486499fc801ebf2a8818"
            }
        ]
    }
}

1. The documentType is the report-package definition.
2. The signatures list contains:
   1. file: the relative path to the file for which the digest and signature are generated.
   2. aid: the AID that generated the sigs list.
   3. sigs: the list of signatures that satisfies the signing threshold of the AID.
   4. digest: the digest of the binary of the file referenced in the file field.

The initial questions to discuss:

1. How is the digest generated/verified? Lets use signify-ts code to ground this.
   1. For instance see a simple KERI Diger test here https://github.com/WebOfTrust/signify-ts/blob/850af59e4a6e136f2e9393d86c4edb5819904d79/test/core/diger.test.ts#L12
2. Is the signature on the digest? This has been discussed previously as the most straight forward solution, but lets settle on that.
   1. We should end up with simple tests like this, but the ser is the digest of the file bytes. https://github.com/WebOfTrust/signify-ts/blob/850af59e4a6e136f2e9393d86c4edb5819904d79/test/core/signer.test.ts#L23

[lenkan]

Lets use signify-ts code to ground this.

Are you suggesting to use a blake3 digest then? As defined here: https://github.com/WebOfTrust/signify-ts/blob/850af59e4a6e136f2e9393d86c4edb5819904d79/test/core/diger.test.ts#L17.

Does this have a significant advantage over the browser native API https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest ?

[2byrds]

Digital Signatures in XBRL 1.0 specification recommended to express digest as Digest String.

A digest string is a string encapsulating a digest algorithm and digest value. A digest string consists of a digest algorithm prefix and a lower case hexadecimal digest value, separated by the - character. The digest algorithm prefix MUST be the prefix for a supported digest type.
The supported digest methods are SHA-256, SHA-384 and SHA-512

Example digest string:

sha256-7b7fd29647c0ced9a71541ca2eb041d07fdbafb46d1f193d2c4b02ac0a059f7e

We can also use the same format and digest algorithm here also.

@lenkan to start, are you okay with sha256 prefixed digest string like what the spec lists? We can systematically tackle the suggestions from Arsh regarding packaging and fields from the XBRL spec after this.

[lenkan]

@2byrds that sounds like a good first step!

[daviddm]

@Arsh-Sandhu @2byrds
When reading the spec I can't find what you're supposed to sign, maybe it's obvious to you guys.
Should the file digest('a1b2c3...') or should the entire embedded digest string('sha256-a1b2c3...') be signed?

[2byrds]

My intuition is to sign only the digest, without the prefix

[daviddm]

So does mine, but I don't trust it.
I'm gonna change the current embedded digest string to include the algorithm prefix, without changing the report envelop. That will be a later decision.

[lenkan]

The documentType is the report-package definition.

Is the documentType URL supposed to be a real resource on the internet? http://xbrl.org/PWD/2020-12-09/report-package is a 404 at the moment.

[2byrds]

I'm not as familiar with the XBRL formats. I have included @Arsh-Sandhu in our conversation as he is much more well-versed on XBRL and can also help us be forward looking in our file signing.

[Arsh-Sandhu]

The documentType depends on the type and number of reports in a report package. Since you are using the report.json file and not the reportPackage.json file, as per the report-package specification, if the META-INF/reportPackage.json file is absent, then the type of report package is considered an unconstrained report package, and the value of the /documentInfo/documentType property should be https://xbrl.org/report-package/2023.

I also want to point out that the specification mentions the unconstrained report package type exists primarily for backward compatibility and SHOULD NOT be used when packaging individual reports. Instead, the report package should include a file called reportPackage.json in the META-INF directory.

[2byrds]

@Arsh-Sandhu excellent and thank you for the additional details below.

[2byrds]

Thank you @lenkan for posting the digest difference (blake vs. sha) here GLEIF-IT/vlei-verifier#10 (comment)

[Arsh-Sandhu]

@2byrds have we considered using Digital Signatures in XBRL 1.0 specification for signing xBRL-CSV files?

A xBRL-CSV report package signed as per this spec will look like this:

example-signed-csv-report/
├─ META-INF/
│  ├─ reportPackage.json
│  ├─ d6.json
│  ├─ signatures/
│  │  ├─ target_report.json
│  │  ├─ signature_report.cesr
│  │  ├─ target_file1.json
│  │  ├─ signature_file1.cesr
│  │  ├─ target_file2.json
│  │  └─ signature_file2.cesr
└─ reports/
   ├─ report.json
   ├─ file1.csv
   └─ file2.csv            

d6.json file

{
  "documentInfo": {
    "documentType": "https://xbrl.org/CR/2024-05-15/d6"
  },
  "signatures": [
    {
      "filename": "./signature/signature_report.cesr",
      "type": "https://weboftrust.org/d6/cesr",
      "targets": ["./signature/target_report.json"],
      "aid": "EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw"
    },
    {
      "filename": "./signature/signature_file1.cesr",
      "type": "https://weboftrust.org/d6/cesr",
      "targets": ["./signature/target_file1.json"],
      "aid": "EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw"
    },
    {
      "filename": "./signature/signature_file2.cesr",
      "type": "https://weboftrust.org/d6/cesr",
      "targets": ["./signature/target_file2.json"],
      "aid": "EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw"
    }  
  ]
}

report.json target file

{
  "documentInfo": {
    "documentType": "https://xbrl.org/CR/2024-05-15/d6/target"
  }, 
  "report": "../../reports/report.json",
}

file1.csv target file

{
  "documentInfo": {
    "documentType": "https://xbrl.org/CR/2024-05-15/d6/target"
  }, 
  "report": "../../reports/file1.csv",
}

signature_file1.cesr

-FABENib6xC8YfUOzCTqnNVf1H3UEOXaLpGGEDcDAG7mRoWf0AAAAAAAAAAAAAAAAAAAAAAAENib6xC8YfUOzCTqnNVf1H3UEOXaLpGGEDcDAG7mRoWf-AABAACbQBYot59torE3XNKI67CQ0F_W19RJzITfiAPFy6fQ-nH0ZwHFfBJ7bFNuuSFwSIqzWprNAa1kdiNUrBEWAXYM

[2byrds]

@Arsh-Sandhu thank you for sharing! I'll have to check with EBA about their processing/workflow to see what report formatting they are expecting. Ideally we land on this spec sooner or later

[2byrds]

@bogtieba @daviddm @lenkan @tibor19 @Arsh-Sandhu @rodolfomiranda we have finished the first part of our transition to the agreed upon sha256 implementation. We no longer use Blake3, we now use sha256 prefixed hashes. We still need to sign the digest, instead of signing the file bytes (i'll post again when thats ready). And unless we hear otherwise, we plan to sign only the digest, not the prefixed digest.

The following PRs from @aydarng (and small fix PRs) we have upgraded the vlei-verifier, reg-pilot-api, and reg-pilot tests to use/validate sha256 prefixed digests:
https://github.com/GLEIF-IT/vlei-verifier/pull/26/files
GLEIF-IT/reg-pilot-api#31
#53

I've updated the test images. We'll be transitioning to gleif docker hub soonish:
2byrds/reg-pilot-api:latest
2byrds/vlei-verifier:latest

[2byrds]

@bogtieba @daviddm @lenkan @tibor19 @Arsh-Sandhu @rodolfomiranda @aydarng we completed the second round of changes to sign the non-prefixed digest. We have also migrated the image pushes to be in the gleif docker hub:
https://hub.docker.com/r/gleif/reg-pilot-api/tags
https://hub.docker.com/r/gleif/vlei-verifier/tags

Any merged pull request in those repos will automatically post a new 'dev' image.

Next up, we will compare our digests and signatures with those produced by NordLEI to make sure we have converged. Then we'll make the verifier a little more flexible to allow non-parent-folder zips.