tpm_eventlog shouts "WARN: Event 3's digest does not match its payload" for PCR0 measurements

[Sinihopea]

Hello,

Me and my co-worker have been hitting our heads into a wall trying to figure this weird issue with Dell PowerEdge XR4520c recently when using SUSE Enterprise Linux Server 15 SP4, SP5 and SP6. When parsing the /sys/kernel/security/tpm0/binary_bios_measurements... the reconstruction of the PCR0 shouts out warning...

WARN: Event 3's digest does not match its payload

...meaning in our case that the measured EV_S_CRTM_VERSION digest doesn't match the measured BIOS version. Or at least I think it should match the UEFI / BIOS version string of the XR4520c based on how this is explained in TCG's TPM 2.0 documentation regarding PCR measurements.

We have no idea why that is the case. It occurs in every Dell PowerEdge XR4520c compute sled we have tested. Other devices (laptops, servers, compute nodes, etc.) do not give such warning or error when tpm2_eventlog is run.

We've configured this compute sled according to Dell's instructions and upgraded firmware(s) to all the latest releases. TPM is configured to use SHA-256 for PCRs as per recommendations. It's notable that tpm2_eventlog gives this warning for PCR0 not only from TCG binary log captured from SUSE Linux but also when parsing Windows Server 2022's TCG binary log. Which it should do from what I understand.

After spending days on testing this in various ways... I am willing to come to a conclusion that this must be perhaps a some sort of bug in the tpm2_eventlog utility. It probably doesn't know how to parse this particular Nuvoton NPCT 75x TPM chip's binary log... or maybe there's a bug in TPM firmware itself. Or Dell has something weird going-on in their BIOS version string and it changes when booting compute sled.

My co-worker pulled the latest tpm2.0-tools from git master branch, compiled the tools and that did not change anything regarding this warning we see here. Is there anything we could do to help anyone figure out why this happens and how it could be addressed? It's undermining our ability to trust this hardware platform.

Thank you!

[Sinihopea]

I'm going to add that this sounds similar to an issue that is currently open in fwupd / firmware-dell... fwupd/firmware-dell#34

[JuergenReppSIT]

@Sinihopea what is displayed if you exec:
tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements | grep CRTM -A 10

[Sinihopea]

@Sinihopea what is displayed if you exec: tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements | grep CRTM -A 10

@JuergenReppSIT I reinstalled the SLES15 SP6 QU2 into the XR4520c. When using the SUSE's tpm2.0-tools package:

tool="tpm2_eventlog" version="5.2" tctis="libtss2-tctildr" tcti-default=tcti-abrmd

This is what I get...

localhost:~ # tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements|grep -A10 CRTM
WARN: Event 3's digest does not match its payload
  EventType: EV_S_CRTM_CONTENTS
  DigestCount: 1
  Digests:
  - AlgorithmId: sha256
    Digest: "92c329cbcc6177a0f4268c4044fd7bd2c38021513cf0cefb98d74e7de19f52a2"
  EventSize: 27
  Event:
    BlobBase: 0x61754720746f6f42
    BlobLength: 0x757361654d206472
- EventNum: 3
  PCRIndex: 0
  EventType: EV_S_CRTM_VERSION
  DigestCount: 1
  Digests:
  - AlgorithmId: sha256
    Digest: "76d538bb3a43ef27b2e9ccca3b0bad02b9ee15ffbf0dadbdfa823dbf43d83a1f"
  EventSize: 14
  Event: "31002e00310036002e0032000000"
- EventNum: 4
  PCRIndex: 0
  EventType: EV_EFI_PLATFORM_FIRMWARE_BLOB
  DigestCount: 1

I also tested tpm2.0-tools package which my co-worker compiled:

tool="tpm2_eventlog" version="v5.7.9999" tctis="libtss2-tctildr" tcti-default=tcti-abrmd

And this is what it outputs...

WARN: Event 18446744073709551615's digest does not match its payload
  EventType: EV_S_CRTM_CONTENTS
  DigestCount: 1
  Digests:
  - AlgorithmId: sha256
    Digest: "92c329cbcc6177a0f4268c4044fd7bd2c38021513cf0cefb98d74e7de19f52a2"
  EventSize: 27
  Event:
    BlobBase: 0x61754720746f6f42
    BlobLength: 0x757361654d206472
- EventNum: 3
  PCRIndex: 0
  EventType: EV_S_CRTM_VERSION
  DigestCount: 1
  Digests:
  - AlgorithmId: sha256
    Digest: "76d538bb3a43ef27b2e9ccca3b0bad02b9ee15ffbf0dadbdfa823dbf43d83a1f"
  EventSize: 14
  Event: "31002e00310036002e0032000000"
- EventNum: 4
  PCRIndex: 0
  EventType: EV_EFI_PLATFORM_FIRMWARE_BLOB
  DigestCount: 1

It's notable that when I run my co-worker's version after RPM is installed and abrmd is restarted... it will change the number (I guess from unsigned 64-bit integer... is it attempting to show -1?) of the event it complains about. After one or two re-runs and it becomes 3 again and then stays that way:

WARN: Event 3's digest does not match its payload

UEFI / BIOS release for XR4520c currently running in this unit is version "1.16.2".

I hope this helps.

[JuergenReppSIT]

@Sinihopea Yes the version 1.16.2 is utf-16 little endian encoded stored in the EV_S_CRTM_VERSION event. But the computed sha256 digest for this 14 byte field is wrong.

[Sinihopea]

@Sinihopea Yes the version 1.16.2 is utf-16 little endian encoded stored in the EV_S_CRTM_VERSION event. But the computed sha256 digest for this 14 byte field is wrong.

@JuergenReppSIT By "computed sha256 digest" do you mean that there is an issue in TPM chip itself?

[JuergenReppSIT]

@Sinihopea No I would not expect that it's an issue in the TPM chip. Perhaps you could try to update the BIOS to the recommended version 1.17.3:
https://www.dell.com/support/product-details/en-us/product/poweredge-xr4520c/drivers

[Sinihopea]

@Sinihopea No I would not expect that it's an issue in the TPM chip. Perhaps you could try to update the BIOS to the recommended version 1.17.3: https://www.dell.com/support/product-details/en-us/product/poweredge-xr4520c/drivers

We've tested several BIOS versions already... upgrading in stages and the problem has persisted all the way from initial 1.0.2 release so I'm bit skeptical about upgrading solving the issue. But sure, I'll get it done.

[Sinihopea]

@JuergenReppSIT Ok, I upgraded XR4520c BIOS into 1.17.3 and the problem still persists.

localhost:~ # tpm2_eventlog /sys/kernel/security/tpm0/binary_bios_measurements|grep -A 10 CRTM
WARN: Event 3's digest does not match its payload
EventType: EV_S_CRTM_CONTENTS
  DigestCount: 1
  Digests:
  - AlgorithmId: sha256
    Digest: "c0276941c41048d7353f6fd57de26e4b654f3396a5eff00bbfe93ace054ab961"
  EventSize: 27
  Event:
    BlobBase: 0x61754720746f6f42
    BlobLength: 0x757361654d206472
- EventNum: 3
  PCRIndex: 0
  EventType: EV_S_CRTM_VERSION
  DigestCount: 1
  Digests:
  - AlgorithmId: sha256
    Digest: "76d538bb3a43ef27b2e9ccca3b0bad02b9ee15ffbf0dadbdfa823dbf43d83a1f"
  EventSize: 14
  Event: "31002e00310037002e0033000000"
- EventNum: 4
  PCRIndex: 0
  EventType: EV_EFI_PLATFORM_FIRMWARE_BLOB
  DigestCount: 1

I've been bothered by those final zeroes at the end of the EV_S_CRTM_VERSION string. It's like it's missing the UTF-16LE BOM characters from the beginning. It would make sense for the string to begin with those two bytes instead of having two zero bytes at the end:

ff fe 31 00 2e 00 31 00 37 00 2e 00 33 00
      31 00 2e 00 31 00 37 00 2e 00 33 00 00 00

Unfortunately this change doesn't result the expected SHA256 digest. So it's probably something else.

Could it be that in XR4520c the Intel Boot Guard and BIOS measurements do not match which is the source of the issues with reconstruction... I'm referring to this: https://mjg59.dreamwidth.org/67602.html?thread=2145298

But there's a problem here. The event log is created by the firmware, and the Boot Guard measurements occur before the firmware is executed. So how do we get a log that represents them? That one's fairly simple - the firmware simply re-calculates the same measurements that Boot Guard did and creates a log entry after the fact[1]. All good.

Except. What if the firmware screws up the calculation and comes up with a different answer? The entry in the event log will now not match what was sent to the TPM, and replaying will fail. And without knowing what the actual value should be, there's no way to fix this, which means there's no way to verify the contents of PCR 7 and determine whether or not secure boot was enabled.

Edit: I'm too used to newer languages... of course the final zeroes are for C-string NUL. Silly me.

[Sinihopea]

I realised that the digest for EV_S_CRTM_VERSION cannot be correct. It seems that regardless of BIOS version the digest stays the same... hence the tpm2_eventlog complaint about the payload. I've opened a case with Dell support regarding this. Lets hope that they will take a look and comment on this.

[Yinette]

G'day, I think I'm having the same issue on a Dell PowerEdge R760, which appears to have the same Nuvoton NPCT 75x TPM chip:

TPM2_PT_MANUFACTURER:
  raw: 0x4E544300
  value: "NTC"
TPM2_PT_VENDOR_STRING_1:
  raw: 0x4E504354
  value: "NPCT"
TPM2_PT_VENDOR_STRING_2:
  raw: 0x37357800
  value: "75x"

Using Canonical's tcglog-check from https://github.com/canonical/tcglog-parser:

*** FAIL ***: The log is not consistent with what was measured in to the TPM for some PCRs:
	- PCR 0, bank TPM_ALG_SHA256 - actual value from TPM: 5fe4bcdcb22d5738f75a9b6bb6b4e1121bc565df05e030a9f757ce3da03e9a32, expected value from log: e63951196e922d71a4f498b08c5fd1560796d12a4c5e922ece66f8a388adaae6

and from tpm2_eventlog:

- EventNum: 3
  PCRIndex: 0
  EventType: EV_S_CRTM_VERSION
  DigestCount: 1
  Digests:
  - AlgorithmId: sha256
    Digest: "2272efc6f8c3db535ee6a54c8f3da2c36a9a631cdd374f38aae6ab4c6a1cc275"
  EventSize: 12
  Event: "32002e0034002e0034000000"
WARN: Event 3's digest does not match its payload

I'd say this is a problem with Dell's UEFI implementation/Bug in UEFI Firmware handling this particular chip. I'll have to make contact with Dell as well I think... this could be fun.