[CERT-TEST-FAILURE] [TC-TSTAT-2.2] several test steps use hard-coded values for writing instead of the values read from another attribure

[bauerschwan]

Feature Area

Other

Test Case

TC-TSTAT-2.2

Reproduction steps

Perform test case with a DUT which uses different AbsMinHeatSetpointLimit values than the values hard-coded in the YAML script (f.e. 1000 for AbsMinHeatSetpointLimit)

Expected result:
Step 6a should use values derived from AbsMinHeatSetpointLimit and should succeed

Actual result:
Step 6a uses a hard-coded value (800) which is not in range of [AbsMinHeatSetpointLimit... AbsMaxHeatSetpointLimit]

Bug prevalence

every time this test case is performed

GitHub hash of the SDK that was being used

17b1a38

Platform

nrf

Anything else?

No response

[ReneJosefsen]

As a nitpick here, the test case number is wrong in the description.

[raul-marquez-csa]

@bauerschwan, please provide the following:

• Full list of steps with the stated issue
• App used as (DUT)
• Command used to run the test
• Default PICS or custom PICS?
• Logs describing the failure/error

[raul-marquez-csa]

@bauerschwan

I have run the test several times without failure, cannot reproduce the bug.
Without the info from my previous comment, I ran it like this:
./scripts/tests/run_test_suite.py --log-level debug --chip-tool out/linux-x64-chip-tool/chip-tool --target Test_TC_TSTAT_2_2 run --all-clusters-app out/linux-x64-thermostat/thermostat-app

However, some observations:

Step 6a uses a hard-coded value (800) which is not in range of [AbsMinHeatSetpointLimit... AbsMaxHeatSetpointLimit]

In the test plan and the YAML, the max value used is MaxHeatSetpointLimit, not AbsMaxHeatSetpointLimit

Test Plan
https://github.com/CHIP-Specifications/chip-test-plans/blob/b9cb9be6fa63c16433928b0ffb4d53d2ee04343c/src/cluster/thermostat.adoc#L759-L762

YAML

connectedhomeip/src/app/tests/suites/certification/Test_TC_TSTAT_2_2.yaml

Lines 842 to 884 in 52a5589

	#Using saved values when optional attributes are available
	- label:
	"Step 6a: Test Harness Reads MinHeatSetpointLimit attribute from
	Server DUT and verifies that the value is within range"
	command: "readAttribute"
	attribute: "MinHeatSetpointLimit"
	PICS: TSTAT.S.F00 && TSTAT.S.A0015 && TSTAT.S.A0016 && TSTAT.S.A0003
	response:
	constraints:
	type: int16s
	minValue: AbsMinHeatSetpointLimitValue
	maxValue: MaxHeatSetpointLimitValue
	#Using hard coded values when optional attributes are not available
	- label:
	"Step 6a: Test Harness Reads MinHeatSetpointLimit attribute from
	Server DUT and verifies that the value is within range"
	command: "readAttribute"
	attribute: "MinHeatSetpointLimit"
	PICS: TSTAT.S.F00 && TSTAT.S.A0015 && !TSTAT.S.A0016 && !TSTAT.S.A0003
	response:
	constraints:
	type: int16s
	minValue: 700
	maxValue: 3000
	- label:
	"Step 6a: Test Harness Writes a value back that is different but valid
	for MinHeatSetpointLimit attribute"
	command: "writeAttribute"
	attribute: "MinHeatSetpointLimit"
	PICS: TSTAT.S.F00 && TSTAT.S.A0015
	arguments:
	value: 800
	- label:
	"Step 6a: Test Harness Reads it back again to confirm the successful
	write of MinHeatSetpointLimit attribute"
	command: "readAttribute"
	attribute: "MinHeatSetpointLimit"
	PICS: TSTAT.S.F00 && TSTAT.S.A0015
	response:
	value: 800

Also, as stated by Rene, the test case name in the description doesn't match the one in the title, are the issue details correct in the description?

Thanks.

[bauerschwan]

I changed the wrong test case name. Sorry for the confusion.

[fessehaeve]

@raul-marquez-csa the issue is not with the first 6a step - but the 3rd and 4th steps labeled 6a.
The 3rd "step 6a" attempts to write 800 to MinHeatSetpointLimit regardless of the AbsMinHeatSetpointLimit value.

YAML

connectedhomeip/src/app/tests/suites/certification/Test_TC_TSTAT_2_2.yaml

Lines 867 to 885 in 52a5589

	- label:
	"Step 6a: Test Harness Writes a value back that is different but valid
	for MinHeatSetpointLimit attribute"
	command: "writeAttribute"
	attribute: "MinHeatSetpointLimit"
	PICS: TSTAT.S.F00 && TSTAT.S.A0015
	arguments:
	value: 800
	- label:
	"Step 6a: Test Harness Reads it back again to confirm the successful
	write of MinHeatSetpointLimit attribute"
	command: "readAttribute"
	attribute: "MinHeatSetpointLimit"
	PICS: TSTAT.S.F00 && TSTAT.S.A0015
	response:
	value: 800

You are not running to this issue since you are running the test against all-clusters-app which has AbsMinHeatSetpointLimit value of 700.

[CKnoerzer]

@raul-marquez-csa Is it possible to have in line 875 and line 884 instead of:
value: 800
something like:
value: AbsMinHeatSetpointLimitValue+100

[raul-marquez-csa]

A python implementation of this test case has been developed by @hasty, PR
[HVAC] Implement TC-TSTAT-2.2 #36023

@bauerschwan, is this update still a priority?

[bauerschwan]

As soon as #36023 is merged, this issue can be closed