Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing DTS E2E tests to adhere to real workflows on real hardware. #653

Open
Tracked by #651
DaniilKl opened this issue Jan 9, 2025 · 0 comments
Open
Tracked by #651
Labels
dasharo-tools-suite Targeting DTS tests enhancement New feature or request

Comments

@DaniilKl
Copy link
Contributor

DaniilKl commented Jan 9, 2025

The idea of DTS E2E tests is to test DTS workflows for every supported hardware automatically on QEMU going from DTS main menu up to rebooting. But there is one problem, not all DTS E2E tests adhere to real workflows for the hardware they are said to test.

What are these hardware workflows? Consider following diagram:
image

This diagram shows an example DTS installation workflow. As you can see, it consists of several steps and the correct way to complete this workflow is to execute all these steps in a specific way (green arrows on the diagram). Which steps to execute depends on the platform DTS workflow is being run on. The problem is that, though we can simulate a specific platform during E2E tests, not all needed steps are being executing in the tests (for example some E2E tests omit EC transition step as shown on the diagram), therefore not all DTS E2E tests reflect real DTS workflows.

How to fix this? As was said, whether a specific step is being executed or not depends on the platform and its features that are being detected by DTS. In case of testing on QEMU and simulating a specific platform, DTS has a platform simulation interface in a form of shell variables that begin with TEST_ (check common-mock-func.sh for more inf.) that could be exported at the start. These variables are responsible for configuring some features on simulated platform, including platform model, firmware version, EC presence etc..

The only problem that remains is the way how to prove, that the workflow being tested in E2E test is similar to the same DTS workflow on real hardware. The only way I see it right now, is to develop the same E2E test on QEMU and real hardware at the same time, and compare its results. In that case, this issue depends on #654.

@DaniilKl DaniilKl changed the title Fixing tests to adhere to real workflows on real hardware. Fixing DTS E2E tests to adhere to real workflows on real hardware. Jan 9, 2025
@DaniilKl DaniilKl added enhancement New feature or request dasharo-tools-suite Targeting DTS tests labels Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dasharo-tools-suite Targeting DTS tests enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant