Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework iree_tests to keep going after compile XPASS. #149

Merged
merged 4 commits into from
Apr 3, 2024

Conversation

ScottTodd
Copy link
Member

If a test unexpectedly passes the compilation phase, this now keeps going and tests the runtime. Depending on the outcome, a specific message is logged allowing for easier updating of config files. Updating is still a manual process, but it now requires less moving test cases back and forth between XFAIL lists.

Sample logs:

________ IREE compile and run: test_averagepool_2d_precomputed_strides::cpu_llvm_sync_test ________ [gw20] win32 -- Python 3.11.2 D:\dev\projects\SHARK-TestSuite\iree_tests\nightly_pip.venv\Scripts\python.exe
[XPASS(strict)] Expected compile to fail (remove from 'expected_compile_failures')
___ IREE compile and run: test_dynamicquantizelinear_min_adjusted_expanded::cpu_llvm_sync_test ____ [gw50] win32 -- Python 3.11.2 D:\dev\projects\SHARK-TestSuite\iree_tests\nightly_pip.venv\Scripts\python.exe
Expected compile failure but run failed (move to 'expected_run_failures'):
Error invoking iree-run-module
Error code: 1
Stderr diagnostics:

Stdout diagnostics:
EXEC @test_dynamicquantizelinear_min_adjusted_expanded
[FAILED] result[0]: metadata is 3x4xi8; expected that the view matches 3x4xui8; expected that the view is equal to contents of a view of 3x4xui8
  expected:
3x4xui8=[64 134 83 159][213 255 96 166][249 255 191 149]
  actual:
3x4xi8=[64 -122 83 -97][-43 -1 96 -90][-7 -1 -65 -107]
[FAILED] result[2]: metadata is i8; expected that the view matches ui8; expected that the view is equal to contents of a view of ui8
  expected:
ui8=0
  actual:
i8=0

Compiled with:
  cd D:\dev\projects\SHARK-TestSuite\iree_tests\onnx\node\generated\test_dynamicquantizelinear_min_adjusted_expanded && iree-compile model.mlir --iree-hal-target-backends=llvm-cpu -o model_cpu_llvm_sync_test.vmfb

Run with:
  cd D:\dev\projects\SHARK-TestSuite\iree_tests\onnx\node\generated\test_dynamicquantizelinear_min_adjusted_expanded && iree-run-module --module=model_cpu_llvm_sync_test.vmfb --device=local-sync --flagfile=test_data_flags.txt

@ScottTodd
Copy link
Member Author

cc @dan-garvey since we chatted about this over at iree-org/iree#16717 (comment)

@saienduri saienduri self-requested a review April 3, 2024 07:11
Copy link
Contributor

@saienduri saienduri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ScottTodd ScottTodd merged commit e67779c into nod-ai:main Apr 3, 2024
1 check passed
@ScottTodd ScottTodd deleted the xfail-keep-going branch April 3, 2024 15:26
renxida pushed a commit that referenced this pull request Jul 18, 2024
If a test unexpectedly passes the compilation phase, this now keeps
going and tests the runtime. Depending on the outcome, a specific
message is logged allowing for easier updating of config files. Updating
is still a manual process, but it now requires less moving test cases
back and forth between XFAIL lists.

Sample logs:
```
________ IREE compile and run: test_averagepool_2d_precomputed_strides::cpu_llvm_sync_test ________ [gw20] win32 -- Python 3.11.2 D:\dev\projects\SHARK-TestSuite\iree_tests\nightly_pip.venv\Scripts\python.exe
[XPASS(strict)] Expected compile to fail (remove from 'expected_compile_failures')
```

```
___ IREE compile and run: test_dynamicquantizelinear_min_adjusted_expanded::cpu_llvm_sync_test ____ [gw50] win32 -- Python 3.11.2 D:\dev\projects\SHARK-TestSuite\iree_tests\nightly_pip.venv\Scripts\python.exe
Expected compile failure but run failed (move to 'expected_run_failures'):
Error invoking iree-run-module
Error code: 1
Stderr diagnostics:

Stdout diagnostics:
EXEC @test_dynamicquantizelinear_min_adjusted_expanded
[FAILED] result[0]: metadata is 3x4xi8; expected that the view matches 3x4xui8; expected that the view is equal to contents of a view of 3x4xui8
  expected:
3x4xui8=[64 134 83 159][213 255 96 166][249 255 191 149]
  actual:
3x4xi8=[64 -122 83 -97][-43 -1 96 -90][-7 -1 -65 -107]
[FAILED] result[2]: metadata is i8; expected that the view matches ui8; expected that the view is equal to contents of a view of ui8
  expected:
ui8=0
  actual:
i8=0

Compiled with:
  cd D:\dev\projects\SHARK-TestSuite\iree_tests\onnx\node\generated\test_dynamicquantizelinear_min_adjusted_expanded && iree-compile model.mlir --iree-hal-target-backends=llvm-cpu -o model_cpu_llvm_sync_test.vmfb

Run with:
  cd D:\dev\projects\SHARK-TestSuite\iree_tests\onnx\node\generated\test_dynamicquantizelinear_min_adjusted_expanded && iree-run-module --module=model_cpu_llvm_sync_test.vmfb --device=local-sync --flagfile=test_data_flags.txt
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants