Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testsuite: catch if exception is raised in the solution function #165

Open
edoardob90 opened this issue Dec 8, 2023 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@edoardob90
Copy link
Member

This is something that we could not add to #160.

The issue is that, when a solution function raises any exception, the test suite takes that result as a TEST_ERROR, while it should be considered a FAIL.

This is different from when a test itself raises an exception. There we correctly set the outcome to TEST_ERROR.

The idea we came up with is to parse the stack trace (extracted from the traceback with traceback.extract_tb()), and inspect the StackSummary object to see if it contains any "failure".
That is, if the names of the functions present in the stack trace mention a failure or an exception raised, then we know that the exception must come from the solution code, and not from the test.

We may need to revise this implementation idea if we find a better solution.

@edoardob90 edoardob90 added the enhancement New feature or request label Dec 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant