You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The issue is that, when a solution function raises any exception, the test suite takes that result as a TEST_ERROR, while it should be considered a FAIL.
This is different from when a test itself raises an exception. There we correctly set the outcome to TEST_ERROR.
The idea we came up with is to parse the stack trace (extracted from the traceback with traceback.extract_tb()), and inspect the StackSummary object to see if it contains any "failure".
That is, if the names of the functions present in the stack trace mention a failure or an exception raised, then we know that the exception must come from the solution code, and not from the test.
We may need to revise this implementation idea if we find a better solution.
The text was updated successfully, but these errors were encountered:
This is something that we could not add to #160.
The issue is that, when a solution function raises any exception, the test suite takes that result as a
TEST_ERROR
, while it should be considered aFAIL
.This is different from when a test itself raises an exception. There we correctly set the outcome to
TEST_ERROR
.The idea we came up with is to parse the stack trace (extracted from the traceback with
traceback.extract_tb()
), and inspect theStackSummary
object to see if it contains any "failure".That is, if the names of the functions present in the stack trace mention a failure or an exception raised, then we know that the exception must come from the solution code, and not from the test.
We may need to revise this implementation idea if we find a better solution.
The text was updated successfully, but these errors were encountered: